Davidson Ekpokpobe
← All notes
Engineering note

Reducing Thousands of Vulnerability Findings Without Hiding the Risk

How cleaning stale container images, remediating what was running, tightening SSH exposure, and documenting exceptions made SOC 2 vulnerability management more useful.

At one point, our container security scans were reporting vulnerabilities in the thousands.

That number looked alarming.

It also wasn't particularly useful on its own.

My work around our SOC 2 program forced me to think about vulnerability management differently. The goal wasn't to make a dashboard turn green. It was to understand what was actually vulnerable, remove risk we no longer needed to carry, fix what we controlled, document what we didn't, and leave behind a process we could defend.

That distinction became important very quickly.

A vulnerability count is not the same thing as a risk model

Our container images lived in Azure Container Registry.

Like many registries that have been running for a while, it had accumulated old image tags from previous builds and deployments.

Security scanners don't necessarily care that an image hasn't been deployed in months.

If it still exists in the registry, it can still be scanned.

And if twenty historical images all contain the same vulnerable package, the resulting dashboard can make one underlying dependency issue appear twenty times.

Conceptually:

CVE-XXXX
   |
   |-- api:build-712
   |-- api:build-713
   |-- api:build-714
   |-- api:build-715
   |-- api:build-716
   -- ...

The scanner isn't necessarily wrong.

Those vulnerable artifacts really do exist.

But the raw number doesn't tell you whether you're looking at:

So before trying to "fix thousands of vulnerabilities", I needed to understand what those thousands actually represented.

First: remove artifacts we had no reason to keep

One of the highest-leverage pieces of work was cleaning up Azure Container Registry.

Over time, our build and deployment process had left behind large numbers of older image tags.

Those images were no longer useful operationally, but they were still part of our attack surface and still being picked up by vulnerability scans.

Keeping them had several costs:

So we introduced cleanup around obsolete and duplicated image tags.

This sounds almost administrative compared with patching application code.

It wasn't.

Security inventory is part of security.

If an artifact doesn't need to exist, the safest vulnerable package inside it is the one you delete with the artifact.

After cleanup, the vulnerability picture became significantly more representative of the infrastructure we actually cared about.

Then fix what is actually running

Registry cleanup cannot become an excuse for ignoring vulnerable software.

Once stale artifacts were out of the way, the remaining findings were easier to reason about.

That meant working through things such as:

For example, part of the remediation work included upgrading infrastructure dependencies such as Debezium when newer versions addressed issues relevant to our security scans.

The important distinction was:

Finding
   |
   v
Is this artifact relevant?
   |
   |-- No  -> remove it
   |
   |-- Yes
         |
         v
     Is a fix available?
         |
         |-- Yes -> upgrade / patch / rebuild
         |
         -- No
               |
               v
          document risk
          track upstream
          review exception

That is a much more useful process than attempting to drive the scanner count to zero at any cost.

Some findings cannot be fixed immediately

This was another important part of the work.

There were vulnerabilities for which a clean remediation wasn't immediately available because the affected dependency was upstream of us.

Suppressing those findings without explanation would have been the wrong answer.

Keeping them permanently red without context would also have been unhelpful.

Instead, exceptions needed to be deliberate:

A security exception should be a risk decision with an expiry path, not a button used to make a dashboard prettier.

Port 22 was another kind of finding

Not all of the SOC 2 work was about vulnerable packages.

Some of it was basic infrastructure exposure.

One control highlighted that SSH access over port 22 should not be broadly exposed to the public internet.

A rule that effectively allows:

0.0.0.0/0 -> TCP 22

is easy to operate.

It is also a very large trust boundary.

With guidance from our compliance tooling and the security requirements we were implementing, I tightened the relevant network rules so administrative connectivity didn't depend on globally exposing SSH.

The interesting part was that security controls have to coexist with delivery infrastructure.

CI/CD still needed to perform the operations required for deployment.

So the problem wasn't simply:

Close port 22.

It was:

Remove unnecessary public exposure while preserving the trusted path the deployment system actually needs.

That difference matters.

Security controls that break delivery will eventually be bypassed.

The better control is the one that reduces access while keeping the legitimate workflow explicit.

Vanta was useful, but it wasn't the security work

We used Vanta heavily during the SOC 2 process.

It helped surface failing controls, collect evidence and expose areas where our environment did not match the expected security posture.

But the distinction I came to appreciate was:

compliance tooling can tell you that a control is failing. It cannot make the underlying system secure for you.

For example, Vanta might reveal:

Public SSH exposure
        |
        v
     FAILED

The actual engineering still involves understanding:

The same applied to vulnerability remediation.

The dashboard was the signal.

The engineering work happened underneath it.

Evidence became part of the engineering work

One thing SOC 2 changed for me was how I thought about evidence.

Normally, an engineer might consider a vulnerability fixed when the patch reaches production.

For an audited control, that is only part of the story.

You may also need to demonstrate:

Finding
  |
  v
Remediation work
  |
  v
Code / infrastructure change
  |
  v
Review
  |
  v
Deployment
  |
  v
New scan
  |
  v
Evidence

That meant maintaining evidence such as:

At one point, even a high-level report showing that the vulnerability count had fallen wasn't enough evidence by itself.

The control needed examples connecting specific findings to specific remediation work.

That was a useful lesson.

Security posture has to be inspectable.

"I fixed it" is not a control.

A reproducible trail showing what was wrong, what changed, who reviewed it, and what the system looked like afterward is much closer to one.

Why this mattered beyond the audit

It would be easy to describe all of this as work we did because SOC 2 required it.

That misses the more important point.

For a B2B software company, especially one handling customer data and selling to larger organisations, security controls become part of the product.

An enterprise customer evaluating your software is also evaluating questions such as:

SOC 2 gives customers and procurement teams a recognised way to reason about some of those questions.

So reducing vulnerabilities, tightening network exposure and producing defensible evidence wasn't merely about passing controls in Vanta.

It strengthened the company's security posture and made that posture easier to demonstrate externally.

That's where security engineering starts having a direct relationship with commercial credibility.

What I took away from it

Inventory is part of vulnerability management. You cannot reason clearly about risk if your scanners are dominated by artifacts that should not exist anymore.

A lower vulnerability count is not automatically a safer system. The important question is what disappeared and why.

Exceptions need ownership. When an upstream dependency prevents remediation, the finding should become tracked risk rather than forgotten noise.

Network exposure should be intentional. Convenience is not a sufficient reason for privileged ports to be reachable from everywhere.

Evidence matters. A security control is stronger when someone other than the person who implemented it can inspect what happened.

And perhaps the biggest lesson:

Compliance works best when it is the consequence of good engineering, not a separate exercise performed for an auditor.

The dashboards eventually became much quieter.

More importantly, we understood why.