Skip to content

Monitoring vulnerabilities across your cluster

This guide sets up continuous vulnerability monitoring for every image running in a cluster, then shows how to read the results, cut them down to what matters, and keep them current.

It assumes the Kubescape operator is installed. Everything here is read with kubectl, because results are stored as Kubernetes objects.

1. Check scanning is enabled

Image scanning is on by default. Two capabilities matter:

capabilities:
  vulnerabilityScan: enable
  relevancy: enable

vulnerabilityScan produces the results. relevancy adds a second, smaller set of counts covering only the code actually loaded at runtime, which is what section 4 uses.

If you changed either, set them back:

helm upgrade kubescape kubescape/kubescape-operator \
  --reuse-values --set capabilities.vulnerabilityScan=enable

2. Wait for the first results

Images are scanned when a workload is created, and when its image tag changes. On an existing cluster the first full pass comes from the scheduled scan, so you can trigger one immediately instead:

kubectl create job --from=cronjob/kubevuln-scheduler first-scan -n kubescape
kubectl get pods -n kubescape -w

Large clusters take a while, since every distinct image is pulled and catalogued once.

3. Read the results

Start with one row per namespace:

kubectl get vulnerabilitysummaries

Then one row per workload and image:

kubectl get vulnerabilitymanifestsummaries -A

A summary carries severity counts, which is what you want for tracking. The all figure is everything found; relevant is the subset actually loaded:

kubectl get vulnerabilitymanifestsummaries -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,CRIT:.spec.severities.critical.all,HIGH:.spec.severities.high.all,CRIT_REL:.spec.severities.critical.relevant'

To see the individual CVEs behind a summary, read the manifest it points at:

kubectl get vulnerabilitymanifestsummaries -n default Deployment-nginx-deployment-nginx -o yaml
kubectl get vulnerabilitymanifests -A

Manifests are large. Filter rather than reading them whole:

kubectl get vulnerabilitymanifests -n default nginx-e06153 -o json \
  | jq -r '.spec.payload.matches[] | select(.vulnerability.severity=="Critical") | .vulnerability.id' | sort -u

See Vulnerability scanning for the full object reference.

4. Narrow to what is reachable

Most images carry vulnerabilities in code that never runs. With relevancy enabled, the node agent watches which files a container actually loads and produces a second set of counts, so severities.*.relevant is the number worth acting on first.

Relevancy needs a learning period per container, and it only observes new or restarted containers, so counts stay at zero until that completes. Vulnerability relevancy covers the details and limitations.

5. Silence what you have already accepted

Findings you have reviewed and accepted will otherwise reappear in every scan. A SecurityException records that decision in the cluster:

apiVersion: kubescape.io/v1beta1
kind: SecurityException
metadata:
  name: except-cve-2023-45853
  namespace: production
spec:
  reason: "zlib is present but the vulnerable path is never called"
  author: "platform-team"
  match:
    images:
      - "docker.io/library/nginx:*"
  vulnerabilities:
    - vulnerability:
        id: CVE-2023-45853
      status: not_affected
      justification: vulnerable_code_not_present
      expiredOnFix: true

This needs capabilities.riskAcceptance=enable. expiredOnFix is worth setting on anything you are accepting only because there is no fix yet: the exception stops applying as soon as one exists, so it cannot quietly hide a finding you could now act on.

Suppressed CVEs are not deleted. They move to the ignored list on the manifest with the rule that suppressed them, so the decision stays auditable. See Vulnerability exceptions.

6. Keep the picture current

A scan reflects the image at the time it ran. New CVEs are published against images you have already scanned, so results go stale even when nothing in the cluster changes.

The kubevuln-scheduler CronJob re-scans everything daily at midnight by default:

helm upgrade kubescape kubescape/kubescape-operator \
  --reuse-values --set kubevulnScheduler.scanSchedule="0 3 * * *"

Between scheduled runs, results are refreshed when a workload is created or its image tag changes, and when an exception is added, changed, or removed.

7. Tracking over time

For dashboards and alerting, enable the Prometheus exporter:

helm upgrade kubescape kubescape/kubescape-operator \
  --reuse-values --set capabilities.prometheusExporter=enable

It publishes severity counts as gauges at three levels, for critical, high, medium, low and unknown:

metric labels
kubescape_vulnerabilities_total_cluster_<severity> none
kubescape_vulnerabilities_total_namespace_<severity> namespace
kubescape_vulnerabilities_total_workload_<severity> namespace, workload, workload_kind, workload_container_name

There is a matching kubescape_vulnerabilities_relevant_<level>_<severity> set carrying the relevancy-filtered counts from section 4, which is usually the better thing to alert on.

- alert: RelevantCriticalVulnerability
  expr: kubescape_vulnerabilities_relevant_workload_critical > 0
  for: 30m
  annotations:
    summary: "{{ $labels.namespace }}/{{ $labels.workload }} has a reachable critical vulnerability"

negligible has no metric, so a Prometheus total will not match a summary object's own total for an image with negligible findings.

See Prometheus integration for wiring the exporter into your Prometheus.

Without Prometheus, the summary objects are the tracking surface directly. They are ordinary Kubernetes objects, so existing tooling works on them:

kubectl get vulnerabilitymanifestsummaries -A -o json \
  | jq '[.items[] | select(.spec.severities.critical.relevant > 0)] | length'

Where next