Introduction
Every deployment in a distributed system is an act of trust.
When a service binary rolls out across hundreds of nodes in multiple regions, the operator implicitly trusts that the artifact being deployed is the same one that was built, tested, and approved.
This trust is easily violated.
A single compromised dependency, a tampered container image, or a misconfigured build pipeline can inject malicious code into production infrastructure at scale, as demonstrated by the SolarWinds (2020) and Codecov (2021) incidents.
Software supply chain integrity is the property that every artifact deployed to a production system is a faithful, unmodified representation of source code that passed through an authorized, auditable build and release process.
In distributed deployments, where artifacts fan out across many nodes, regions, and environments, maintaining this property requires cryptographic verification at every handoff point, from source commit to running process.
Threat Model
Understanding supply chain integrity requires a clear threat model.
The attack surface spans the entire path from source code to running binary.
Source Threats
An attacker may compromise a developer's credentials and push malicious commits, or submit carefully disguised pull requests that introduce backdoors (the xz-utils backdoor of 2024 is a textbook example).
Dependency confusion attacks exploit package manager resolution logic to substitute internal packages with attacker-controlled public ones.
Build Threats
A compromised build system can inject code that does not appear in the source repository.
If builds are not reproducible, there is no way to independently verify that a given binary corresponds to a given source tree.
Shared CI/CD runners introduce cross-tenant contamination risks.
Distribution Threats
Artifact registries (container registries, package repositories) can be compromised or spoofed.
Man-in-the-middle attacks during artifact pull can substitute payloads if transport-layer verification is the only safeguard.
In distributed deploys, artifacts are often cached at regional mirrors or edge nodes, each of which is an additional point of potential tampering.
The Update Framework (TUF) was designed specifically to address these registry and distribution integrity problems: it provides a role-based, threshold-signed metadata scheme that protects against compromise of any single distribution key, rollback attacks, and indefinite freeze attacks, making it a natural complement to per-artifact signing in multi-region distribution architectures.
Deployment Threats
Even after an artifact arrives at a node, runtime mutation is possible.
An attacker with node-level access could replace binaries on disk or manipulate container layers.
Without continuous runtime attestation, post-deployment integrity is unverifiable.
Core Mechanisms
Cryptographic Artifact Signing
The foundation of supply chain integrity is cryptographic signing of artifacts at build time and verification at deploy time.
This creates a chain of accountability: a signature binds an artifact's content (via its digest) to an identity (the build system or release engineer).
Sigstore's Cosign and similar tools have made keyless signing practical by issuing short-lived certificates tied to OIDC identities, eliminating the operational burden of long-lived signing keys.
For container images, signatures can be stored alongside the image manifest in an OCI registry, making verification a standard part of the pull workflow.
Provenance Attestations
A signature proves who signed an artifact.
A provenance attestation proves how it was built.
SLSA (Supply-chain Levels for Software Artifacts) defines a framework for provenance that captures the source repository, commit hash, build platform, build configuration, and builder identity.
At SLSA Build L3 (the highest build level in SLSA v1.0), the provenance must be generated by a hardened, isolated build service that the developer cannot influence.
This prevents an insider from modifying the build process to inject code not present in the source.
The attestation itself is signed and distributed alongside the artifact.
Reproducible Builds
Reproducibility means that given the same source inputs, any independent party can produce a bit-for-bit identical artifact.
This is the strongest form of build integrity because it removes the need to trust the build system entirely.
If two independent builds produce the same digest, the artifact is almost certainly faithful to the source.
Achieving reproducibility is difficult in practice.
Non-deterministic elements like timestamps, filesystem ordering, and compiler randomization must be eliminated.
Languages with hermetic build systems (Go, Bazel-managed C++) fare better than ecosystems with heavy native dependency chains.
Policy Enforcement at Admission
Verification must be enforced, not optional.
In Kubernetes environments, admission controllers (such as Kyverno, OPA Gatekeeper, or the built-in ValidatingAdmissionPolicy) can reject any pod whose image lacks a valid signature or provenance attestation.
Binary Authorization (GCP) and similar services implement this pattern as a managed offering.
The policy engine acts as a chokepoint: regardless of how an artifact enters the cluster, it cannot run without passing verification.
This is critical in distributed deploys where multiple teams or automation systems may trigger deployments.
Walkthrough
The following walkthrough describes the end-to-end verification flow for deploying a signed container image with provenance attestation across a multi-region distributed system.
Step 1: Build and Attest
The CI system builds the container image in an isolated, hardened environment.
Upon completion, it generates a SLSA provenance attestation containing the source commit, repository URI, builder identity, and build timestamp.
Both the image digest and the attestation are signed using a short-lived Sigstore certificate tied to the CI system's OIDC identity.
build(source_commit, build_config):
image = compile_and_package(source_commit, build_config)
digest = sha256(image_manifest) # OCI digest is over the manifest
provenance = {
"source_repo": source_commit.repo,
"source_ref": source_commit.hash,
"builder_id": current_builder.identity,
"build_config": build_config.uri,
"timestamp": now_utc()
}
sig_image = sign_keyless(digest, oidc_token)
sig_provenance = sign_keyless(sha256(provenance), oidc_token)
push(registry, image, sig_image, provenance, sig_provenance)
return digest
Step 2: Replicate to Regional Registries
The image and its associated signatures and attestations are replicated to regional registries (or pull-through caches).
Replication preserves the original digests.
Regional registries do not re-sign; they serve the original signatures.
replicate(source_registry, target_registries, digest):
manifest = pull_manifest(source_registry, digest)
signatures = pull_signatures(source_registry, digest)
attestations = pull_attestations(source_registry, digest)
for region_registry in target_registries:
push(region_registry, manifest, signatures, attestations)
# Verify using the immutable digest, not a mutable tag
assert pull_manifest_digest(region_registry, digest) == digest
Step 3: Admission Verification
When a deployment controller submits a pod spec to the API server, the admission controller intercepts the request and performs verification before the pod is scheduled.
admit(pod_spec, policy):
for container in pod_spec.containers:
digest = resolve_tag_to_digest(container.image)
sig = fetch_signature(registry, digest)
provenance = fetch_attestation(registry, digest)
if not verify_signature(sig, digest, policy.trusted_identities):
return REJECT("invalid or missing signature")
if not verify_signature(provenance.sig, provenance, policy.trusted_identities):
return REJECT("invalid provenance signature")
if not matches_policy(provenance, policy.source_repos, policy.builders):
return REJECT("provenance does not match policy")
return ADMIT
Step 4: Runtime Verification (Optional, Defense in Depth)
On the node, a daemon periodically re-verifies the digest of running container images against the expected digest recorded at admission time.
The digest is computed over the OCI image manifest (which in turn references layer digests), consistent with how OCI image identity is defined.
Any mismatch triggers an alert or automatic pod eviction.
runtime_verify(node, expected_digests):
for container in node.running_containers:
# Digest is over the OCI manifest, not raw layer bytes
actual_digest = sha256(container.image_manifest)
if actual_digest != expected_digests[container.id]:
alert("runtime integrity violation", container)
evict(container)
Challenges in Practice
Dependency depth. A typical application has hundreds of transitive dependencies.
Signing the final artifact does not guarantee the integrity of each dependency.
SBOMs (Software Bills of Materials) help enumerate components, but verifying each one's provenance remains an unsolved problem at scale.
Key management and identity. Keyless signing shifts trust to the OIDC provider and the certificate transparency log.
If the OIDC provider is compromised, an attacker can obtain valid signing certificates.
Organizations must carefully scope which identities are authorized to produce trusted signatures.
Performance at scale. Signature and provenance verification add latency to every deployment.
In a system deploying thousands of pods per minute during a rollout, the verification path must be highly optimized.
Caching verification results (keyed by digest) helps, but introduces a cache invalidation concern if a signature is revoked.
Retrofitting existing systems. Most organizations cannot adopt SLSA Build L3 overnight.
The framework is designed for incremental adoption, but achieving meaningful integrity guarantees requires hardened build infrastructure, which is a significant investment.
Multi-tenant registries. When multiple teams share a registry, namespace isolation and per-team signing policies become critical.
A permissive policy in one namespace can undermine the integrity guarantees of another if artifacts can cross namespace boundaries.
Key Points
- Software supply chain integrity requires cryptographic verification at every stage: build, distribution, and deployment.
- Provenance attestations (as defined by SLSA) provide verifiable evidence of how an artifact was built, complementing signatures that prove who built it.
- Admission controllers serve as the critical enforcement point, ensuring no unverified artifact can run in production.
- Reproducible builds offer the strongest integrity guarantee by removing the need to trust the build system itself.
- Keyless signing via short-lived certificates reduces operational overhead but shifts trust to the OIDC identity provider.
- Runtime re-verification provides defense in depth against post-deployment tampering on individual nodes.
- Transitive dependency integrity remains a largely unsolved problem, with SBOMs providing visibility but not full verification.
- The Update Framework (TUF) addresses distribution-layer integrity threats such as registry compromise, rollback attacks, and key compromise, complementing per-artifact signing schemes.
References
SLSA Team. "SLSA: Supply-chain Levels for Software Artifacts, Specification v1.0." https://slsa.dev/spec/v1.0/, 2023.
Sigstore Project. "Cosign: Container Signing, Verification and Storage in an OCI Registry." https://github.com/sigstore/cosign, 2021.
Lamb, C. and Zacchiroli, S. "Reproducible Builds: Increasing the Integrity of Software Supply Chains." IEEE Software, vol. 39, no. 2, pp. 62-70, 2022.
The Update Framework (TUF). "Specification v1.0." https://theupdateframework.io/, 2019.
Torres-Arias, S., Afzali, H., Kuppusamy, T. K., Trisovic, A., and Cappos, J. "in-toto: Providing Farm-to-Table Guarantees for Bits and Bytes." Proceedings of the 28th USENIX Security Symposium, 2019.