Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.
Introduction
A Docker image is not a single opaque artifact — it is a stack of content-addressable layers, each a diff of the filesystem produced by one instruction in the Dockerfile. That layering makes builds fast and caching efficient, but it also means the image’s history is preserved: a file written in one RUN step and deleted in a later step is not actually gone, it is still sitting in the earlier layer’s tarball and shippable with the image. Anyone who can pull the image can pull every layer, and with them any secret or private source file that ever touched the build context, even if the final running container never shows it.
This matters because container images are the unit of distribution and trust in modern deployments — pushed to shared registries, pulled by CI runners, scanned (or not) once, and run unmodified across dev, staging, and production. A secret baked into a layer during a rushed build is duplicated into every environment that pulls that image, and a vulnerable package baked into a base image is silently inherited by every image FROM it — which is why layer discipline, secret handling, and vulnerability scanning are treated as first-class supply-chain controls.
Attack Prerequisites
Extracting secrets or vulnerabilities from a Docker image generally requires only pull access — a much lower bar than most assume:
- Pull access to the image or its registry — a public Docker Hub image, a misconfigured private registry, or internal registry credentials (often the same ones used by CI to push).
- A secret present in the build context or an intermediate
RUN/COPYlayer — an.envfile, SSH key, cloud credential, or API token used topip install/npm installfrom a private index. - No multi-stage build discipline — secrets and build tooling end up in the final shipped image instead of being discarded with an intermediate stage.
- No
.dockerignore, so the entire build context (.git,.aws/credentials,.env, local config) is sent to the daemon and available toCOPY .-style instructions.
How It Works
Every RUN, COPY, and ADD instruction produces a new, immutable layer, and the image manifest is an ordered list of layers stacked with a union filesystem (overlay2 in modern Docker). A later layer can *hide* a file from view by deleting it, but hiding is implemented with a whiteout marker — the underlying layer tarball, and the file inside it, is unchanged and still distributed. A secret written in layer 3 and rm‘d in layer 5 is still sitting in layer 3’s tarball, extractable with nothing more than tar and grep.
docker history <image> lists every layer with the command that produced it, often enough to reveal an ARG/ENV secret passed on the command line (--no-trunc shows the full command). It only shows the *command*, not file contents, though — for that, dive interactively browses each layer’s filesystem and diffs it against the previous one, which is exactly how a secret added then deleted gets found. docker save | tar -x and grepping each layer.tar reaches the same files without extra tooling.
Vulnerability scanning works on a different axis: Trivy and Grype build a Software Bill of Materials (SBOM) of every OS package and dependency baked into the image — reading package manager databases (dpkg/rpm/apk) and lockfiles (package-lock.json, requirements.txt, Go module info) embedded in the layers — and diff it against CVE feeds. A base image like python:3.9 bundles hundreds of OS packages, so “my app”‘s vulnerability surface is overwhelmingly the base image’s, which is why scanning must run on the final built image, not just source.
Vulnerable Code / Configuration
A textbook secret-leaking Dockerfile — a credential authenticates a private package fetch, then is “cleaned up,” but the clean-up happens in a later layer while the earlier layer still ships:
FROM node:18
WORKDIR /app
# VULNERABLE: secret baked into a layer via a build ARG
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && \
npm install && \
rm .npmrc
# .npmrc with the plaintext token still exists in the layer created by
# this RUN instruction -- 'rm' only removes it from the *next* layer's view.
# `docker history --no-trunc` also reveals NPM_TOKEN if passed via
# `docker build --build-arg NPM_TOKEN=...` without BuildKit secret mounts.
COPY . .
CMD ["node", "server.js"]
Missing .dockerignore compounding it — the build context carries the secrets straight into the image via a broad COPY:
# Directory being built (no .dockerignore present):
myapp/
├── Dockerfile
├── .git/ <- full history, may contain old committed secrets
├── .env <- live database/API credentials
├── .aws/credentials <- developer's AWS keys, if run from $HOME
├── node_modules/
└── server.js
# Dockerfile:
COPY . .
# Every file above -- including .env, .git, and any local credential file
# left in the build directory -- is sent to the daemon and copied into
# the image layer, with no .dockerignore to exclude them.
The fix, for contrast — multi-stage build with BuildKit secret mounts (the secret never touches a layer) and a non-root, minimal runtime stage:
# syntax=docker/dockerfile:1
FROM node:18 AS build
WORKDIR /app
COPY package*.json .
# --mount=type=secret: token exists only in this RUN's process env,
# never written to a layer or the final image.
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN=$(cat /run/secrets/npm_token) \
npm config set //registry.npmjs.org/:_authToken=${NPM_TOKEN} && \
npm ci --omit=dev
COPY . .
FROM gcr.io/distroless/nodejs18-debian12
# distroless: no shell, no package manager -- nothing for RCE to pivot on
WORKDIR /app
COPY --from=build /app /app
USER nonroot
CMD ["server.js"]
Walkthrough / Exploitation
Pulling and inspecting layer history for leaked build-time secrets is the first move on any target image:
docker pull target/app:latest
docker history --no-trunc target/app:latest | less
# grep for common secret-passing patterns in the command history
docker history --no-trunc target/app:latest | grep -iE \
'token|secret|password|key|AKIA'
Browsing every layer’s filesystem contents with dive to catch files added and later deleted in a subsequent layer (the .npmrc scenario above):
dive target/app:latest
# in dive's UI: 'Ctrl+F' to filter added/modified files per layer;
# a file present in layer N and absent (whiteout) in layer N+1 is
# still fully readable by pressing enter on layer N.
Without extra tooling, docker save plus tar reaches the same files and is scriptable for bulk auditing:
docker save target/app:latest -o app.tar
mkdir extracted && tar -xf app.tar -C extracted
# each layer is its own tarball; extract and grep all of them
for l in extracted/blobs/sha256/*; do
tar -tf "$l" 2>/dev/null | grep -E '\.npmrc|\.env|id_rsa|credentials'
done
Running a vulnerability scan to enumerate known-vulnerable packages inherited from the base image:
trivy image --severity HIGH,CRITICAL target/app:latest
grype target/app:latest
Note:
docker historytruncates long commands by default — always add--no-trunc, otherwise the exact secret value is cut off mid-string and easy to miss on a quick scroll.
Opsec: When auditing images you don’t control, pull by digest (
docker pull target/app@sha256:...) rather than a mutable tag to avoid tipping off a defender watching pull logs, and to guarantee the layer contents analyzed are exactly reproducible.
Detection and Defense
Image hygiene is enforced at build time — a secret pushed to a shared registry should be treated as compromised and rotated, not just deleted from a future layer:
- Never pass secrets via
ARG/ENV/plainCOPY— use BuildKit--mount=type=secret(or--mount=type=ssh) so the value exists only in theRUNprocess’s memory and never becomes part of a layer. - Multi-stage builds — compile/fetch in an early stage with build tooling and credentials, then
COPY --from=buildonly the compiled artifact into a clean final stage that never saw the secret. - Maintain a
.dockerignorecovering.git,.env,*.pem,.aws/, and any local credential files, mirroring.gitignoreat minimum. - Use minimal or distroless base images and a non-root
USERso a container compromise has no shell or package manager to pivot with. - Scan every image in CI before push, and on a schedule against images already in the registry, with Trivy, Grype, or the registry’s built-in scanner; fail the pipeline on HIGH/CRITICAL findings.
- Rotate any credential found in an image layer immediately — removing it from the Dockerfile does not invalidate a token already pushed and potentially pulled elsewhere.
Real-World Impact
Secrets embedded in image layers are a routine finding in image audits and a well-documented category in Docker’s own security documentation and OWASP’s Docker Security Cheat Sheet, precisely because the union-filesystem behavior (deleted files persisting in earlier layers) is unintuitive to developers who assume rm in a later RUN step means the file never existed. Public registries such as Docker Hub have repeatedly been shown, in independent research scans, to host images with recoverable API keys and cloud credentials in exactly this pattern.
Conclusion
A Docker image is its full layer history, not just its final visible filesystem — “deleted” in a later layer does not mean “gone” from what gets pushed and pulled everywhere. The fix is structural: keep secrets out of layers with BuildKit secret mounts, discard build tooling with multi-stage builds, ship minimal non-root runtime images, and scan every image before and after it reaches a registry.
You Might Also Like
If you found this useful, these related deep-dives cover adjacent techniques and their defenses:
Comments