Secrets Management and Hunting Leaked Secrets in Git

Containers & DevSecOps
2025.09.08
Time it takes to read this article 5 minutes.

Introduction / Overview

Disclaimer: This article is for educational purposes and authorized security testing only. Scanning repositories you do not own, exfiltrating credentials, or accessing systems without explicit written permission is illegal. Only run these techniques against assets you own or are contractually engaged to test.

Hardcoded secrets remain one of the most reliable footholds in modern engagements. A single AWS access key committed three years ago and "removed" in the next commit still sits in git history, ready to be replayed. On the defensive side, the fix is twofold: stop leaking secrets, and centralize their lifecycle in a system like HashiCorp Vault. This article walks both sides — hunting with gitleaks and trufflehog, then remediating with proper secrets management.

How it works / Background

Git is content-addressable. When you commit a .env file containing AWS_SECRET_ACCESS_KEY, that blob is stored in .git/objects forever. Deleting the file in a later commit only changes the tree — the original blob is still reachable from earlier commits and via the reflog. This is the core reason git rm is not remediation.

Two tools dominate secret hunting:

  • gitleaks — a fast, regex/entropy-based scanner. It ships with a large rule set (AWS keys, GitHub PATs, Stripe keys, private keys) and walks the entire commit graph.
  • trufflehog — focuses on verified secrets. It not only detects candidate credentials but actively calls the provider API to confirm a key is live, dramatically cutting false positives.

This maps to MITRE ATT&CK T1552.001 (Unsecured Credentials: Credentials In Files) and T1552.004 (Private Keys).

Prerequisites / Lab setup

Install the tooling. On macOS:

brew install gitleaks trufflehog
# or pull the official containers
docker pull zricethezav/gitleaks:latest
docker pull trufflesecurity/trufflehog:latest
Bash

Create a deliberately vulnerable lab repo so nothing real is at risk:

mkdir secrets-lab && cd secrets-lab && git init
cat > .env <<'EOF'
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
EOF
git add .env && git commit -m "add config"

# Pretend to "fix" the leak — but the blob survives in history
git rm .env && echo ".env" > .gitignore
git add .gitignore && git commit -m "remove secrets, add gitignore"
Bash

The AKIA...EXAMPLE value above is AWS's documented non-functional example key, safe to use in a lab.

Walkthrough / PoC

1. Scan the full history with gitleaks

gitleaks detect walks every commit, not just the working tree:

gitleaks detect --source . --report-format json --report-path leaks.json -v
Bash

Even though .env was removed at HEAD, gitleaks flags the commit where it was introduced, printing the rule ID, file, commit hash, and author. Use git log mode to scope a range, or scan an unstaged working tree pre-commit:

gitleaks protect --staged -v   # pre-commit guard, scans staged changes only
Bash

2. Verify live secrets with trufflehog

trufflehog's killer feature is --only-verified, which filters to credentials it could authenticate against the provider:

# Scan a local git repo, all branches and history
trufflehog git file://. --only-verified --json

# Scan a remote without cloning manually
trufflehog github --repo=https://github.com/example/app --only-verified
Bash

For containers and filesystems, trufflehog covers more than git:

trufflehog docker --image=myregistry/app:latest --only-verified
trufflehog filesystem /path/to/build --only-verified
Bash

A verified AWS hit means the key is active right now — in a real engagement that is an immediate, high-severity finding worth confirming with a benign aws sts get-caller-identity.

3. Inspect the raw object that the "fix" left behind

To prove the blob persists after git rm, list dangling objects:

git rev-list --all --objects | grep ".env"
git cat-file -p <blob-hash>   # prints the original secret content
Bash

4. Remediate: rotate, purge, then centralize in Vault

The correct order is rotate first (assume any committed secret is burned), then scrub history with git filter-repo:

pip install git-filter-repo
git filter-repo --path .env --invert-paths --force
git push origin --force --all   # rewrites remote history
Bash

Finally, move the secret out of files entirely and into Vault:

# Start a dev Vault (NOT for production)
vault server -dev &
export VAULT_ADDR='http://127.0.0.1:8200'

# Store the secret in the KV v2 engine
vault kv put secret/app/aws \
  access_key="AKIA..." secret_key="wJ..."

# Application reads it at runtime — nothing touches disk
vault kv get -field=secret_key secret/app/aws
Bash

In production you replace static tokens with AppRole auth or short-lived dynamic secrets (Vault's AWS secrets engine mints temporary IAM credentials per request), so there is no long-lived key to leak.

Mermaid diagram

Secrets Management and Hunting Leaked Secrets in Git diagram 1

The diagram shows the lifecycle: detection feeds verification, verified secrets trigger rotation and history purging, and everything converges on Vault-managed short-lived credentials.

Detection & Defense (Blue Team)

Preventing leaks is cheaper than chasing them. Apply these controls, weighted to match the offensive surface above:

  • Pre-commit hooks. Block secrets before they ever land. Use the gitleaks pre-commit hook so gitleaks protect --staged runs on every commit:
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks
YAML
  • CI gate. Run a full-history scan on every PR and fail the build on findings. GitHub's native Push Protection and Secret Scanning (free for public repos) provide a second layer that also notifies partner providers to auto-revoke leaked tokens.
# .github/workflows/scan.yml (excerpt)
- uses: gitleaks/gitleaks-action@v2
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
YAML
  • Centralize and rotate. Adopt HashiCorp Vault (or cloud equivalents — AWS Secrets Manager, GCP Secret Manager). Prefer dynamic secrets and short TTLs so a leaked credential expires quickly. Enable Vault audit devices (vault audit enable file) to log every secret access.
  • Least privilege. Scope IAM keys narrowly so a leaked key has minimal blast radius. Pair with anomaly detection — AWS GuardDuty flags credential use from unusual ASNs or geographies, catching replay of a stolen key.
  • Assume breach on detection. When a secret is confirmed leaked, rotation is non-negotiable even after git filter-repo, because anyone could have cloned the repo before the rewrite. Treat history scrubbing as cleanup, not containment.
  • Audit logging and alerting. Monitor CloudTrail / Vault audit logs for the leaked key's identifier. A spike in AccessDenied after rotation often reveals an attacker still trying the dead credential.

For broader supply-chain hardening, see my notes on container image scanning and SBOM and dependency auditing.

Conclusion

Secrets in git are a footgun precisely because deletion feels like remediation but isn't. The offensive workflow — gitleaks for breadth, trufflehog --only-verified for confirmed live keys, raw object inspection to prove persistence — is fast and high-yield. The defensive answer is structural: catch secrets at the pre-commit and CI gates, centralize them in Vault with short-lived dynamic credentials, and treat any confirmed leak as a rotation event. If you also operate CI/CD pipelines, review hardening GitHub Actions runners to close the runner-side exposure.

References

You Might Also Like

If you found this useful, these related deep-dives cover adjacent techniques and their defenses:

Comments

Copied title and URL