HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Well-Architected Framework

Remediate leaked secrets

Once you detect a leaked secret, immediate remediation is critical. Attackers quickly exploit exposed credentials, and breach notification regulations, such as GDPR and HIPAA, impose strict timelines. The remediation process involves forming an incident response team, following NIST guidelines, and systematically removing secrets from all locations including version control, documentation, and backups.

If you have not already done so, form an incident response team that includes technical representatives from the security, operations, and development teams, as well as business and legal representatives.

The remediation process varies based on the following factors:

  • The type of system the leaked secret gives access to
  • The type of data available in that system
  • Whether regulations require you to report the incident to a government or industry regulator

Your remediation process should include communication with your stakeholders, both internal, such as development teams, and external, such as customers, to ensure they stay informed of any actions they may need to take.

Why remediate leaked secrets immediately

Stop attackers before they establish persistence: Once attackers discover leaked credentials, they quickly exploit access. Delayed remediation allows attackers to establish persistence, access data, or create backdoor access that remains even after you rotate the original secret.

Meet breach notification deadlines: Regulations like GDPR, HIPAA, and state breach notification laws impose strict reporting deadlines. Organizations that fail to detect and remediate breaches promptly face significant fines and mandatory public disclosure.

Prevent system re-compromise: Rotating a leaked secret without removing it from all locations allows attackers to re-compromise systems. Secrets remain in Git history, backups, logs, and documentation unless you systematically remove them from every location.

Reduce breach costs: Breaches can be expensive, with incident response, forensics, legal fees, and customer notification consuming significant resources. Organizations that remediate quickly limit exposure and reduce total breach costs.

Follow the NIST incident response framework

The National Institute of Standards and Technology (NIST) is a helpful resource for defining requirements for incident response. NIST SP 800-61: Computer Security Incident Handling Guide identifies the following four stages of incident response:

  1. Preparation: Ensure you have the right tools and processes in place to identify, manage, and remediate incidents.
  2. Detection and analysis: Identify the incident, and analyze the impact of the incident.
  3. Containment, eradication, and recovery: Contain the incident, remove the cause of the incident, and recover from the incident.
  4. Post-incident activity: Review the incident, and update your processes to prevent similar incidents in the future.

Remove leaked secrets from documentation

If you identify a leaked secret in documentation, first identify what teams, workflows, or applications depend on this secret. A common practice is to document requirements in a configuration management database (CMDB). Similarly, if you are using Terraform to manage your Vault policies, you can cross-reference the policies to determine what teams or applications are using the secret.

If the processes using the secret already retrieve the secret from Vault, complete the following steps:

  • Rotate or change the secret
  • Remove the plaintext secret from the documentation
  • Update the secret value in Vault

Any process retrieving the secret receives the updated secret value from Vault.

The following example shows how to rotate a leaked password in HashiCorp Vault:

# Generate new secret value
NEW_SECRET=$(openssl rand -base64 32)

# Update secret in Vault
vault kv put secret/app/password password="$NEW_SECRET"

# Verify rotation
vault kv get secret/app/password

Applications configured to read from Vault automatically receive the updated credentials without code changes, eliminating downtime during rotation. The KV v2 secrets engine serves the new password on the next request or connection retry, making rotation seamless for workloads that don't cache credentials. Run this rotation immediately upon detecting a leaked secret to revoke attacker access.

Remediate secrets exposed in source code

Leaked secrets in source code require immediate rotation followed by systematic removal from version control history. Simply deleting the file or line does not remove the secret. It remains accessible in Git history until you explicitly purge it.

Immediate actions

Complete these steps in order to contain the exposure and prevent re-compromise:

  1. Rotate the compromised secret immediately: Assume the secret is already compromised. Generate a new secret and update it in Vault before modifying any code.

  2. Update source code to retrieve secrets securely: Replace hardcoded secrets with secure retrieval methods. Choose based on your constraints:

    • Vault Agent: For legacy applications or quick remediation with minimal code changes. The agent handles authentication and writes secrets to files or environment variables that applications read without modification.
    • Vault Secrets Operator: For Kubernetes workloads. Syncs secrets from Vault into Kubernetes Secret objects that pods consume through standard mechanisms.
    • Native Vault SDK/API: For maximum security when you can refactor applications. Retrieves secrets at runtime through Vault APIs, eliminating secrets from disk and environment variables entirely.

    Organizations typically start with Vault Agent for quick wins, then migrate to native SDK integration or Secrets Operator as applications mature.

  3. Remove the secret from Git history: Carefully remove the secret from all commits. Follow the Removing sensitive data from a repository guide from GitHub.

  4. Invalidate cached copies: Contact your Git hosting provider to clear cached views of the exposed commits. Providers like GitHub, GitLab, and Bitbucket have different processes for cache invalidation, so follow their specific instructions.

HashiCorp resources

Related WAF guidance:

Get started with secret rotation and remediation:

Vault integration methods:

External resources

Next steps

In this section of Managing leaked secrets, you learned how to remediate leaked secrets following NIST incident response framework, remove secrets from documentation and source code, and implement secure credential retrieval methods. Remediating leaked secrets is part of the Secure systems pillar.

After remediating leaked secrets, implement prevention strategies to reduce future incidents:

Edit this page on GitHub