Identify, prevent, and manage leaked secrets

21min
|
Terraform
Boundary
Vault
Video

In a technology-enabled world, almost everything relies on secure access to systems whether you are checking your email, swiping a loyalty card at your favorite store, or your company's payroll system making a deposit in your bank account. These systems and processes have one thing in common — they all require some sort of secret to access them and ensure data confidentiality, integrity, and availability. Secrets can be a password, an API key, or a JSON web token. These secrets authenticate people accessing the system or different systems interacting with each other.

Developer productivity relies on the ability to securely access and use secrets to connect applications and services. If teams are unable to efficiently access secrets, the likelihood of secrets being leaked increases.

A secret is considered leaked when it is published in a location that it was not intended for. For example, an API key would be considered leaked if it was committed to a Git repository or added to documentation in plain text. The API key is now potentially accessible to anyone with access to the repository or documentation exposing the system to unauthorized access.

If a secret is leaked, people, systems, and organizations that rely on the secure exchange of information are at risk. When managed properly, you should store secrets in a secure location, verify the secrets are used as intended, and ensure they are not stored or written down in plain text.

This article provides guidance and resources to ensure you can prevent leaked secrets, identify leaked secrets, and remediate the incident if a secret is leaked.

How do secrets leak?

Some of the main ways organizations leak secrets are code exposure, process friction, insider threats, and trusting 3rd party tools such as AI or data processing software.

Code repositories are one of the most common sources of secret exposure. Developers accidentally commit API keys, passwords, and tokens directly into source code. Once secrets are in code and pushed to repositories (whether public or private), they become accessible to anyone with repository access and remain in the commit history until someone removes them.
According to GitHub, over 39 million secrets were leaked in GitHub repositories in 2024. Bad actors automatically scan public code repositories, looking for leaked secrets, and use these secrets to gain access to systems.
Process friction is when the ability to securely access a secret is so difficult, people find a way to bypass the process to retrieve the secret. Process friction harms operational excellence at organizations, effectively lowering the security posture.
The following are examples of process friction:
- Overly complex or lengthy approval processes.
- Lack of documentation or training on how to securely access a secret.
- Lack of centralized management services such as secrets management or single sign-on (SSO).
- Secret sprawl when teams manage secrets in several locations with no single source of truth.
Implementing systems to solve these problems can improve operational excellence and the security posture of organizations. Some ways to mitigate process friction include centralizing secret management systems like Vault, creating clear documentation and training for creating and accessing secrets, and reducing the risk of human error by automating the use of secrets through CI/CD systems.
Another cause of leaked secrets is insider threats. Insider threats are when an employee or ex-employee, either intentionally or unintentionally, leaks sensitive information.
An intentional insider threat is when someone purposely steals sensitive material, like an employee taking private encryption keys. An unintentional insider threat if someone who accidentally exposes sensitive information, such as pushing a secret to a public repository or is socially engineered to provide credentials to bad actors.
The chance of insider threats increases dramatically when there is process friction. Consider an environment that does not use single sign-on and requires users to have different logins for each system. When a user leaves the organization, an operations team has to shut down accounts in every system. If the account for one system is missed, that user is still able to access the system and associated data.
A new cause of leaked secrets is the improper use of AI or other data processing tools. AI increases developer productivity, but also increases the risk of leaked secrets. A developer may use an AI tool to generate code that requires a secret. The AI tool may include the secret in plaintext, and the developer intentionally pushes that secret to a code repository.
An AI tool might also either have bad intentions by using the code provided for nefarious purposes, or lacks security for server-side processing. Unless your organization self-hosts AI systems, you should never pass an AI system sensitive information. Or, if you must provide it with a key, do so in a development environment and rotate that key when you're done.
Always consult with the security team at your organization before using AI tools.

HashiCorp resources:

Learn to use Vault client libraries inside your application code to store and retrieve your first secret value. Centralized secrets management with HashiCorp Vault
HashiCorp Vault Enterprise secrets sync

External resources:

Impact on the organization

Along with technical implications of leaked secrets, there are also major business and professional implications. When organizations leak secrets, they often experience a loss of trust. Losing trust can lead to application downtime, and the loss of new and existing customers which may result in a loss of revenue.

From 2015 through 2024, the three major credit bureaus all suffered major data breaches. The breaches exposed the personal information of millions of people. These breaches caused a loss of trust in the credit bureaus, but it also led to the United States government passing the Economic Growth, Regulatory Relief, and Consumer Protection Act. This bill requires the credit bureaus to provide free credit freezes and fraud alerts to consumers, among other requirements. Credit freezes had been a source of revenue for the credit bureaus.

A popular cryptocurrency exchange suffered a data breach stemming from insider threats. The company expects to pay up to $400 million to make up for the loss of customer funds.

In addition to the loss of revenue, individuals have been fired, fined, and even had lawsuits brought against them, resulting in prison sentences for attempting to cover up data breaches.

External resources:

Identify leaked secrets

Organizations should use a defense-in-depth approach. Defense-in-depth is a security strategy that uses multiple layers of security and controls to protect data rather than relying on any single protective measure.

To start building out your defense-in-depth strategy for leaked secrets, you should start with a solution to scan for and identify secrets in various data sources. Common types of data sources you should monitor include:

Version control systems (VCS) such as BitBucket, GitHub, or GitLab.
Documentation and communication systems such as Confluence, Google Docs, or Slack.
General storage locations like local file systems, or cloud storage such as Amazon S3.
Local developer environments, or CI/CD systems and supporting services such as Docker.

HCP Vault Radar scans your data sources for secrets and integrates with various providers to alert your teams when they find a secret. Vault Radar supports multiple scanning options to support your workloads and data confidentiality requirements. You can use the cloud scanner for publicly accessible systems, (such as GitHub) or the Vault Radar agent for on-premises systems (like self-hosted Confluence). Alternatively, you can use the Vault Radar CLI to integrate with local developer environments or CI/CD systems.

Vault Radar also prioritizes the severity of the leaked secret. For example, it can identify if the secret is a placeholder or example text versus a real secret. Radar helps your security team prioritize the remediation of the leaked secret.

However, no amount of technical scanning and detection capabilities will be effective if your team members are afraid to report mistakes or potential security incidents. Creating a positive, blame-free culture is essential — when people fear losing their jobs for admitting errors, they will hide their mistakes rather than focus on fixing the underlying processes that allowed the incident to occur.

The Google SRE book advises removing blame from the incident response process. This includes a "focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior."

HashiCorp resources:

Prevent leaked secrets

There is no single solution to prevent leaked secrets. While every organization is different, there are common practices you can apply to your organization.

This section outlines essential strategies to safeguard your secrets:

Separation of duties: Distribute security responsibilities across teams
Restrict access to secrets: Implement least-privilege access controls
Dynamic credentials: Use short-lived, automatically rotating credentials
Single sign-on (SSO): Centralize authentication across systems
Centralized secret management: Store and retrieve secrets from an secrets management platform
Secure remote access: Apply zero-trust principles to network access

Separation of duties

Separation of duties (also known as segregation of duties) is a security practice that reduces the risk of a data breach by ensuring no single person has control over all aspects of a workflow. Separation of duties may be required by government or industry regulations, for example, Sarbanes-Oxley (SOX) in the United States.

A common implementation of separation of duties is to have separate teams responsible for software development, testing, building, and releasing the software to production. This separation ensures no single person can introduce security issues into the workflow. For example, if a developer attempts to introduce a bug to steal fractions of a penny from every transaction, stages in the software development lifecycle would discover the bug.

Restrict access to secrets

When managing secrets, a good practice is to restrict access to secrets, even from the security team, while allowing them to manage and support the secrets management platform. This separation ensures that the security team does not have access to all stored secrets.

Zero trust is a security model that assumes that all users, devices, and networks are untrusted until proven otherwise. Consider remote access solutions — conventional remote access solutions such as virtual private networks (VPNs) typically allow access to the entire network, once authenticated. This potentially exposes systems and data to unauthorized access.

When using a zero trust model for remote access, users and devices have access to the specific resources required. This helps to reduce the risk of a data breach because access to resources is restricted.

HashiCorp resources:

External resources:

Dynamic credentials

Dynamic credentials are short-lived credentials generated on demand and valid for a limited time. Dynamic credentials are useful in scenarios where SSO integration is not available. Because these credentials are valid for a limited time, the risk of exposing the secret is reduced.

Workload identity federation (WIF) creates a trust between systems. Once you complete the WIF configuration, a service like Terraform or Vault creates dynamic credentials in the target platform. Compare this to conventional methods where you might provision a long lived service account with a key or password. These accounts still need to managed, and rotated. With WIF, there is a trust between the systems so there is no risk to the service account secret being leaked. The service that provisions the dynamic credential revokes it when the workflow is complete or TTL expires.

Where SSO and WIF diverge is that WIF uses the trust relationship to provision temporary, dynamic credentials. These credentials have a short TTL, so there is a limited risk if they become exposed. Credentials are managed by the configured services, so there is no need to revoke, renew, or replace the credentials.

HashiCorp Vault generates dynamic credentials for a variety of systems, including AWS, Azure, GCP, HCP Terraform, and several database platforms.

Vault Enterprise supports WIF integration with AWS, Azure, and GCP. Using WIF, you can configure the cloud provider auth method, or secrets engine without static service accounts.

You can configure your application to request dynamic credentials from Vault before connecting to a cloud service provider, or backend database. You can also use the HCP Terraform secrets engine to generate dynamic credentials for different Terraform providers. This allows Terraform to provision the defined resources without needing to store static credentials.

HashiCorp resources:

Use single sign-on

You can use single sign-on to prevent leaking secrets. Single sign-on (SSO) is a process that allows users or workloads to authenticate to different services and applications using a single set of credentials. Common identity providers used to configure SSO include Microsoft Active Directory and Okta.

Having different logins for different systems is a common source of process friction. People, teams, and workloads often need to use different authentication workflows. This process friction causes frustration and often leads people to try to circumvent security processes.

NIST SP 800-63 Digital identity guidelines and NIST SP 800-171 recommend using federated identity providers, and centralizing authentication to reduce security risks.

When reviewing tools and services to use for scanning for leaked secrets, ensure these services support SSO. This will limit friction in adopting the tools.

HashiCorp tools like HCP Vault Radar, HCP Terraform, Vault, and Boundary support SSO. SSO lets you adopt HashiCorp tools and integrate with your IdP. Your security team can access Vault Radar in the HCP Portal, and Vault with the same credentials.

For workloads that require access to Vault, there are several auth methods that allow authentication from trusted platforms such as AWS, Azure, GCP, and Kubernetes.

HashiCorp resources:

External resources:

Centralize secret management

Once you have a process in place to identify leaked secrets, you should verify whether the secret is in use and ensure that you are managing it effectively moving forward. These two steps are important to manage properly so you do not inadvertently rotate a secret used by another system and bring those services down.

When you identify a leaked secret, your solution should determine if the secret is in use and whether it's in your secret manager. Determining whether it's in use and where it's located helps in the next phase of the process — remediation. If the leaked secret is not stored in your secret manager, add the secret before you begin the remediation process.

Another important feature for centralizing secrets management is the ability to copy or sync secrets to other systems. The ability to sync secrets helps with workloads or platforms that may be isolated, or systems that you cannot update to support a centralized secrets management solution.

HashiCorp Vault is a centralized secrets management and security solution with several deployment options. If you are new to secrets management, you can start by storing secrets as key-value pairs, and update your workflows to read the secret directly from Vault. As your secrets management process matures, you can use short-lived, dynamic credentials which are valid for a specified period of time. This further limits the effect if you do leak a secret because Vault will automatically revoke it after the specified time, or you can manually revoke it and request new credentials.

You can prevent storing secrets in your code by securing your secrets in a centralized secrets management system such as Vault, and then referencing these secrets in your code. The following Python example shows how to use Vault client libraries inside your application code to store and retrieve secret values.

read_response = client.secrets.kv.read_secret_version(path='my-secret-password')
password = read_response['data']['data']['password']

HashiCorp resources:

External resources:

OWASP Secrets Management Cheat Sheet

Secure remote access

Securing remote access to your systems is an important part of a defense-in-depth strategy. Traditional remote access solutions such as VPNs often provide access to entire segments of the network, potentially exposing system to users and workloads that do not need to connect. This access is anti-pattern to modern security practices such as separation of duties, and zero trust.

Another challenge is with modern deployment practices. Workloads are often ephemeral, meaning they are created and destroyed as needed. Given the ephemeral nature of modern deployments, this adds challenges to managing remote access and authentication to these systems.

HashiCorp Boundary is a remote access solution that follows zero trust principles and supports dynamic host catalogs which is useful for ephemeral workloads. Boundary lets platform teams specify only the systems that a user requires access to instead of the entire network. This reduces the risk of a data breach and allows organizations to prove compliance relative to separation of duties.

Dynamic host catalog plugins for major cloud providers automate the discovery and configuration of Boundary. As workloads are provisioned and destroyed, the Boundary host catalog remains up to date with the current state of the environment.

Boundary also supports credential brokering and SSH certificate injection. This allows users to connect to systems they have been granted access to without having to pass credentials through the CLI or Boundary Desktop application. In addition to Boundary native credential management, you can also integrate Boundary with HashiCorp Vault. This integration allows you to take advantage of Vault dynamic credentials, limiting the total lifetime of the credential used for a given session.

HashiCorp resources:

Create a remediation plan

Once you identify a leaked secret and store it in your secrets manager, you need to plan for the secret remediation.

If you have not already done so, form an incident response team. This team should include technical representatives from the security, operations, and development teams, as well as business and legal representatives.

The remediation process will vary based on:

The type of system the leaked secret gives access to.
The type of data available in that system.
Whether you have to report the incident to a government or industry regulator.

In any scenario, part of your remediation process should include communication with your stakeholders, both internal (such as development teams) and external (such as customers), to ensure they stay informed of any actions they may need to take.

The National Institute of Standards and Technology (NIST) is a helpful resource for defining requirements for incident response. NIST SP 800-61: Computer Security Incident Handling Guide identifies four stages of incident response:

Preparation: Ensure you have the right tools and processes in place to identify, manage, and remediate incidents.
Detection and analysis: Identify the incident, and analyze the impact of the incident.
Containment, eradication, and recovery: Contain the incident, remove the cause of the incident, and recover from the incident.
Post-incident activity: Review the incident, and update your processes to prevent similar incidents in the future.

HashiCorp resources:

External resources:

Remediate leaked secrets

Now that you have set up HCP Vault Radar to identify leaked secrets and HashiCorp Vault to manage your secrets, you can begin to remediate the leaked secret.

Like the incident response plan, the remediation process will vary. One of the biggest factors revolves around the data source of the leaked secret.

Leaked secret in documentation

If you identify a leaked secret in documentation, the first step is to identify what teams, workflows, or applications depend on this secret. A common practice is to document requirements in a configuration management database (CMDB). Similarly, if you are using Terraform to manage your Vault policies, you can cross-reference the policies to determine what teams or applications are using the secret.

If the processes using the secret already retrieve the secret from Vault, you can:

Rotate or change the secret.
Remove the plaintext secret from the documentation.
Update the secret value in Vault.

Any process retrieving the secret will receive the updated secret value from Vault.

Leaked secrets in source code

When you identify a leaked secret in source code, the process is more involved and may require involving other teams.

If the application or workflow built from the source code requires access to this secret, you must update the source code to reference it in a secure manner.

There are several options to retrieve a secret from Vault and you may use one or more methods during the remediation process. Each method has unique advantages and disadvantages. You must weigh the impact of each option against any business or regulatory requirements.

For native integration, you need to refactor your application or service to integrate your application directly with Vault using the secrets engine for the type of secrets you need. This may be your best approach if you maintain consistent development practices across your application teams, such as the use of standardized frameworks like Spring Boot, or retrieve all application configurations from a configuration server.

Use the Vault Secrets Operator for Kubernetes workloads or Vault secret sync to copy the secrets from Vault to your platform (for example, AWS, Azure, or GCP) built-in secrets management solution. Once you can retrieve the secret from the built-in secrets management solution, update the source code to reference the secret as required by your platform.
Use the Vault Agent to write the secret to a secure location such as an environment variable. Then, update the source code to reference the environment variable. This option introduces the least amount of friction, and is a good first step to quickly remove the secret from your source code before refactoring the code to use the Vault Secrets Operator, Vault SDK or API.
Update your application using the Vault SDK, or your programming language's native API/secrets management library. This is the most secure option, but may require significant development effort. Refactoring a legacy application may not be possible. In that scenario, consider the other methods described.

Once you update the data source to retrieve the secret from Vault, you can rotate the secret and securely store it in Vault.

HashiCorp resources:

Next steps

Leaked secrets pose significant risks to organizations, impacting both technical operations and business reputation. By understanding how secrets are leaked, implementing defense-in-depth strategies, centralizing secrets management, and establishing robust remediation and incident response plans, you can reduce the likelihood and impact of secret exposure. Leveraging tools like HCP Vault Radar, HashiCorp Vault, and Boundary, along with best practices such as single sign-on and dynamic credentials, helps secure your environment and streamline operations. Continually review and update your processes to adapt to evolving threats and maintain a strong security posture.

Secure entities with X.509 certificates

Next Collection

Reference Architecture