Consul key/value (KV) store overview

Consul KV is a core feature of Consul and is installed with the Consul agent. Once installed with the agent, Consul KV has reasonable defaults. Consul KV lets you store indexed objects, though its main uses are storing configuration parameters and metadata. It is a basic KV store and is not intended to be a full featured datastore (such as DynamoDB).

The Consul KV datastore is located on the servers, but any client or server agent may access it. The natively integrated RPC functionality lets clients forward requests to servers, including key/value reads and writes. Part of Consul's core design allows automatic data replication across all the servers. Having a quorum of servers decreases the risk of data loss if an outage occurs.

If you have not used Consul KV, complete this Getting Started tutorial on HashiCorp.

Accessing the KV store

Access the KV store with the consul kv CLI subcommands, HTTP API and Consul UI. To restrict access, enable and configure ACLs. Once the ACL system has been bootstrapped, users and services need a valid token with KV privileges to access the data store. This includes read-only access. We recommend creating a token with limited privileges. For example, you could create a token with write privileges on one key for developers to update the value related to their application.

The datastore itself is located on the Consul servers in the data directory. To ensure data is not lost in the event of a complete outage, use the consul snapshot feature to backup the data.

Using Consul KV

Objects are opaque to Consul, meaning there are no restrictions on the type of object stored in a key/value entry. The main restriction on an object is a maximum size of 512 KB. Due to the maximum object size and main use cases, you should not need extra storage. The general sizing recommendations are usually sufficient.

Keys, like objects, are not restricted by type and can include any character. However, we recommend using URL-safe chars such as [a-zA-Z0-9-._~] with the exception of /, which can be used to help organize data. Note, / is treated like any other character and is not fixed to the file system. This means that including / in a key does not fix it to a directory structure. This model is similar to Amazon S3 buckets. However, / is still useful for organizing data and when recursively searching within the data store. We also recommend that you avoid the use of *, ?, ', and % because they can cause issues when using the API and in shell scripts.

Using Sentinel to apply policies for Consul KV

You can also use Sentinel as a Policy-as-code framework for defining advanced key-value storage access control policies. Sentinel policies extend the ACL system in Consul beyond static "read", "write", and "deny" policies to support full conditional logic and integration with external systems. Reference the Sentinel documentation for high-level Sentinel concepts.

To get started with Sentinel in Consul, refer to the Sentinel documentation or Consul documentation.

Extending Consul KV

Consul Template

If you plan to use Consul KV as part of your configuration management process, review the Consul Template tutorial on how to update configuration based on value updates in the KV. Consul Template is based on Go Templates and allows for a series of scripted actions to be initiated on value changes to a Consul key.

Watches

Extend Consul KV with the use of Consul watches, which are a way to monitor data for updates. When an update is detected, an external handler is invoked. To use watches with the KV store, use the key watch type.

Refer to the Consul watches documentation for more information.

Consul Sessions

Use Consul sessions to build distributed locks with Consul KV. Sessions act as a binding layer between nodes, health checks, and key/value data. The KV API supports an acquire and release operation. The acquire operation acts like a Check-And-Set operation. On success, Consul updates the key, increments the LockIndex and then updates the session value to reflect the session holding the lock. Refer to the K/V integration documentation for more information.

Refer to the following tutorials to learn how to use Consul sessions:

• Application leader election to learn the process for building client-side leader elections for service instances using Consul's session mechanism and the Consul key/value store.
• Sessions and distributed locks overview to build distributed semaphores

Vault

If you plan to use Consul KV as a backend for Vault, refer to the Configure Vault cluster with Integrated Storage tutorial.