Identity and access management

Legacy initial configuration scope

This section describes how the product will be configured after installation for self-managed Consul Enterprise deployments. It may take several weeks to work through all of the recommendations from each section, but this should be the primary focus of the Consul Enterprise platform operators after the initial deployment of Consul Enterprise has been completed. Establishing these foundational configuration best practices is essential before you begin targeting consumer workflows to onboard. This will ensure solid footing for your current and future use cases.

• Configure Consul as code
• Configure initial ACL roles and policies
• Configure DNS forwarding
  • Local agent resolution
  • External resolution via DNS forwarding
  • Kubernetes DNS forwarding with CoreDNS
  • Forwarding destination for external resolution and Kubernetes configurations
  • DNS ACL permissions
  • DNS performance considerations
• Configure admin partition and namespaces
  • Admin partition
  • Namespaces
• Configure platform monitoring
  • Consul control plane monitoring
  • Raft monitoring
  • Consul data plane monitoring
  • Consul platform monitoring and alerts
• Setup backup, restore, and upgrades
  • Backup Consul
  • Restore Consul
  • Upgrade Consul
• Security operational procedures
  • Gossip key rotation
  • Certificate rotation

Configure Consul as code

When managing Consul Enterprise configuration entries or other configuration resources, make automation a foundational component of your approach from the beginning. There are several correct ways to manage your Consul configuration, and different organizations adopt different GitOps and configuration management practices. However, you should avoid making manual changes to your production Consul Enterprise environment to minimize edge cases and control the promotion of changes across your environments. Also, try not to mix different configuration styles whenever possible.

When managing Consul configuration as code, a Git repository serves as the sole mechanism for contributors and operators to modify Consul configuration through a pull request driven workflow. To establish a self service configuration workflow, you should block direct commits to the main branch linked to the production configuration and require approvals and code reviews prior to accepting pull requests. When using a code driven configuration workflow, it is important to restrict manual operator level configuration changes to reduce drift between the configuration defined as code and the state of the real world.

There are two approaches to configure service discovery within Consul Enterprise. Both approaches support all server platforms.

1. Terraform If you manage configuration with Terraform, it assumes control over the state of the configuration values and will overwrite any changes made when using a different configuration method. Standardizing on one approach simplifies tasks for your operators. Having a single central place to manage configuration makes it easier for operators to create new configurations or adjust existing ones. You can also leverage Sentinel policies to govern behaviors around any resources managed with the Consul Terraform provider. Terraform does not have coverage for every configuration option within Consul, but should be the primary choice when automating configuration resources.
2. API and cli The API has coverage for every configuration type. However, It is more challenging to build configuration workflows using the API because you need to incorporate logic around any dependencies, creation ordering, and the existing state of your configuration resources.

Terraform

We recommend standardizing on Terraform for maintaining your Consul Enterprise configuration resources. Some key advantages to leveraging Terraform are that it manages state across lifecycle changes, so it can track which resources have already been created and any dependencies between the resources between changes. Another advantage is that common patterns can be composed into repeatable modules so that you can collect commonly used configuration entries or security policy and easily replicate it when onboarding new teams into Consul Enterprise. Changing only the team specific inputs when referencing a common module allows you to create a consistent usage model across similar consumer varieties.

Governance From a governance standpoint, leveraging HCP Terraform allows you to define policy around any of the Consul configuration resources using the Sentinel policy(opens in new tab) language which allows you to provide autonomy to your consumers with a self-service workflow while incorporating guardrails around any policies you want to enforce. An example of this might be enforcing the ideal quantity of servers to prevent operators from inadvertently scaling up the cluster size without understanding the quorum implications. Another example could be enforcing the inclusion of a TTL greater than zero inside prepared query resources to ensure that prepared query service queries can be cached within your DNS infrastructure.

GitOps repo structure In the earlier stages of adoption, it is common for the central team operating Consul to use one monolithic configuration repo where the policies and roles, and configuration values are defined for all consumers and Partitions while allowing for collaboration and pull requests. As you continue to mature in your usage and adoption of Consul Enterprise, you might choose to split some of the Partition specific configuration entries into their own repositories and delegate more control to the consumer teams to define configuration at their level. An example of this might be pushing service export configuration to the team responsible for managing a specific Partition where services are exported from.

An example Git configuration monorepo hierarchy:

• Consul Datacenter
  • Git Repo 1
    • Default Partition - VM Server Deployment
      • Config Entries
      • Prepared Queries
      • Auth Binding Rules
      • Cross Partition Policies and Roles
        • Cluster Wide Operators
        • Cross Partition Admins
        • Cross Partition Read Only
        • Cluster Wide GitOps/TFC Write/Management Token
      • Partition Specific Policies and Roles - TF Sub-Module
        • Partition Z Admins
        • Partition Z Read Only
    • Partition Z..A - Kubernetes Cluster/VM App Team
      • Config Entries

API or CLI

If your organization hasn't adopted Terraform for managing application configuration, you can use the API or command line interface to automate Consul Enterprise configuration through various configuration management tools or GitOps pipeline driven automation workflows. The API or CLI offers great flexibility regarding the supported tooling, but orchestrating it might be more challenging since you'll need to track the state of configuration changes and dependencies within your chosen workflow, which can require additional code. However, one key advantage to using the API is its full coverage for all configuration types.

Configure admin partition and namespaces

Based on the architecture defined in the Installation Guide it’s important to understand the purpose of admin partitions and namespaces for segmenting consumer workloads and Consul gossip communications. As part of establishing your initial consumer workloads within Kubernetes and virtual machine instances, admin partitions will be used to create a logical separation between each networking boundary.

Configure admin partitions

You can use ACL tokens and permissions to manage admin partitions.

The following recommendations show you how to configure admin partitions for Kubernetes and VMs.

Kubernetes

The recommended architecture is to have each Kubernetes cluster operate as its own admin partition. As an example, if you have 10 Kubernetes clusters, you should have 10 distinct admin partitions, with each Kubernetes cluster as a unique admin partition.

You can define the admin partition in the Helm chart. The following configuration creates an admin partition named my-admin-partition.

global:
    adminPartitions:
        enabled: true
        name: "my-admin-partition"

Helm will create the admin partition in the non-default partition Kubernetes clusters during the installation process. Unlike with VM based environments, you do not need to manually create the admin partition first.

VM

Determining boundaries for admin partitions is more nuanced when working in non-Kubernetes environments. One of the key considerations, among others, is how gossip pools are used in Consul.

Each admin partition operates within its own unique gossip pool. Consul servers must belong to the default partition and participate in the gossip pools of every partition in the Consul datacenter.

A single gossip pool should not exceed 5000 Consul nodes. Every node within a single pool must be able to communicate over TCP and UDP with every other node in the same pool. If applications need to be network isolated from each other, you must place them within their own admin partition. This will place the services in a unique gossip pool.

You must create the admin partition using Terraform(opens in new tab) or the API(opens in new tab) before you can use them. You can define the admin partition in the Consul agent configuration:

partition = "my-admin-partition"

Service discovery with admin partitions

After you have created and deployed services to admin partitions, you must specify the admin partition in your DNS query to resolve services within a non-default admin partition. The following is the structure of the DNS query for admin partitions. Substitute service_name, namespace, and partition as applicable.

<service_name>.service.<namespace>.ns.<partition>.ap.consul

Configure namespaces

Consul namespaces(opens in new tab) provide a way to isolate and group resources within an admin partition within a Consul cluster. All resources are placed within a default namespace unless explicitly assigned to a non-default namespace.

One of the primary benefits of grouping resources into namespaces is to reduce service name collisions. For example, multiple applications can have a service named “database” and each will be resolved correctly within their respective namespaces.

ACL tokens and permissions are used to manage Namespaces.

Kubernetes

Consul Catalog Sync(opens in new tab) automatically syncs Kubernetes namespaces to Consul namespaces. Example, if you have a service in the xyz Kubernetes namespace, Consul catalog sync will automatically create a corresponding service in the xyz Consul namespace.

VMs

Before you can assign services to a namespace, you must first create the namespace with Terraform(opens in new tab) or the API(opens in new tab).

Once you have a namespace, specify the namespace in the service configuration file. The following example creates a service named my-service in the my-namespace namespace.

service {
    name = "my-service"
    partition = "my-partition"
    namespace = "my-namespace"
    ...}

Service discovery with namespaces

To resolve services within a non-default Consul Namespace, the DNS query must explicitly include the namespace name:

<service_name>.service.<namespace>.ns.consul

Namespace policy defaults

If you want to allow services to discover other services located within different namespaces inside a partition, each namespace should have a policy defaults attached to it. The following is an example policy default that enables this behavior.

partition "default" {
    namespace_prefix "" {
        service_prefix "" {
            policy = "read"
    }
        node_prefix "" {
            policy = "read"
        }
    }
}

Security operational procedures

It is important to review the below security recommendations to ensure they align with your organization's security policies and standards. We recommend performing certificate and encryption key rotation on a recurring schedule whereas the CA and leaf TTL configurations can be updated as part of the initial deployment.

Rotate gossip key

Consul uses a 256 bit symmetric gossip encryption key to protect Serf traffic between Consul client agents. To prevent the risk of exposure, you should periodically rotate this key(opens in new tab) based on your organization’s existing encryption key rotation standards.

Rotate ca certificate

A certificate rotation process is important to protect the integrity of traffic flowing between your Consul Enterprise servers and any client agents. Sensitive information that may be exchanged can include things like ACL tokens or routing information. If a certificate is ever compromised, having a rotation schedule ensures that any attacker is only able to intercept or tamper with traffic for as long as the certificate is valid for. We recommend that you leverage the Vault Enterprise PKI secrets engine to automate the rotation and distribution of your Consul Enterprise server certificates. Future HVD guides will include more prescriptive information around leveraging Vault with Consul Enterprise.

Certificate TTLs

When leveraging auto encrypt for client certificate distribution, you may want to consider adjusting the expiration times for the Root CA, as well as any leaf certificates to align with security practices defined within your organization. The shorter the lifetime of any leaf certificates or CA, the less risk that they can be used for any extended time period if compromised. The default values should work for most organizations, which is 10 years for the CA certificate, and 3 days for the leaf certificates.

$ curl http://127.0.0.1:8500/v1/connect/ca/configuration
{
    "Provider": "consul",
    "Config": {
        "LeafCertTTL": "72h",
        "IntermediateCertTTL": "8760h",
        "RootCertTTL": "87600h"
    },
    "ForceWithoutCrossSigning": false
}

Update the CA configuration.

# payload.json
{
    "Provider": "consul",
    "Config": {
        "LeafCertTTL": "72h",
        "IntermediateCertTTL": "8760h",
        "RootCertTTL": "87600h"
    },
    "ForceWithoutCrossSigning": false
}
 
$ curl --request PUT --data @payload.json http://127.0.0.1:8500/v1/connect/ca/configuration