Best Practices for Service Mesh Security

Consul service mesh enables secure service-to-service communication over mutual TLS. This provides both in-transit data encryption as well as authorization. This page will document how to secure the service mesh. To try service mesh locally, complete the Getting Started guide or for a full security model reference, see the dedicated Consul security model page. When setting up service mesh in production, review this tutorial.

Consul service mesh will function in any Consul configuration. However, unless the checklist below is satisfied, the service mesh is not providing the security guarantees it was built for. The checklist below can be incrementally adopted towards full security if you prefer to operate in less secure models initially.

Checklist

Request Normalization Configured for L7 Intentions

Atypical traffic patterns may interfere with the enforcement of L7 intentions. For example, if a service makes request to a non-normalized URI path and Consul is not configured to force path normalization, it becomes possible to circumvent path match rules. While a default deny policy can limit the impact of this issue, we still recommend that you review your current request normalization configuration. Normalization is critical to avoid unwanted traffic, especially when using unrecommended security options such as a default allow intentions policy.

Consul adopts a default normalization mode that adheres to RFC 3986, but additional options to enable stricter normalization are available in the cluster-wide Mesh configuration entry. We recommend reviewing these options and enabling the strictest set that does not interfere with application traffic.

We also recommend that you review L7 intention header match rules for potential issues with multiple header values. Refer to the service intentions configuration entry reference for more information.

You do not need to enable request normalization if you are not using L7 intentions. However, normalization may also benefit the use of other service mesh features that rely on L7 attribute matching, such as service routers.

ACLs Enabled with Default Deny

Consul must be configured to use ACLs with a default deny policy. This forces all requests to have explicit anonymous access or provide an ACL token. The configuration also forces all service-to-service communication to be explicitly allowed via an allow intention.

To learn how to enable ACLs, please see the tutorial on ACLs.

If ACLs are enabled but are in default allow mode, then services will be able to communicate by default. Additionally, if a proper anonymous token is not configured, this may allow anyone to edit intentions. We do not recommend this. If ACLs are not enabled, deny intentions will still be enforced, but anyone may edit intentions. This renders the security of the created intentions effectively useless.

The advantage of a default deny policy in combination with specific "allow" rules is that at worst, a failure of intentions due to misconfiguration will result in denied traffic, rather than unwanted allowed traffic.

TCP and UDP Encryption Enabled

TCP and UDP encryption must be enabled to prevent plaintext communication between Consul agents. At a minimum, verify_outgoing should be enabled to verify server authenticity with each server having a unique TLS certificate. verify_incoming provides additional agent verification, but doesn't directly affect service mesh since requests must also always contain a valid ACL token. Clients calling Consul APIs should be forced over encrypted connections.

See the Consul agent encryption page to learn more about configuring agent encryption.

If encryption is not enabled, a malicious actor can sniff network traffic or perform a man-in-the-middle attack to steal ACL tokens, always authorize connections, etc.

Prevent Unauthorized Access to the Config and Data Directories

The configuration and data directories of the Consul agent on both clients and servers should be protected from unauthorized access. This protection must be done outside of Consul via access control systems provided by your target operating system.

The full Consul security model explains the risk of unauthorized access for both client agents and server agents. In general, the blast radius of unauthorized access for client agent directories is much smaller than servers. However, both must be protected against unauthorized access.

Prevent Non-Mesh Traffic to Services

For services that are using proxies (are not natively integrated), network access via their unencrypted listeners must be restricted to only the proxy. This requires at a minimum restricting the listener to bind to loopback only. More complex solutions may involve using network namespacing techniques provided by the underlying operating system.

For scenarios where multiple services are running on the same machine without isolation, these services must all be trusted. We call this the trusted multi-tenancy deployment model. Any service could theoretically connect to any other service via the loopback listener, bypassing the service mesh completely. In this scenario, all services must be trusted or isolation mechanisms must be used.

For developer or operator access to a service, we recommend using a local service mesh proxy. This is documented in the development and debugging guide.

If non-proxy traffic can communicate with the service, this traffic will not be encrypted or authorized via service mesh.

Restrict Access to Envoy's Administration Interface

Envoy exposes an unauthenticated administration interface that can be used to query and modify the proxy. This interface allows potentially sensitive information to be retrieved, such as:

• Envoy configuration
• TLS certificates
• List of upstream services and endpoints

We strongly advise only exposing the administration interface on a loopback address (default configuration) and restricting access to a subset of users.

If the administration interface is exposed externally, for example by specifying a routable -admin-bind address, it may be possible for a malicious actor to gain access to Envoy's configuration, or impact the service's availability within the cluster.