HashiConf 2024 Now streaming live from Boston! Attend for free
Consul
Consul Home

You are viewing documentation for version v1.14.x. View latest version.

Consul Connect Envoy

Command: consul connect envoy

The connect Envoy command is used to generate a bootstrap configuration for Envoy proxy for use with Consul Connect.

Refer to the examples for guidance on common use cases, such as launching a service instance's sidecar proxy when ACLs are enabled.

The default behavior is to generate the necessary bootstrap configuration for Envoy based on the environment variables and options provided and by talking to the local Consul agent. It execs an external Envoy binary with that configuration leaving the Envoy process running in the foreground. An error is returned on operating systems other than linux or macOS since Envoy does not build for other platforms currently.

If the -bootstrap option is specified, the bootstrap config is generated in the same way and then printed to stdout. This allows it to be redirected to a file and used with envoy --config-path bootstrap.json. This works on all operating systems allowing configuration to be generated on a host that Envoy doesn't build on but then used in a virtualized environment that can run Envoy.

Usage

Usage: consul connect envoy [options] [-- pass-through options]

Envoy Options for both Sidecars and Gateways

  • -proxy-id - The proxy service ID. This service ID must already be registered with the local agent unless a gateway is being registered with the -register flag. As of Consul 1.8.0, this can also be specified via the CONNECT_PROXY_ID environment variable.

  • -envoy-binary - The full path to a specific Envoy binary to exec. By default the current $PATH is searched for envoy.

  • -admin-bind - The host:port to bind Envoy's admin HTTP API. Default is localhost:19000. Envoy requires that this be enabled. The host part must be resolvable DNS name or IP address.

  • -bootstrap - If present, the command will simply output the generated bootstrap config to stdout in JSON protobuf form. This can be directed to a file and used to start Envoy with envoy --config-path bootstrap.json.

    Security Note: If ACLs are enabled the bootstrap JSON will contain the ACL token from -token or the environment and so should be handled as a secret. This token grants the identity of any service it has service:write permission for and so can be used to access any upstream service that that service is allowed to access by Connect intentions.

  • -envoy-version - The version of envoy that is being started. Default is 1.23.1. This is required so that the correct configuration can be generated.

  • -grpc-ca-file - Path to a CA file to use for TLS when communicating with the Consul agent through xDS. This can also be specified via the CONSUL_GRPC_CACERT environment variable.

  • -grpc-ca-path - Path to a directory of CA certificates to use for TLS when communicating with the Consul agent through xDS. This can also be specified via the CONSUL_GRPC_CAPATH environment variable.

  • -no-central-config - By default the proxy's bootstrap configuration can be customized centrally. This requires that the command run on the same agent as the proxy will and that the agent is reachable when the command is run. In cases where either assumption is violated this flag will prevent the command attempting to resolve config from the local agent.

  • -envoy-ready-bind-address - By default the proxy does not have a readiness probe configured on it. This flag in conjunction with the envoy-ready-bind-port flag configures where the envoy readiness probe is configured on the proxy. A /ready HTTP endpoint will be instantiated at the specified address and port. When configured, Consul uses /ready HTTP endpoints to check proxy health.

  • -envoy-ready-bind-port - By default the proxy does not have a readiness probe configured on it. This flag in conjunction with the envoy-ready-bind-address flag configures where the envoy readiness probe is configured on the proxy. A /ready HTTP endpoint will be instantiated at the specified address and port. When configured, Consul uses /ready HTTP endpoints to check proxy health.

  • -prometheus-backend-port - Sets the backend port for the "prometheus_backend" cluster that envoy_prometheus_bind_addr will point to. Without this flag, envoy_prometheus_bind_addr would point to the "self_admin" cluster where Envoy metrics are exposed. The metrics merging feature in consul-k8s uses this to point to the merged metrics endpoint combining Envoy and service metrics. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -prometheus-scrape-path - Sets the path where Envoy will expose metrics on the envoy_prometheus_bind_addr listener. Default is /metrics. For example, if envoy_prometheus_bind_addr is 0.0.0.0:20200, and this flag is set to /scrape-metrics, prometheus metrics would be scrapable at 0.0.0.0:20200/scrape-metrics. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -prometheus-ca-file - Path to a CA file for Envoy to use when serving TLS on the Prometheus metrics endpoint. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -prometheus-ca-path - Path to a directory of CA certificates for Envoy to use when serving the Prometheus metrics endpoint. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -prometheus-cert-file - Path to a certificate file for Envoy to use when serving TLS on the Prometheus metrics endpoint. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -prometheus-key-file - Path to a private key file for Envoy to use when serving TLS on the Prometheus metrics endpoint. Only applicable when envoy_prometheus_bind_addr is set in proxy config.

  • -- [pass-through options] - Any options given after a double dash are passed directly through to the envoy invocation. See Envoy's documentation for more details. The command always specifies --config-file and --v2-config-only and by default passes --disable-hot-restart see hot restart.

  • -enable-config-gen-logging - This flag enables debug message logging when generating the Envoy bootstrap configuration to help troubleshoot issues. Logs output to stderr. For more information about generating the Envoy bootstrap configuration, refer to Envoy proxy configuration.

Envoy Sidecar Proxy Options

  • -sidecar-for - The ID (not name if they differ) of the service instance this proxy will represent. The target service doesn't need to exist on the local agent yet but a sidecar proxy registration with proxy.destination_service_id equal to the passed value must be present. If multiple proxy registrations targeting the same local service instance are present the command will error and -proxy-id should be used instead. As of Consul 1.8.0, this can also be specified via the CONNECT_SIDECAR_FOR environment variable.

    Note: If ACLs are enabled, a token granting service:write for the target service (configured in proxy.destination_service_name) must be passed using the -token option or CONSUL_HTTP_TOKEN environment variable. This token authorizes the proxy to obtain TLS certificates representing the target service.

Envoy Gateway Options

  • -gateway - Flag to indicate that Envoy should be configured as a Gateway. Must be one of: terminating, ingress, or mesh. If multiple gateways are managed by the same local agent then -proxy-id should be used as well to specify the instance this represents.

  • -register - Indicates that the gateway service should be registered with the local agent instead of expecting it to already exist. This flag is unused for traditional sidecar proxies.

  • -address - The address to advertise for services within the local datacenter to use to reach the gateway instance. This flag is used in combination with -register. This takes the form of <ip address>:<port> but also supports go-sockaddr templates.

    If Envoy is configured as a terminating or mesh gateway, traffic from services within the mesh will be received at the specified IP and port.

    If Envoy is configured as an ingress gateway, a /ready HTTP endpoint will be instantiated at the specified IP and port. Consul uses /ready HTTP endpoints to check gateway health. The specified IP will also be used by the ingress gateway when instantiating user-defined listeners configured in the ingress gateway configuration entry.

    Note: Ensure that user-defined ingress gateway listeners use a different port than the port specified in -address so that they do not conflict with the health check endpoint.

  • -admin-access-log-path The path to write the access log for the administration server. If no access log is desired specify /dev/null. By default it will use /dev/null.

  • -bind-address - The bind address to use instead of the default binding rules. Specified as <name>=<ip>:<port> pairs. This flag may be used multiple times to add multiple bind addresses.

  • -expose-servers - Expose the servers for WAN federation via this mesh gateway.

  • -wan-address - The address to advertise for services within remote datacenters to use to reach the gateway instance. This flag is used in combination with -register. This takes the form of <ip address>:<port> but also supports go-sockaddr templates.

  • -service - The name of the gateway service to register. This flag is used in combination with -register.

  • -deregister-after-critical - The amount of time the gateway services health check can be failing before being deregistered. This flag is used in combination with -register

Enterprise Options

  • -partition=<string> - Specifies the partition to query. If not provided, the partition will be inferred from the request's ACL token, or will default to the default partition. Partitions are a Consul Enterprise feature added in v1.11.0.
  • -namespace=<string> - Specifies the namespace to query. If not provided, the namespace will be inferred from the request's ACL token, or will default to the default namespace. Namespaces are a Consul Enterprise feature added in v1.7.0.

API Options

The standard API options are used to connect to the local agent to discover the proxy configuration needed.

  • -grpc-addr=<addr> - Address of the Consul agent with grpc port. This can be an IP address or DNS address, but it must include the port. This can also be specified via the CONSUL_GRPC_ADDR environment variable. In Consul 1.3 and later, the default value is 127.0.0.1:8502, and https can optionally be used instead. The scheme can also be set to HTTPS by setting the environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket using unix:///path/to/socket if the agent is configured to listen that way.

    Note: gRPC uses the same TLS settings as the HTTPS API. If HTTPS is enabled then gRPC will require HTTPS as well.

    • -ca-file=<value> - Path to a CA file to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CACERT environment variable.

    • -ca-path=<value> - Path to a directory of CA certificates to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CAPATH environment variable.

    • -client-cert=<value> - Path to a client cert file to use for TLS when verify_incoming is enabled. This can also be specified via the CONSUL_CLIENT_CERT environment variable.

    • -client-key=<value> - Path to a client key file to use for TLS when verify_incoming is enabled. This can also be specified via the CONSUL_CLIENT_KEY environment variable.

    • -http-addr=<addr> - Address of the Consul agent with the port. This can be an IP address or DNS address, but it must include the port. This can also be specified via the CONSUL_HTTP_ADDR environment variable. In Consul 0.8 and later, the default value is http://127.0.0.1:8500, and https can optionally be used instead. The scheme can also be set to HTTPS by setting the environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket using unix:///path/to/socket if the agent is configured to listen that way.

    • -tls-server-name=<value> - The server name to use as the SNI host when connecting via TLS. This can also be specified via the CONSUL_TLS_SERVER_NAME environment variable.

    • -token=<value> - ACL token to use in the request. This can also be specified via the CONSUL_HTTP_TOKEN environment variable. If unspecified, the query will default to the token of the Consul agent at the HTTP address.

    • -token-file=<value> - File containing the ACL token to use in the request instead of one specified via the -token argument or CONSUL_HTTP_TOKEN environment variable. This can also be specified via the CONSUL_HTTP_TOKEN_FILE environment variable.

Examples

Assume a local service instance is registered on the local agent with a sidecar proxy (using the sidecar service registration helper) as below.

service {
  name = "web"
  port = 8080
  connect { sidecar_service {} }
}

Basic sidecar proxy

The sidecar Envoy process can be started with.

$ consul connect envoy -sidecar-for web

This example assumes that the correct environment variables are used to set the local agent connection information and ACL token, or that the agent is using all-default configuration.

Sidecar proxy with ACLs enabled

In secure deployments, Consul's ACL system is enabled with a default deny policy. To access Consul API resources, an API request must present a Consul ACL token with the necessary privileges. If ACLs are enabled, you must provide an ACL token to the sidecar proxy. The token must grant the proxy privileges to register itself and the service it fronts and to access all potential upstreams of that service. We recommend using a service identity to directly create a token with the privileges necessary for a service and its sidecar proxy, rather than creating a unique ACL policy for every service.

After creating a token using a service identity, provide the token when launching the Envoy sidecar proxy instance with the consul connect envoy command. You can provide the token through an environment variable or CLI flag.

Environment variables:

CLI flags:

$ export CONSUL_HTTP_TOKEN="<secret-id-of-token-with-service-identity-for-web>"
$ consul connect envoy -sidecar-for web

Additional Envoy Arguments

To pass additional arguments directly to Envoy, for example output logging level, you can use:

$ consul connect envoy -sidecar-for web -- --log-level debug

Multiple Proxy Instances

To run multiple different proxy instances on the same host, you will need to use -admin-bind on all but one to ensure they don't attempt to bind to the same port as in the following example.

$ consul connect envoy -sidecar-for db -admin-bind localhost:19001

Mesh Gateways

The mesh gateway Envoy process can be auto-registered and started with the following command.

$ consul connect envoy -gateway=mesh -register \
  -address '{{ GetInterfaceIP "eth0" }}:8443' \
  -wan-address '{{ GetInterfaceIP "eth1" }}:8443'

Terminating Gateways

The terminating gateway Envoy process can be auto-registered and started with the following command.

$ consul connect envoy -gateway=terminating -register -service my-gateway \
  -address '{{ GetInterfaceIP "eth0" }}:8443'

Ingress Gateways

The ingress gateway Envoy process can be auto-registered and started with the following command.

$ consul connect envoy -gateway=ingress -register -service ingress-service \
  -address '{{ GetInterfaceIP "eth0" }}:8888'

Exec Security Details

The command needs to pass the bootstrap config through to Envoy. Envoy currently only supports passing this as a file path or passing a whole string on the command line with --config-yaml. Since the bootstrap needs to contain the ACL token to authorize the proxy, this secret needs careful handling.

Passing a secret via command option is unacceptable as on many unix systems these are readable to any user on the host for example via /proc or via a setuid process like ps.

Creating a temporary file is more secure in that it can only be read by the current user but risks leaving secret material on disk for an unbounded length of time and in a location that is opaque to the operator.

To work around these issues, the command currently creates a temporary file and immediately unlinks it so it can't be read by any other process that doesn't already have the file descriptor. It then writes the bootstrap JSON, and unsets the CLOEXEC bit on the file handle so that it remains available to the Envoy process after exec. Finally it execs Envoy with --config-file /dev/fd/X where X is the the file descriptor number of the temp file.

This ensures that Envoy can read the file without any other normal user process being able to (assuming they don't have privileged access to /proc). Once the Envoy process stops, there is no longer any reference to the file to clean up.

Envoy Hot Restart

Envoy supports hot restart which requires simple external coordination. By default, this command will add --disable-hot-restart when it runs Envoy.

The reason for this default behavior is to make it easy to test and run local demonstrations with multiple Envoy instances outside of cgroups or network namespaces.

To use hot restart, Envoy needs to be started with either the --restart-epoch option. If this command detects that option in the pass-through flags it will not add --disable-hot-restart allowing hot restart to work normally.

The only difference to note over running Envoy directly is that --restart-epoch must be explicitly set to 0 for the initial launch of the Envoy instance to avoid disabling hot restart entirely. The official hot-restarter.py always sets this option so should work as recommended.