Service Mesh Observability Overview

In order to take advantage of Connect's L7 observability features you will need to:

• Deploy sidecar proxies that are capable of emitting metrics with each of your services. We have first class support for Envoy.
• Define where your proxies should send metrics that they collect.
• Define the protocols for each of your services.
• Define the upstreams for each of your services.

If you are using Envoy as your sidecar proxy, you will need to enable gRPC on your client agents. To define the metrics destination and service protocol you may want to enable configuration entries and centralized service configuration.

Kubernetes

If you are using Kubernetes, the Helm chart can simplify much of the configuration needed to enable observability. See our Kubernetes observability docs for more information.

Metrics destination

For Envoy the metrics destination can be configured in the proxy configuration entry's config section.

kind = "proxy-defaults"
name = "global"
config {
   "envoy_dogstatsd_url": "udp://127.0.0.1:9125"
}

Find other possible metrics syncs in the Connect Envoy documentation.

Service protocol

You can specify the protocol for all service instances in the service-defaults configuration entry. You can also override the default protocol when defining and registering proxies in a service definition file. Refer to Expose Paths Configuration Reference for additional information.

By default, proxies only provide L4 metrics. Defining the protocol allows proxies to handle requests at the L7 protocol and emit L7 metrics. It also allows proxies to make per-request load balancing and routing decisions.

Service upstreams

You can set the upstream for each service using the proxy's upstreams sidecar parameter, which can be defined in a service's sidecar registration.