Service mesh troubleshoot overview

This topic provides an overview of Consul’s built-in service-to-service troubleshooting capabilities. When communication between an upstream service and a downstream service in a service mesh fails, you can run the consul troubleshoot command to initiate a series of automated validation tests.

For more information, refer to the consul troubleshoot CLI documentation or the consul-k8s troubleshoot CLI reference.

Introduction

When communication between upstream and downstream services in a service mesh fails, you can diagnose the cause manually with one or more of Consul’s built-in features, including health check queries, the UI topology view, and agent telemetry metrics.

The consul troubleshoot command performs several checks in sequence that enable you to discover issues that impede service-to-service communication. The process systematically queries the Envoy administration interface API and the Consul API to determine the cause of the communication failure.

The troubleshooting command validates service-to-service communication by checking for the following common issues:

• Upstream service does not exist
• One or both hosts are unhealthy
• A filter affects the upstream service
• The CA has expired mTLS certificates
• The services have expired mTLS certificates

Consul outputs the results of these validation checks to the terminal along with suggested actions to resolve the service communication failure. When it detects rejected configurations or connection failures, Consul also outputs Envoy metrics for services.

Envoy proxies in a service mesh

Consul validates communication in a service mesh by checking the Envoy proxies that are deployed as sidecars for the upstream and downstream services. As a result, troubleshooting requires that Consul’s service mesh features are enabled.

For more information about using Envoy proxies with Consul, refer to Envoy proxy configuration for service mesh.

Technical constraints

When troubleshooting service-to-service communication issues, be aware of the following constraints:

• The troubleshooting tool does not check service intentions. For more information about intentions, including precedence and match order, refer to service mesh intentions.
• The troubleshooting tool validates one direct connection between a downstream service and an upstream service. You must run the consul troubleshoot command with the Envoy ID for an individual upstream service. It does support validating multiple connections simultaneously.
• The troubleshooting tool only validates Envoy configurations for sidecar proxies. As a result, the troubleshooting tool does not validate Envoy configurations on upstream proxies such as mesh gateways and terminating gateways.