Implement circuit breaking in Consul service mesh with Envoy

When you have a set of services calling each other, a service failure or increased latency in one service can result in cascading failures to downstream dependencies. These cascading failures can exhaust application resources. One way to prevent this type of failure is the circuit breaker pattern. A circuit breaker quickly detects failures and returns an error for a fixed amount of time.

This tutorial covers:

• Using a service mesh to eject failed instances and prevent cascading failures
• Configuring Consul to implement circuit breaking

Circuit breaking is typically achieved through user code or network configuration. While some libraries and frameworks offer circuit breaking through code, if you have services written in a variety of languages you can easily introduce inconsistency across implementations. It may also be the case that adding additional application code is either not desirable or not possible. Implementing circuit breaking using the service mesh may be more desirable since it reduces application code and decouples infrastructure concerns from application logic.

To implement circuit breaking in Consul, you must configure two Envoy settings: one for the Envoy circuit breaking feature and another for Envoy outlier detection. Envoy circuit breaking implements the bulkhead pattern, which sets the maximum, pending, and concurrent connections for a pool of upstream service instances. Envoy outlier detection handles the ejection of services that are flagged by the circuit breaker. If an upstream service instance returns the maximum allowed consecutive HTTP 5xx errors, Envoy will eject the service instance from the pool of upstream service instances. The combination of these two settings effective implements the circuit breaker pattern, first by detecting failures and then by ejecting the failed service instance.

In this tutorial, you will start two services: api and web. The web service connects to the upstream api service using Consul service mesh which load balances between two versions of the api service. You will introduce an error into api-v2 that trips the circuit breaker, sets it to an open state, and sends requests directly to the working api-v1.

This tutorial focuses on configuring Consul for circuit breaking. The Consul deployment in this tutorial is not suitable for production environments as it includes Consul dev agents, internal proxies, and mock services.

Prerequisites

• Docker Compose
• Docker
• jq
• curl
• Consul 1.9.0+: The UI described here is only available in Consul version 1.9.0 and later.

Clone the GitHub repository that contains the files you'll use with this tutorial.

$ git clone https://github.com/hashicorp/learn-consul-service-mesh.git

Change directories into the repository you just cloned.

$ cd learn-consul-service-mesh

Checkout the tagged version verified for this tutorial.

$ git checkout tags/v0.0.1

This directory will be referred to as your working directory, and you will run the rest of the commands in this tutorial from this directory.

Configure services and defaults

Once you have set up the prerequisites, configure the Consul services to use Envoy's circuit breaker and outlier detection.

Open central_config/global-defaults.hcl. The web and api services default to the HTTP protocol. You can set the central configuration to this default. Consul propagates this configuration to Envoy.

$ cat central_config/global-defaults.hcl

Kind = "proxy-defaults"
Name = "global"

Config {
  envoy_prometheus_bind_addr = "0.0.0.0:9102"
  protocol = "http"
}

Open service_config/api-v1.hcl. This file configures the sidecar proxy for api-v1. A similar configuration exists for api-v2.

$ cat service_config/api-v1.hcl

service {
  name = "api"
  id = "api-v1"
  address = "10.5.0.4"
  port = 9092

  tags = ["v1"]
  meta = {
    version = "1"
  }

  connect {
    sidecar_service {
      port = 20000

      check {
        name     = "Connect Envoy Sidecar"
        tcp      = "10.5.0.4:20000"
        interval = "10s"
      }
    }
  }
}

Open service_config/web.hcl for the Consul configuration of the web service. The web service connects to the api service upstream. By default, Consul load balances requests round-robin between each version of the api service without additional configuration. In addition to defining upstream services, set the proxy.config.limits for the maximum connections, pending requests, and concurrent requests. The upstream api service represents a pool with api-v1 and api-v2. Configure the proxy.config.passive_health_check setting to trip after 10 api service failures (with the max_failures setting) and retry an ejected instance after thirty seconds (with the interval setting).

$ cat service_config/web.hcl

service {
  name    = "web"
  id      = "web"
  address = "10.5.0.3"
  port    = 9091

  tags = ["v1"]
  meta = {
    version = "1"
  }

  connect {
    sidecar_service {
      port = 20000

      check {
        name     = "Connect Envoy Sidecar"
        tcp      = "10.5.0.3:20000"
        interval = "10s"
      }

      proxy {
        upstreams {
          destination_name   = "api"
          local_bind_address = "127.0.0.1"
          local_bind_port    = 9092

          config {
            connect_timeout_ms = 5000
            limits {
              max_connections         = 3
              max_pending_requests    = 4
              max_concurrent_requests = 5
            }
            passive_health_check {
              interval     = "30s"
              max_failures = 10
            }
          }
        }
      }
    }
  }
}

You will simulate a failure in the api-v2 service instance. The api-v2 service instance will be ejected after 10 HTTP 5xx errors. Consul will check the service every 30 seconds to determine if api-v2 should rejoin the cluster.

Deploy Consul and the services

After configuring circuit breaking settings, deploy a Consul server, the web service, an api-v1 and an api-v2 service, and Prometheus. Prometheus aggregates metrics from services which can be viewed in the Consul UI.

Configure api-v1

Open docker-compose.yaml. This file deploys Consul, the web service and sidecar proxy, the api-v1 service and sidecar proxy, and Prometheus. The docker-compose.yaml uses volume mounts to add the central and service configurations to the containers.

$ cat docker-compose.yaml

version: "3.3"
services:

  consul:
    image: consul:1.9.0
    command: ["consul","agent","-config-file=/config/consul-single-dc.hcl","-config-dir=/config"]
    volumes:
      - "./consul_config:/config"
    ports:
      - 8500:8500
    networks:
      vpcbr:
        ipv4_address: 10.5.0.2

  # Define web service and envoy sidecar proxy
  web:
    image: nicholasjackson/fake-service:v0.19.1
    environment:
      LISTEN_ADDR: 0.0.0.0:9091
      UPSTREAM_URIS: "http://localhost:9092"
      MESSAGE: "Hello World"
      NAME: "web"
      SERVER_TYPE: "http"
    ports:
    - "9091:9091"
    networks:
      vpcbr:
        ipv4_address: 10.5.0.3
  web_proxy:
    image: joatmon08/consul-envoy:v1.9.0-v1.16.0
    restart: always
    environment:
      CONSUL_HTTP_ADDR: 10.5.0.2:8500
      CONSUL_GRPC_ADDR: 10.5.0.2:8502
      SERVICE_CONFIG: /config/web.hcl
      CENTRAL_CONFIG_DIR: /central_config
    volumes:
      - "./service_config:/config"
      - "./central_config:/central_config"
    command: ["consul", "connect", "envoy","-sidecar-for", "web"]
    network_mode: "service:web"

  # Define api v1 service and envoy sidecar proxy
  api_v1:
    image: nicholasjackson/fake-service:v0.19.1
    environment:
      LISTEN_ADDR: 0.0.0.0:9092
      MESSAGE: "Hello World"
      NAME: "api-v1"
      SERVER_TYPE: "http"
    networks:
      vpcbr:
        ipv4_address: 10.5.0.4
  api_v1_proxy:
    image: joatmon08/consul-envoy:v1.9.0-v1.16.0
    restart: always
    environment:
      CONSUL_HTTP_ADDR: 10.5.0.2:8500
      CONSUL_GRPC_ADDR: 10.5.0.2:8502
      SERVICE_CONFIG: /config/api-v1.hcl
      CENTRAL_CONFIG_DIR: /central_config
    volumes:
      - "./service_config:/config"
      - "./central_config:/central_config"
    command: ["consul", "connect", "envoy","-sidecar-for", "api-v1"]
    network_mode: "service:api_v1"

  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - "./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"
    networks:
      vpcbr:
        ipv4_address: 10.5.0.6

networks:
  vpcbr:
    driver: bridge
    ipam:
      config:
        - subnet: 10.5.0.0/16

Configure api-v2

Open docker-compose-success.yaml. This file configures api-v2 without introducing a failure into the service. Recall Consul will load balance requests between service instances that share a service name. In this case, Consul directs requests to the api service and load balancers between api-v1 and api-v2.

$ cat docker-compose-success.yaml

version: "3.3"
services:

  # Define api v2 service and envoy sidecar proxy
  # with error toggle disabled
  api_v2:
    image: nicholasjackson/fake-service:v0.19.1
    environment:
      LISTEN_ADDR: 0.0.0.0:9092
      MESSAGE: "Hello World"
      NAME: "api-v2"
      SERVER_TYPE: "http"
    networks:
      vpcbr:
        ipv4_address: 10.5.0.5
  api_v2_proxy:
    image: joatmon08/consul-envoy:v1.9.0-v1.16.0
    restart: always
    environment:
      CONSUL_HTTP_ADDR: 10.5.0.2:8500
      CONSUL_GRPC_ADDR: 10.5.0.2:8502
      SERVICE_CONFIG: /config/api-v2.hcl
      CENTRAL_CONFIG_DIR: /central_config
    volumes:
      - "./service_config:/config"
      - "./central_config:/central_config"
    command: ["consul", "connect", "envoy","-sidecar-for", "api-v2"]
    network_mode: "service:api_v2"

networks:
  vpcbr:
    external:
      name: learn-consul-service-mesh_vpcbr

Deploy the services

Deploy Consul, the web service, the api-v1 service, the api-v2 service, and Prometheus.

$ bash start.sh

Creating network "learn-consul-service-mesh_vpcbr" with driver "bridge"
Creating learn-consul-service-mesh_prometheus_1   ... done
Creating learn-consul-service-mesh_web_1        ... done
Creating learn-consul-service-mesh_api_v1_1     ... done
Creating learn-consul-service-mesh_consul_1       ... done
Creating learn-consul-service-mesh_web_proxy_1    ... done
Creating learn-consul-service-mesh_api_v1_proxy_1 ... done
WARNING: Found orphan containers (learn-consul-service-mesh_web_proxy_1, learn-consul-service-mesh_prometheus_1, learn-consul-service-mesh_api_v1_1, learn-consul-service-mesh_web_1, learn-consul-service-mesh_api_v1_proxy_1, learn-consul-service-mesh_consul_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating learn-consul-service-mesh_api_v2_1 ... done
Creating learn-consul-service-mesh_api_v2_proxy_1 ... done

Open the Consul UI in your browser at http://localhost:8500. The web and api services register as healthy.

Use curl to make an API request to the web service. It returns which version of the api service it called and the HTTP status code.

$ curl -s localhost:9091 | jq -r '.upstream_calls."http://localhost:9092" | "\(.name),\(.code)"'

api-v1,200

Running the request multiple times illustrates the load balancing of requests between api-v1 and api-v2.

$ curl -s localhost:9091 | jq -r '.upstream_calls."http://localhost:9092" | "\(.name),\(.code)"'

api-v2,200

Trip the circuit breaker

Deploy a failing instance of api-v2.

$ bash fail.sh

WARNING: Found orphan containers (learn-consul-service-mesh_api_v1_proxy_1, learn-consul-service-mesh_consul_1, learn-consul-service-mesh_api_v1_1, learn-consul-service-mesh_prometheus_1, learn-consul-service-mesh_web_1, learn-consul-service-mesh_web_proxy_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating learn-consul-service-mesh_api_v2_1 ... done
Recreating learn-consul-service-mesh_api_v2_proxy_1 ... done

Perform a GET request to the web service a few times. Any time the request gets forwarded to api-v1, the status code registers an HTTP 200 (success).

$ curl -s localhost:9091 | jq -r '.upstream_calls."http://localhost:9092" | "\(.name),\(.code)"'

api-v1,200

However, requests forwarded to api-v2 return an HTTP 500 error.

$ curl -s localhost:9091 | jq -r '.upstream_calls."http://localhost:9092" | "\(.name),\(.code)"'

api-v2,500

Recall that you configured Envoy's outlier detection to set the circuit breaker to open after 10 failed attempts to connect to api-v2. Trip the circuit breaker by sending 1000 API requests to the web service. Any time the web service calls api-v2, it receives an HTTP 500 error. Eventually, Envoy will flag api-v2 as an outlier and eject it from the api cluster. Any additional requests to api-v2 causes the proxy to immediately throw an HTTP 503 error (indicated by null,503) and direct all future requests to api-v1.

$ bash request.sh

api-v2,500
api-v2,500
api-v1,200
api-v2,500
api-v1,200
api-v1,200
api-v2,500
api-v2,500
null,503
api-v1,200
api-v1,200
api-v1,200
api-v2,500
api-v1,200
api-v1,200
api-v1,200
api-v1,200
api-v1,200
api-v1,200
api-v1,200

...TRUNCATED...

Verify ejection of failing service instance

To verify that outlier detection ejected api-v2, open this link in your browser. This link opens a Prometheus graph to the envoy_cluster_outlier_detection_ejections_active metric.

Click the "Execute" button to refresh the graph.

This gauge illustrates how Envoy manages the ejection of api-v2. The first time api-v2 throws an error, Envoy ejects it for 30 seconds, which is the interval setting. When Envoy retries api-v2, Envoy receives an error and ejects api-v2 a second time. This time api-v2 will be ejected for 60 seconds because it has been ejected twice. As api-v2 continues to return an error, the base ejection time of 30 seconds gets multiplied by the number of times the host has been ejected. The active ejection period becomes longer as the service continues to error.

Check availability of web service

Recall that a circuit breaker prevents cascading failures to downstream services, such as the web service. The web service should remain available since api-v1 successfully completes requests and the circuit breaker is open. Periodically, the proxy will determine if it should bring api-v2 back into the pool of services instances. Given api-v2 will continue to return an error, the web service should register a failure quickly and its next request will be to api-v1.

Open the Consul UI in your browser to view the api service's topology.

The graph includes the error rates of the web service and api service. Periodically, the api service returns an error but the overall error rate of the web service remains less than 1%. When the circuit breaker opens, it prevents cascading errors to the web service.

Clean up

Once you are done with this tutorial, you can stop the requests to the web service.

$ bash request.sh

...TRUNCATED...
^C

Delete the containers for Consul, Prometheus, web, api-v1, and api-v2.

$ bash cleanup.sh

Stopping learn-consul-service-mesh_web_proxy_1    ... done
Stopping learn-consul-service-mesh_api_v1_proxy_1 ... done
Stopping learn-consul-service-mesh_consul_1       ... done
Stopping learn-consul-service-mesh_prometheus_1   ... done
Stopping learn-consul-service-mesh_web_1          ... done
Stopping learn-consul-service-mesh_api_v1_1       ... done
Removing orphan container "learn-consul-service-mesh_api_v2_1"
Removing orphan container "learn-consul-service-mesh_api_v2_proxy_1"
Removing learn-consul-service-mesh_web_proxy_1    ... done
Removing learn-consul-service-mesh_api_v1_proxy_1 ... done
Removing learn-consul-service-mesh_consul_1       ... done
Removing learn-consul-service-mesh_prometheus_1   ... done
Removing learn-consul-service-mesh_web_1          ... done
Removing learn-consul-service-mesh_api_v1_1       ... done
Removing network learn-consul-service-mesh_vpcbr

Next steps

Now that you have completed this tutorial you have familiarized yourself with how to manually configure Consul and Envoy for circuit breaking. You simulated a failing upstream service and tripped the circuit breaker to eject the failing service from the pool.

To learn how to configure Consul health checks without Envoy, complete the tutorial to Ensure Only Healthy Services are Discoverable.

In the next tutorial, Load Balancing Services in Consul Service Mesh with Envoy you will learn how to change the load balancing policy for a service in your mesh.