Load balance with HAProxy

This guide describes how to use HAProxy's native integration to automatically configure the load balancer with service discovery data from Consul. This allows HAProxy to automatically scale its backend server pools by leveraging its server-template function and Consul's service discovery.

In this guide, you set up a basic HAProxy configuration based on HAProxy's server-template. This template generates the load balancer backend server pool configuration based on the available service instances registered in Consul service discovery.

Requirements

To complete this guide, you need previous experience with Consul and HAProxy. Additionally, you should have the following infrastructure configured:

• A Consul datacenter with the web UI enabled.
• At least three Consul client nodes, to register the web services and health checks referenced in this guide. enable_local_script_checks must be set to true on the client nodes.
• Standard web servers running on each node, listening on HTTP port 80.
• HAProxy of version 2.6+ (LTS) installed and co-located with a Consul client. Refer to the HAProxy documentation for the manuals specific to your HA Proxy version.

After you've completed the steps listed in this guide, your infrastructure should look like the following diagram.

Create the HAProxy configuration

Start with a default installation of HAProxy. Add the following configuration at the end of the /etc/haproxy/haproxy.cfg file.

defaults
   timeout connect 5s
   timeout client 1m
   timeout server 1m

frontend stats
   bind *:1936
   mode http
   stats uri /
   stats show-legends
   no log

frontend http_front
   bind *:80
   default_backend http_back

backend http_back
    balance roundrobin
    server-template mywebapp 1-10 _web._tcp.service.consul resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check

resolvers consul
    nameserver consul 127.0.0.1:8600
    accepted_payload_size 8192
    hold valid 5s

Frontend

frontend http_front
   bind *:80
   default_backend http_back

The frontend http_front stanza instructs HAProxy to listen for HTTP requests on TCP port 80 and to use the http_back server pool as the respective load balancer backend.

Backend

backend http_back
    balance roundrobin
    server-template mywebapp 1-10 _web._tcp.service.consul resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check

The backend http_back stanza includes balance type and server-template settings. Refer to the HAProxy Configuration Manual for explanations of these settings.

The balance type is round robin, which load balances across the available service in order.

The HAProxy server-template is what allows Consul service registrations to configure HAProxy's backend server pool. This means you do not need to explicitly add your backend servers' IP addresses. The configuration specifies a server-template named mywebapp. The template name is not tied to the service name, which is registered in Consul.

The server-template provisions 1-10 slots for backend servers in HAProxy's runtime configuration. It reserves the memory for up to ten instances even if you do not use them all. If you have more than ten healthy and available services, HAProxy only uses ten of them and backfills available slots in case of a service error. You can configure the number of available slots.

The _web._tcp.service.consul parameter instructs HAProxy to use the DNS SRV record for the backend service _web.service.consul_ to discover the available services.

This configuration also allows running two service instances on the same IP address using different ports resolve-opts allow-dup-ip and resolves IPv4 addresses for the service endpoints resolve-prefer ipv4.

Finally, specify that the server template should look at the resolvers consul stanza to discover where to find Consul's DNS interface.

Resolvers

resolvers consul
    nameserver consul 127.0.0.1:8600
    accepted_payload_size 8192
    hold valid 5s

The resolvers consul stanza defines the actual service discovery endpoint to be used by HAProxy.

nameserver consul 127.0.0.1:8600 points HAProxy to the DNS interface of the local Consul client.

You need to allow for a larger payload by configuring accepted_payload_size 8192, since DNS SRV records can result in larger DNS replies from Consul service discovery. This is based on the number of available and healthy services.

Finally hold valid 5s instructs HAProxy to check Consul's service catalog every 5 seconds for updates on available and healthy service endpoints. This value is also tuned in this example for faster service discovery.

Reload HAProxy

Reload your HAProxy instance to apply the adjusted haproxy.cfg configuration file.

$ service haproxy reload

Check HAProxy settings

Access the HAProxy statistics and monitoring page at http://<Your-HAProxy-IP-address>:1936 to verify the new configuration settings.

Register a web service

To register the web service on your first node with Consul, create a service definition in Consul's /etc/consul.d/ configuration directory.

service {
  name = "web"
  port = 80
  check {
    args = ["curl", "localhost"]
    interval = "3s"
  }
}

{
  "service": {
    "Name": "web",
    "Port": 80,
    "check": {
      "args": ["curl", "localhost"],
      "interval": "3s"
    }
  }
}

Reload the client to apply the new service definition.

$ consul reload

Register a second web service by repeating this process on a second node.

Verify services are registered in Consul

If the registration was successful you should see the change in Consul catalog. Choose your preferred method to verify that the services are registered correctly in Consul.

$ consul catalog services
consul
web

$ curl --silent http://127.0.0.1:8500/v1/catalog/services | jq

{
  "consul": [],
  "web": []
}

$ curl --silent http://127.0.0.1:8500/v1/catalog/service/web | jq

[
  {
    "ID": "1b183015-ef7f-52af-bad9-1e260c689a03",
    "Node": "web-app-0",
    "Address": "172.18.0.7",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "172.18.0.7",
      "lan_ipv4": "172.18.0.7",
      "wan": "172.18.0.7",
      "wan_ipv4": "172.18.0.7"
    },
    "NodeMeta": {
      "consul-network-segment": "",
      "consul-version": "1.20.1"
    },
    "ServiceKind": "",
    "ServiceID": "web-0",
    "ServiceName": "web",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 80,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 14342,
    "ModifyIndex": 14342
  },
  {
    "ID": "86be7e7c-cfaf-931f-297b-6992f049c1ff",
    "Node": "web-app-1",
    "Address": "172.18.0.9",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "172.18.0.9",
      "lan_ipv4": "172.18.0.9",
      "wan": "172.18.0.9",
      "wan_ipv4": "172.18.0.9"
    },
    "NodeMeta": {
      "consul-network-segment": "",
      "consul-version": "1.20.1"
    },
    "ServiceKind": "",
    "ServiceID": "web-1",
    "ServiceName": "web",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 80,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 14347,
    "ModifyIndex": 14347
  }
]

$ dig @127.0.0.1 -p 8600 -t srv web.service.consul +short
1 1 80 web-app-1.node.dc1.consul.
1 1 80 web-app-0.node.dc1.consul.

Check load balancing

The HAProxy monitoring UI also reflects the services, so you may verify that two instances of the ten available are now up and running.

Browse to the IP address of your HAProxy load balancer and reload the page several times. Because you registered two services in Consul and configured HAProxy to use round robin load balancing, you should see the connection toggling between both your available web servers.

Scale your backend servers

HAProxy queries Consul's DNS interface every five seconds to check if something changed within the requested service web.

Scale your web service registering the instance running on your third node.

Create a service definition in the third Consul client's configuration directory (/etc/consul.d/).

service {
  name = "web"
  port = 80
  check {
    args = ["curl", "localhost"]
    interval = "3s"
  }
}

{
  "service": {
    "Name": "web",
    "Port": 80,
    "check": {
      "args": ["curl", "localhost"],
      "interval": "3s"
    }
  }
}

Reload the client to apply the new service definition.

$ consul reload

Once you register the new web service, HAProxy detects the change and trigger an update in its runtime configuration. After scaling your backend server from two to three instances, the HAProxy statistics page reflects the resulting runtime load balancer configuration.

Traffic should now be load balanced across all three available services.

Stop one backend service

HAProxy's service discovery integration scales your backend configuration automatically depending on available services. It only uses healthy services for rendering the final configuration.

You configured Consul to perform a basic curl health check in your service definition. Consul notices if the web server instance is in an unhealthy state.

To simulate an error and see how Consul health checks are working, stop the web server process on one of you backend servers.

$ service <WEBSERVER> stop

The state of this service instance is immediately unhealthy in the Consul UI, since the curl health check results in an error because no service responds to requests on HTTP port 80.

Due to its regular check against Consul's DNS interface, your HAProxy instance notices that a change in the health of one of the services has occurred and adjusts its runtime load balancer configuration.

Your HAProxy instance now only balances traffic between the remaining healthy services.

You can again check the resulting runtime load balancer configuration for your HAProxy instance in the statistics page.

Note that the unhealthy instance is not in an error state from HAProxy's perspective. It is in maintenance state (MAINT) as DNS resolution for this specific host is not working anymore.

Restart the stopped web server instance. The Consul health check marks the service as healthy and adds the service back into the load balancer backend configuration to serve traffic.

Add DNS weights to the HAProxy configuration

In addition to Consul's DNS interface for querying only healthy instances, HAProxy's service discovery integration can also evaluate the DNS weight. In this section, you adjust one of your services to give it a higher weight. This enables HAProxy to adjust its runtime value for the backend server weight, which results in sending more traffic to the respective backend server.

This approach is helpful if your servers are different sizes, or some instances of a service are able to handle more requests than others.

Check the current weight

First, check the assigned DNS weights in Consul using the Consul DNS interface.

$ dig @127.0.0.1 -p 8600 -t srv web.service.consul +short
1 1 80 web-app-0.node.dc1.consul.
1 1 80 web-app-1.node.dc1.consul.
1 1 80 web-app-2.node.dc1.consul.

The DNS weight is the column before the port number (which is 80). The DNS weight is currently set to one for all of your healthy services, which is the default that Consul uses.

Add a weight to one service

HAProxy's service discovery integration queries Consul's DNS interface on a regular, configurable basis to get updates about changes for a given service and adjusts the runtime configuration of HAProxy automatically. Adjust your HAProxy runtime configuration by configuring additional options in Consul like DNS weights.

HAProxy supports weights between one and two hundred fifty-six. Consul's officially supported DNS weights do not match the supported HAProxy backend server weights, so you need to convert the desired HAProxy weight to a DNS weight configured in Consul's service definition with the following formula.

([Desired weight in HAProxy] * 256) - 1 = (Service weight in Consul)

In this guide, you want to have a HAProxy weight of two, which results in the following calculation.

(2 * 256) - 1 = 511

Adjust one of your service definitions by adding the weights option to the web service registration.

service {
  name = "web"
  port = 80
  check {
    args = ["curl", "localhost"]
    interval = "3s"
  }
  weights {
    passing = 511
    warning = 1
  }
}

{
  "service": {
    "Name": "web",
    "Port": 80,
    "check": {
      "args": ["curl", "localhost"],
      "interval": "3s"
    },
    "weights": {
      "passing": 511,
      "warning": 1
    }
  }
}

Reload the local Consul client to apply the new service definition.

$ consul reload

Check the new weight

First, check Consul's DNS interface to make sure Consul now shows another weight for the service instance you just configured.

$ dig @127.0.0.1 -p 8600 -t srv web.service.consul +short
1 511 80 web-app-1.node.dc1.consul.
1 1 80 web-app-0.node.dc1.consul.
1 1 80 web-app-2.node.dc1.consul.

Second, you can check the HAProxy statistics page, http://<Your-HAProxy-IP-address>:1936.

Finally, check that traffic is load balanced unequally across the three available service endpoints. Browse to the IP address of your HAProxy load balancer and reload the page several times. One of your instances serves twice as many requests as the others.

Next steps

Refer to the NGINX, F5, and Envoy guides for other load balancing options.