HashiConf More sessions have been added to the conference agenda. Buy your pass and plan your schedule. Register
Consul
Consul Home

Automated upgrades

Enterprise

This feature requires HashiCorp Cloud Platform (HCP) or self-managed Consul Enterprise. Refer to the enterprise feature matrix for additional information.

Consul Enterprise enables the capability of automatically upgrading a cluster of Consul servers to a new version as updated server nodes join the cluster. This automated upgrade will spawn a process which monitors the amount of voting members currently in a cluster. When an equal amount of new server nodes are joined running the desired version, the lower versioned servers will be demoted to non voting members. Demotion of legacy server nodes will not occur until the voting members on the new version match. Once this demotion occurs, the previous versioned servers can be removed from the cluster safely.

Introduction

Upgrades are always a delicate moment for any production environment and it is important to have to have best practices in place that simplify the process where possible.

Consul Enterprise provides automated upgrades with the autopilot feature that enable you to test the upgrade process before making changes to production. The feature allows you to start new Consul servers alongside the old ones and automatically switch to the new servers, as soon as they are able to reach quorum.

This process automates the leader election process and ensures that the new leader is elected among the new nodes so that removing the old nodes from the datacenter does not trigger a leader election.

Warning

This tutorial describes a cluster upgrade process that deploys new servers and then removes the old ones. To upgrade the version of the consul binary running on servers in your cluster without deploying new servers, follow the general upgrade process and disable autopilot's upgrade migration feature.

Using this tutorial you will test the functionality and get examples on how to:

  1. Upgrade Consul servers to a newer version
  2. Patch configuration for server nodes without changing Consul version

This will help you create a standardized and repeatable internal process for server upgrades.

Prerequisites

To test the automated upgrades feature explained in this tutorial you will need:

  • A Consul Enterprise datacenter with three servers running Consul Enterprise 1.7.0.
  • Three extra nodes with Consul Enterprise 1.7.2 binary installed to be used as the new servers once the upgrade is concluded.

You will also need a text editor, the curl executable to test the API endpoints, and optionally the jq command to format the output for curl.

Upgrade to a new Consul version

The automated upgrades feature is enabled by default in Consul Enterprise. You can verify the configuration for your datacenter by using the consul operator autopilot command.

$ consul operator autopilot get-config

##...
DisableUpgradeMigration = false
UpgradeVersionTag = ""

In case the feature is disabled in your datacenter you can enable it using the consul operator autopilot set-config -disable-upgrade-migration=false command.

Add new servers

Once you have verified that DisableUpgradeMigration is set to false, you can start the new servers and make sure they join the existing datacenter with the Consul 1.7.0 servers. In this example the old servers with Consul version 1.7.0 are identified as server-1, server-2, and server-3. While the new ones with Consul version 1.7.2 are identified as server-1a, server-2a, and server-3a.

When autopilot detects that a server with a newer version of Consul has joined the datacenter, it will wait to promote the new server as voter until enough newer-versioned servers have been added to the datacenter to reach quorum. When the count of new servers equals or exceeds that of the old servers, autopilot will begin promoting the new servers to voters and demoting the old servers. After this is finished, the old servers can be safely removed from the datacenter.

To check the Consul version on the old servers, you can either use the autopilot health endpoint or the consul members command.

$ consul members

Node      Address           Status  Type    Build      Protocol  DC   Segment
server-1  10.20.10.11:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-2  10.20.10.12:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-3  10.20.10.13:8301  alive   server  1.7.0+ent  2         dc1  <all>

Now that you have checked the autopilot configuration and Consul version on your existing three servers, start two new servers with Consul version 1.7.2. Be sure to join them to the existing datacenter.

Verify the new servers are not yet part of the voters pool.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  leader    true   3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  true   3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  true   3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  follower  false  3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  false  3

Finally, start the third new server on Consul version 1.7.2. Once the third new server is started and autopilot detects the possibility to reach a quorum among the new servers only it promotes them as voters, triggers a new leader election, and demotes the old nodes as non-voters.

Verify the three new servers on Consul version 1.7.2 are now voters.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  follower  false  3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  false  3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  false  3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  leader    true   3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  true   3
server-3a  36d57c89-dc39-056d-f8eb-b1a63faa85b8  10.20.10.23:8300  follower  true   3

Stop old servers

Once the upgrade is rolled out you can proceed by removing the old servers from the datacenter using the consul leave command. This will leave only the new servers and will prevent a leader election during the process that you would have experienced without autopilot.

Migrations without a Consul version change

In some environments, Consul is not the only component that needs to be upgraded to ensure security or stability for the node. For example, you might need to roll out node updates to apply OS patches or new configuration settings.

To cover these cases, Consul autopilot offers the possibility to set the UpgradeVersionTag parameter to override the version information used during automated upgrades. The same logic can be used for updating the datacenter nodes even without the need to change Consul version.

If the UpgradeVersionTag is configured, Consul will use its value to look for a version in each server's specified -node-meta tag. For example, if UpgradeVersionTag is set to build, and -node-meta build:0.0.2 is used when starting a server, that server's version will be 0.0.2 when considered in a migration. The upgrade logic will follow semantic versioning and the version string must be in the form of either X, X.Y, or X.Y.Z.

Configure autopilot for existing servers

For this example, you can use the three servers started in the previous example (server-1a, server-2a and server-3a). To perform the migration you will start three extra servers called server-1b, server-2b and server-3b. Using three new servers (server-1b, server-2b and server-3b) will help you differentiate the command output from the previous example. If you do not want to start three new nodes you can reuse the ones used previously as long as the Consul binary on the servers is changed to match the one running on server-1a, server-2a and server-3a.

Before proceeding with the upgrade it is necessary to prepare the existing servers to be compared with the new ones. For all the existing servers, add the node_meta option to the configuration.

Consul server agent configuration

## ...
node_meta {
  build = "0.0.1"
}
## ...

This change requires a reload.

$ consul reload

Configuration reload triggered

Once the configuration is updated, you should modify the Consul autopilot configuration to reflect the node_meta configuration.

$ consul operator autopilot set-config -upgrade-version-tag=build

Configuration updated!

Ensure the autopilot configuration has been successfully updated.

$ consul operator autopilot get-config

## ...
DisableUpgradeMigration = false
UpgradeVersionTag = "build"

The UpgradeVersionTag should be set to build.

Start new servers

For the new servers, you will define the UpgradeVersionTag directly in the configuration.

## ...
node_meta {
  build = "0.0.2"
}
autopilot {
  upgrade_version_tag = "build"
}
## ...

Once the third new server is started and autopilot detects the possibility to reach a quorum among the new servers only it promotes them as voters, triggers a new leader election, and demotes the old nodes as non-voters. This time the version comparison happened not using the Consul internal version but the value expressed in the build value.

$ consul operator raft list-peers

Node       ID                                    Address           State     Voter  RaftProtocol
server-1a   dcc542fc-ff83-2cab-5bbb-7615584be019  10.20.10.11:8300  follower  false  3
server-2a   f38a430b-dda0-ad3d-a394-08c59052723c  10.20.10.12:8300  follower  false  3
server-3a   647a98a1-d2ab-f5d3-e911-91c1c10a92f4  10.20.10.13:8300  follower  false  3
server-1b   7770d10c-fcca-f6f9-222a-e1777a7fd29e  10.20.10.21:8300  leader    true   3
server-2b   bace6b4d-5e2e-8942-d553-ca527cbc2386  10.20.10.22:8300  follower  true   3
server-3b   221de624-0f5c-29da-ffaf-39481adcd234  10.20.10.23:8300  follower  true   3

Stop old servers

Once the migration is rolled out you can proceed by removing the old servers from the datacenter using the consul leave command. This will leave only the new servers but will prevent a leader election during the process that you would have experienced without autopilot.

Next steps

In this tutorial you upgraded your Consul datacenter by using autopilot's automated upgrades functionality. The automated upgrade feature allows you to both upgrade the Consul version of your servers and to deploy updated version for your server instances without the need to change the Consul version to distribute patches. You now have an extra tool to use when planning upgrades of critical system and to reduce manual steps to a minimum.

Before planning any Consul upgrade make sure to review version specific instructions provided by:

You can learn more on other autopilot functionalities by checking our autopilot tutorial.

Edit this page on GitHub