Upgrading to Latest 1.10.x

Introduction

This guide explains how to best upgrade a single Consul Enterprise datacenter to latest v1.10.x from a version of Consul that is forward compatible with v1.10. If you are on a major version of Consul prior to 1.8, you must first upgrade to latest 1.8.x before continuing with this guide. If you are already on a major version of 1.8 or 1.9, then this guide will go over the procedures required for upgrading to v1.10.x. This process will require intermediate version upgrades to a forward-compatible release of v1.8 or v1.9, as well as other licensing related configuration changes. If you have multiple Consul datacenters, upgrade them in the normal order and take the following instructions into account for each upgrade.

For the open source version of Consul please follow the General Upgrade Process.

You can only upgrade to Consul Enterprise 1.10 from a version of Consul Enterprise 1.8 >= 1.8.13 or 1.9 >= 1.9.7. Other versions of Consul Enterprise are not forward compatible with v1.10 and will cause issues during the upgrade that could result in agents failing to start due to changes in the way we manage licenses.

Requirements

All Consul servers, clients, and snapshot agents should be on a version of Consul >= 1.8.0 and < 1.10.0. If they are not at the minimum version or higher, follow the normal upgrade procedures to upgrade them until the version requirement is met.

Assumptions

This guides makes the following assumptions:

You are familiar with the General Upgrade Process.
You have the ability to run Consul CLI commands.
If ACLs are in use, then you possess a token with at least operator:read permissions.

Considerations

The licensing changes outlined on the Specific Version Details page are the main breaking changes in Consul Enterprise 1.10 that require special handling during the upgrade process. The page also describes other changes that might cause issues during an upgrade. You can also review the licensing FAQ, which includes granular details, as well as the full changelog. We strongly recommend reviewing the changes prior to upgrading.

Procedures

Any steps in the following section that mention upgrading servers/clients assume that normal safe upgrade procedures are followed. Licensing-related configuration changes are required for upgrading to 1.10. Intermediate version upgrades are also required to ensure forward compatibility. Refer to our documentation for the basic server upgrade process, and for autopilot assisted upgrades for more information.

1. Retrieve the datacenter's current license by executing the following CLI command:

consul license get -signed > consul.hclic

Note that you can run the command from anywhere Consul is already running or from a location that has network access to the cluster. Additional command line options for directing the request to the correct Consul API may be necessary if the command runs from another location.

This will save the license to a file in the local directory named consul.hclic. Having this will be useful when preparing the license for the 1.10 upgrades later on in the process.

2. Take a snapshot of the cluster by running the following command:

consul snapshot save original.snap

Note that you will need to ensure that there is enough disk space in the current directory to store the snapshot. In the worst case, the snapshot size could be close to the amount of RAM the Consul server processes are consuming.

This snapshot should not be needed, but if something were to go wrong, having the snapshot would make it possible to restore the previous state.

3. Determine if ACLs are enabled by running the following CLI command.

consul info | grep "acl ="

If you receive output like either of the following, then ACLs are enabled:

Error querying agent: Unexpected response code: 403 (Permission denied)

        acl = enabled

Select the correct tab below based on whether ACLs are enabled or disabled or if you are operating Consul within Kubernetes.

4. Upgrade all server agents to the latest 1.8.x or 1.9.x release.

5. Upgrade all client agents to the latest 1.8.x or 1.9.x release.

6. Upgrade all snapshot agents to the latest 1.8.x or 1.9.x release.

7. Take a snapshot of the upgraded cluster by running the following command:

consul snapshot save intermediate.snap

Note that you can run the command from anywhere Consul is already running or from a location that has network access to the cluster. If run elsewhere, additional command line options may be needed to direct the request to the right Consul API.

Note that you will need to ensure there is enough disk space in the current directory to store the snapshot. In the worst case, the snapshot size could be close to the amount of RAM the Consul server processes are consuming.

This snapshot should not be needed, but if something were to go wrong, having the snapshot would make it possible to restore the previous state.

8. Pre-configure the server agents with a license.

9. Upgrade all server agents to latest v1.10.x.

10. Upgrade all client agents to latest v1.10.x.

11. Upgrade all snapshot agents to latest v1.10.x.

We do not recommend running in production with ACLs disabled. An alternative upgrade path involves first enabling ACLs by following this tutorial and then proceeding with the instructions within the "ACLs Enabled" tab.

4. Upgrade all server agents to the latest 1.8.x or 1.9.x release.

5. Pre-configure the client agents with a license.

This can either be set with the license_path configuration item, the CONSUL_LICENSE_PATH environment variable, or the CONSUL_LICENSE environment variable. See the licensing documentation for more information about how to configure the license. You can also refer to the Apply Enterprise License tutorial for additional information. Note that while Consul Enterprise 1.8.13 and 1.9.7 have the ability to load the license from these configurations, it is intended to only be used during upgrades. Licenses loaded this way are not online reloadable. It is therefore important to promptly proceed with the upgrade and not leave agents in the intermediate state.

6. Upgrade all client agents to the latest 1.8.x or 1.9.x release.

7. Pre-configure the snapshot agents with a license. This is the same process that was executed for the servers in step 5.

8. Upgrade all snapshot agents to the latest 1.8.x or 1.9.x release.

9. Take a snapshot of the upgraded cluster by running the following command:

consul snapshot save intermediate.snap

This snapshot should not be needed but if something were to go wrong, having the snapshot would make it possible to restore the previous state.

10. Pre-configure the server agents with a license. This is the same process that was executed for the servers in step 5.

11. Upgrade all server agents to latest v1.10.x.

12. Upgrade all client agents to latest v1.10.x.

13. Upgrade all snapshot agents to latest v1.10.x.

4. If the Consul servers are running outside the Kubernetes cluster, ensure they are upgraded to the latest 1.8.x or 1.9.x before the next steps. Otherwise, this step can be skipped.

5. Upgrade to the latest Consul helm chart v0.32.0+.

6. Update the value of global.image in the values file to the latest 1.8.x or 1.9.x image.

Additionally, ensure that the Kubernetes secret with the license is specified in the values server.enterpriseLicense.secretName and server.enterpriseLicense.secretKey.

7. Upgrade the cluster.

Follow the steps in the Kubernetes Upgrade Guide to upgrade the cluster. Ensure you check the steps to upgrade the servers and the clients.

8. If the Consul servers are running outside the Kubernetes cluster, ensure they are upgraded to 1.10.0 before the next steps. Otherwise, this step can be skipped.

This will require pre-configuring the servers with a license before running 1.10. This can either be set with the license_path configuration item, the CONSUL_LICENSE_PATH environment variable or the CONSUL_LICENSE environment variable. See the licensing documentation for more information about how to configure the license. You can also refer to the Apply Enterprise License tutorial for additional information. Note that while Consul Enterprise 1.8.13 and 1.9.7 have the ability to load the license from these configurations, it is intended to only be used during upgrades. Licenses loaded this way are not online reloadable. It is therefore important to promptly proceed with the upgrade and not leave agents in the intermediate state.

9. Update the value of global.image in the values file to the 1.10.0 image.

10. Upgrade the cluster.

Follow the steps in the Kubernetes Upgrade Guide to then upgrade the cluster. Ensure you check the steps to upgrade the servers and the clients.

Post-Upgrade Configuration Changes

No configuration changes are required for this upgrade.