Integrate with infrastructure tools

As organizations expand, their applications and services grow in complexity and importance. These services become critical to business success necessitating continuous operation. New services will inevitably be introduced and existing ones need regular upgrades, all with minimal to zero downtime.

To achieve this level of reliability and agility, automation is essential. The industry has largely standardized around CI/CD and GitOps workflows, which HashiCorp also recommends for use with Consul.

This section covers the integration of Consul with infrastructure tools like CI/CD and ArgoCD to automate the deployment and management of services using CI/CD and GitOps workflows. You will focus on the Consul service discovery use case, which includes the deployment of applications/services, Consul client agents, service registration, and health checks.

Prerequisites

• Consul Enterprise v1.18.0 LTS.
• Access to a CI/CD tool (Jenkins, GitHub Actions, GitLab Runners, etc).
• Access to HashiCorp Cloud platform:
  • Packer
  • Terraform
  • Vault

Beginning CI/CD with Consul

Before implementing CI/CD with Consul, it is important to understand available approaches for deploying and managing software systems.

Managing software systems

Traditionally, software systems are managed by making in-place modifications, but this approach presents challenges. To overcome these, the industry is shifting towards immutable infrastructure, a method recommended by HashiCorp.

Considerations

• Teams and responsibilities

  Multiple teams are involved in the safe and reliable development and release of applications. A good understanding of the roles and responsibilities in a team will help establish ownership, along with clear channels of communication. Teams can include:

  • Developers: The primary consumers of CI/CD automation. Their primary responsibility is to write and maintain code for the service, which requires them to interact with VCS repositories.
  • Infrastructure/DevOps Teams: Responsible for setting up and configuring the CI/CD tools. This involves designing and configuring the workflows and setting up self-hosted runners. They collaborate with security teams to ensure proper security measures are taken.
  • Security Teams: Implement security measures for service communication and ensure that any security feature or access is aligned with the organization's security policies.
  • Networking Teams: Design and support the network architecture, working closely with service discovery specialists. For more details, refer to the People and process section of the Operating Guide for Adoptiondocument.

• Components of a typical Consul environment

  A typical Consul environment consists of:

  • Consul servers: Six servers running on VMs as per the HashiCorp Recommended Architecture.
  • Consul clients: One per VM that is running the application or services; one per Pod when running in Kubernetes.

• Runtime

  Knowing the runtime environment where Consul is running will influence how CI/CD will be implemented. The runtime environment could be VMs or Kubernetes.

• Secrets management

  A crucial aspect is the secure retrieval and use of secrets required to execute various steps of the CI/CD workflow. Secrets to keep in mind include:

  • Application secrets: Those such as API keys, DB credentials, or third-party keys
  • Consul client agent: Consul Enterprise License key, Consul ACL tokens, Gossip Key, TLS certificates
  • HCP Packer environment variables: HCP_CLIENT_ID, HCP_CLIENT_SECRET
  • HCP Terraform API tokens: TF_API_TOKEN
  • Cloud provider: Access keys

Software management of Consul

With the above considerations in mind, group the software management of Consul into the following three categories:

1. Consul servers on VMs,
2. Services along with Consul client agents on VMs, and
3. Consul on Kubernetes.

Consul servers on VMs

Deploying Consul servers on VMs is HashiCorp’s recommended approach. For more details, please refer to the Solution Design Guide.

To simplify the initial deployment, HashiCorp has created “Terraform Modules for Consul”. Please contact your HashiCorp account team for access to these modules.

Changes to the Consul control plane are infrequent, so the time and effort needed to implement automation via a CI/CD pipeline may not be justifiable.

The health of the Consul servers is critical to the overall health of the cluster. For better control over how and when changes occur, it is recommended to manage it via Terraform. Moreover, Consul has features such as autopilot to help automate upgrades with Consul Enterprise easily and reliably.

Services along with Consul client agents on VMs

The scale of this topic is large enough to warrant its section. Continue on to the next page to learn more about this topic.

Consul on Kubernetes

Although HashiCorp recommends deploying the control plane Consul servers on VMs, there are scenarios where this may not be feasible. In such cases, HashiCorp fully supports deploying the Consul Control Plane and the dataplane on Kubernetes.

By leveraging the advantages of containerization, Kubernetes orchestration, and the Helm chart for Consul, HashiCorp advises using a GitOps approach to automate the deployment and management of the Consul cluster.

GitOps is a modern infrastructure and application management methodology that uses Git as the single source of truth. In a GitOps workflow, the desired state of systems—including infrastructure, applications, and configurations—is stored in a Git repository. Changes are managed through Git commits, and an automated process ensures that the actual system state aligns with the desired state defined in the repository.

Several third-party GitOps tools can facilitate this process, such as Argo CD, Flux, and Jenkins X, all of which offer up their own levels of automation and flexibility. For an example, refer to the "Deploy Consul on Kubernetes with Argo CD"article for a review of the ArgoCD platform.