• HashiCorp Developer

  • HashiCorp Cloud Platform
  • Terraform
  • Packer
  • Consul
  • Vault
  • Boundary
  • Nomad
  • Waypoint
  • Vagrant
Consul
  • Install
  • Tutorials
  • Documentation
  • API
  • CLI
  • Try Cloud(opens in new tab)
  • Sign up
Networking

Skip to main content
3 tutorials
  • DNS Caching
  • Forward DNS for Consul Service Discovery
  • Federate Multiple Datacenters Using WAN Gossip

  • Resources

  • Tutorial Library
  • Certifications
  • Community Forum
    (opens in new tab)
  • Support
    (opens in new tab)
  • GitHub
    (opens in new tab)
  1. Developer
  2. Consul
  3. Tutorials
  4. Networking
  5. Forward DNS for Consul Service Discovery

Forward DNS for Consul Service Discovery

  • 14min

  • ConsulConsul

By default, DNS is served from port 53. On most operating systems, this requires elevated privileges. Rather than running Consul with an administrative or root account, you can forward appropriate queries to Consul (running on an unprivileged port) from another DNS server or port redirect.

In this tutorial, you will learn how to configure the following DNS servers for DNS forwarding.

  • systemd-resolved
  • BIND
  • dnsmasq
  • Unbound
  • iptables
  • macOS

After configuring forwarding, you will test the configuration. Finally, you will learn how to troubleshoot some common errors.

Prerequisites

To complete this tutorial, you will need a running Consul agent. Consul should be running with default settings and serving DNS on port 8600.

By default, Consul does not resolve DNS records outside the .consul. zone unless the recursors configuration option has been set. For example, if a Consul DNS reply includes a CNAME record pointing outside the .consul top level domain (TLD), then the DNS reply will only include CNAME records by default. When recursors is set and the upstream resolver is functioning correctly, however, Consul will try to resolve CNAMEs and include any records (e.g., A, AAAA, PTR) for them in its DNS reply.

For BIND, dnsmasq and Unbound, you will need to configure /etc/resolv.conf with a nameserver entry poining to localhost. This configures the operating system to forward the request to the chosen DNS daemon.

/etc/resolv.conf
nameserver 127.0.0.1

systemd-resolved setup

You can use the systemd-resolved service to resolve local application names in your network. To use the service, configure systemd-resolved to send .consul domain queries to Consul by creating consul.conf file located in the /etc/systemd/resolved.conf.d/ directory.

Add the following settings to your resolved configuration file:

/etc/systemd/resolved.conf.d/consul.conf
[Resolve]
DNS=127.0.0.1
DNSSEC=false
Domains=~consul

The main limitation with this configuration is that systemd 245 and older does not support configuring port numbers in the DNS field. You can either configure Consul to listen on port 53 instead of 8600, or map port 53 to 8600 using iptables.

Binding to port 53 usually requires running systemd-resolved as a privileged user or running Linux with the CAP_NET_BIND_SERVICE capability. If you are using the Consul Docker image, then you will need to add the following to the environment to allow Consul to use the port: CONSUL_ALLOW_PRIVILEGED_PORTS=yes

The following iptables commands are sufficient to map the ports.

iptables --table nat --append OUTPUT --destination localhost --protocol udp --match udp --dport 53 --jump REDIRECT --to-ports 8600
iptables --table nat --append OUTPUT --destination localhost --protocol tcp --match tcp --dport 53 --jump REDIRECT --to-ports 8600

The above configuration assumes Consul's DNS server is listening on the loopback address. If Consul is not listening on the loopback IP, replace the references to 'localhost' and '127.0.0.1' with the appropriate IP address for your environment.

If your distribution uses systemd 246 and newer, you can specify the DNS port directly in the systemd-resolved configuration file. Previous versions of systemd required iptables rules to direct DNS traffic to Consul.

/etc/systemd/resolved.conf.d/consul.conf
[Resolve]
DNS=127.0.0.1:8600
DNSSEC=false
Domains=~consul

Note: PTR record queries will still be sent out to the other configured resolvers, in addition to Consul.

After creating the resolved configuration, restart systemd-resolved.

# systemctl restart systemd-resolved

Validate the systemd-resolved configuration

Validate that systemd-resolved has restarted and is configured to forward queries to Consul.

# systemctl is-active systemd-resolved
active

# resolvectl domain
Global: ~consul
Link 2 (eth0):

# resolvectl query consul.service.consul
consul.service.consul: 127.0.0.1

-- Information acquired via protocol DNS in 6.6ms.
-- Data is authenticated: no

Confirm that /etc/resolv.conf points to the stub-resolv.conf file managed by systemd-resolved, and that the IP address for systemd-resolved's stub resolver is the configured nameserver.

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 37 Aug 20 22:50 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0

Ensure that the operating system can resolve DNS queries to the .consul domain.

$ host consul.service.consul
consul.service.consul has address 127.0.0.1

Using any local resolver with systemd

By default, the local resolver stub in the resolved.conf file is configured to listen for UDP and TCP requests at 127.0.0.53:53, but you can set the DNSStubListener option to false, which disables the stub. As a result, your system will be able to use any DNS configuration as long as it loads earlier than resolved.

/etc/systemd/resolved.conf
DNSStubListener=false

Disabling the local resolver stub can also solve other DNS configuration issues. See Troubleshooting for additional information.

BIND setup

In this example, BIND and Consul are running on the same machine.

NOTE: If your distribution uses systemd, you may need to disable systemd-resolved prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.

First, disable DNSSEC so that Consul and BIND can communicate. The following example shows a BIND configuration with DNSSEC disabled.

/etc/named.conf
options {
  listen-on port 53 { 127.0.0.1; };
  listen-on-v6 port 53 { ::1; };
  directory       "/var/named";
  dump-file       "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-query     { localhost; };
  recursion yes;

  dnssec-enable no;
  dnssec-validation no;

  /* Path to ISC DLV key */
  bindkeys-file "/etc/named.iscdlv.key";

  managed-keys-directory "/var/named/dynamic";
};

include "/etc/named/consul.conf";

Zone file

Next, set up a zone for your Consul managed records in consul.conf.

/etc/named/consul.conf
zone "consul" IN {
  type forward;
  forward only;
  forwarders { 127.0.0.1 port 8600; };
};

Dnsmasq setup

NOTE: If your distribution uses systemd, you may need to disable systemd-resolved prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.

Dnsmasq is typically configured via a dnsmasq.conf or a series of files in the /etc/dnsmasq.d directory. Add the following settings to your dnsmasq configuration file (e.g., /etc/dnsmasq.d/10-consul):

/etc/dnsmasq.d/10-consul
# Enable forward lookup of the 'consul' domain:
server=/consul/127.0.0.1#8600

# Uncomment and modify as appropriate to enable reverse DNS lookups for
# common netblocks found in RFC 1918, 5735, and 6598:
#rev-server=0.0.0.0/8,127.0.0.1#8600
#rev-server=10.0.0.0/8,127.0.0.1#8600
#rev-server=100.64.0.0/10,127.0.0.1#8600
#rev-server=127.0.0.1/8,127.0.0.1#8600
#rev-server=169.254.0.0/16,127.0.0.1#8600
#rev-server=172.16.0.0/12,127.0.0.1#8600
#rev-server=192.168.0.0/16,127.0.0.1#8600
#rev-server=224.0.0.0/4,127.0.0.1#8600
#rev-server=240.0.0.0/4,127.0.0.1#8600

Restart the dnsmasq service after creating the configuration.

You can configure additional settings in dnsmasq that may be useful in your environments. See dnsmasq(8) for additional details.

/etc/dnsmasq.conf
# Accept DNS queries only from hosts whose address is on a local subnet.
#local-service

# Don't poll /etc/resolv.conf for changes.
#no-poll

# Don't read /etc/resolv.conf. Get upstream servers only from the command
# line or the dnsmasq configuration file (see the "server" directive below).
#no-resolv

# Specify IP address(es) of other DNS servers for queries not handled
# directly by consul. There is normally one 'server' entry set for every
# 'nameserver' parameter found in '/etc/resolv.conf'. See dnsmasq(8)'s
# 'server' configuration option for details.
#server=1.2.3.4
#server=208.67.222.222
#server=8.8.8.8

# Set the size of dnsmasq's cache. The default is 150 names. Setting the
# cache size to zero disables caching.
#cache-size=65536

Unbound setup

NOTE: If your distribution uses systemd, you may need to disable systemd-resolved prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.

Unbound is typically configured in a unbound.conf file or a series of files in the /etc/unbound/unbound.conf.d directory. Add the following settings to your Unbound configuration file (e.g., /etc/unbound/unbound.conf.d/consul.conf):

/etc/unbound/unbound.conf.d/consul.conf
#Allow insecure queries to local resolvers
server:
  do-not-query-localhost: no
  domain-insecure: "consul"

#Add consul as a stub-zone
stub-zone:
  name: "consul"
  stub-addr: 127.0.0.1@8600

You may have to add the following line to the bottom of your /etc/unbound/unbound.conf file for the new configuration to be included.

/etc/unbound/unbound.conf
include: "/etc/unbound/unbound.conf.d/*.conf"

iptables setup

The rules must be set on the same host as the Consul instance. Relay hosts should not be on the same host, otherwise the redirects will intercept the traffic.

On Linux systems that support it, incoming requests and requests to the local host can use iptables to forward to ports on the same machine without using a secondary service. The recursors flag is especially important for resolving other domains with iptables. This is because Consul only resolves the .consul TLD by default. Recursors should not include the local host address because the iptables redirects would intercept the requests.

The iptables method is suitable for situations where an external DNS service is already running in your infrastructure and is used as the recursor. It is also a suitable method if you want to use an existing DNS server as your query endpoint and forward requests for the consul domain to the Consul server. In both cases, you may want to query the Consul server without the overhead of running a separate service on the Consul host.

iptables -t nat -A PREROUTING -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600

macOS setup

On macOS systems, you can use the macOS system resolver to point all .consul requests to Consul. Documentation for this feature is available via: man 5 resolver. You must add a resolver entry in /etc/resolver/ to point at Consul. If you do not have this folder, create it. To configure a resolver entry, you will need sudo/root access. Create a new file /etc/resolver/consul, and add the following text to the file:

/etc/resolver/consul
nameserver 127.0.0.1
port 8600

The configuration informs the macOS resolver daemon to forward all .consul TLD requests to 127.0.0.1 on port 8600.

Query Consul DNS and BIND

First, add a service registration to your Consul agent. You can use the following example service configuration file and registration command.

redis.json
{
  "service": {
    "name": "redis",
    "port": 80
  }
}
$ curl --data @redis.json http://127.0.0.1:8500/v1/catalog/register

To ensure that Consul DNS and your DNS server are configured correctly, use a dig query for the service you've registered.

$ dig @localhost -p 8600 primary.redis.service.dc1.consul. A
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

Next, run the same query against your BIND instance and make sure you get a valid result:

$ dig @localhost -p 53 primary.redis.service.dc1.consul. A
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

If desired, verify reverse DNS using the same method:

$ dig @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; <<>> DiG 9.10.3-P3 <<>> @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3713
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;133.139.16.172.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
133.139.16.172.in-addr.arpa. 0  IN  PTR consul1.node.dc1.consul.

;; Query time: 3 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jan 31 04:25:39 UTC 2016
;; MSG SIZE  rcvd: 109
$ dig @127.0.0.1 -p 8600 +short -x 172.16.139.133
consul1.node.dc1.consul.

Troubleshooting

If you don't get an answer from your DNS server (e.g., BIND, Dnsmasq) but you do get an answer from Consul, turn on your DNS server's query log to check for errors.

For BIND:

rndc querylog
tail -f /var/log/messages

The log may show errors like this:

error (no valid RRSIG) resolving
error (no valid DS) resolving

This indicates that DNSSEC is not disabled properly.

If you see errors about network connections, verify that there are no firewall or routing problems between the servers running BIND and Consul.

For Dnsmasq, see the log-queries configuration option and the USR1 signal.

DNS forwarding may fail if you use the default systemd-resolved configuration and attempt to bind to 0.0.0.0. The default configuration uses a DNS stub that listens for UDP and TCP requests at 127.0.0.53. As a result, attempting to bind to 127.0.0.53 conflicts with the running stub. You can disable the stub as described in the Using any local resolver with systemd section to troubleshoot this problem.

Next steps

In this tutorial, you configured DNS forwarding for Consul DNS and your DNS server. Now that you have configured your environment, you can start using Consul for service discovery or service mesh.

  • Ensure only healthy service instances are available in Consul DNS.
  • Deploy services into a zero trust network with Consul service mesh.
 Previous
 Next

This tutorial also appears in:

  •  
    30 tutorials
    Associate Tutorial List
    Study for the Consul Associate exam by following these tutorials. Login to Learn and bookmark them to track your progress. Study the complete list of study materials (including docs) in the Certification Prep guides.
    • Consul

On this page

  1. Forward DNS for Consul Service Discovery
  2. Prerequisites
  3. systemd-resolved setup
  4. BIND setup
  5. Dnsmasq setup
  6. Unbound setup
  7. iptables setup
  8. macOS setup
  9. Query Consul DNS and BIND
  10. Troubleshooting
  11. Next steps
Give Feedback(opens in new tab)
  • Certifications
  • System Status
  • Terms of Use
  • Security
  • Privacy
  • Trademark Policy
  • Trade Controls
  • Give Feedback(opens in new tab)