Forward DNS for Consul Service Discovery

By default, DNS is served from port 53. On most operating systems, this requires elevated privileges. Rather than running Consul with an administrative or root account, you can forward appropriate queries to Consul (running on an unprivileged port) from another DNS server or port redirect.

In this tutorial, you will learn how to configure the following DNS servers for DNS forwarding.

• systemd-resolved
• BIND
• dnsmasq
• Unbound
• iptables
• macOS

After configuring forwarding, you will test the configuration. Finally, you will learn how to troubleshoot some common errors.

Prerequisites

To complete this tutorial, you will need a running Consul agent. Consul should be running with default settings and serving DNS on port 8600.

By default, Consul does not resolve DNS records outside the .consul. zone unless the recursors configuration option has been set. For example, if a Consul DNS reply includes a CNAME record pointing outside the .consul top level domain (TLD), then the DNS reply will only include CNAME records by default. When recursors is set and the upstream resolver is functioning correctly, however, Consul will try to resolve CNAMEs and include any records (e.g., A, AAAA, PTR) for them in its DNS reply.

For BIND, dnsmasq and Unbound, you will need to configure /etc/resolv.conf with a nameserver entry poining to localhost. This configures the operating system to forward the request to the chosen DNS daemon.

nameserver 127.0.0.1

nameserver 127.0.0.1

systemd-resolved setup

You can use the systemd-resolved service to resolve local application names in your network. To use the service, configure systemd-resolved to send .consul domain queries to Consul by creating consul.conf file located in the /etc/systemd/resolved.conf.d/ directory.

[Resolve]
DNS=127.0.0.1
DNSSEC=false
Domains=~consul

[Resolve]
DNS=127.0.0.1
DNSSEC=false
Domains=~consul

iptables --table nat --append OUTPUT --destination localhost --protocol udp --match udp --dport 53 --jump REDIRECT --to-ports 8600
iptables --table nat --append OUTPUT --destination localhost --protocol tcp --match tcp --dport 53 --jump REDIRECT --to-ports 8600

iptables --table nat --append OUTPUT --destination localhost --protocol udp --match udp --dport 53 --jump REDIRECT --to-ports 8600
iptables --table nat --append OUTPUT --destination localhost --protocol tcp --match tcp --dport 53 --jump REDIRECT --to-ports 8600

[Resolve]
DNS=127.0.0.1:8600
DNSSEC=false
Domains=~consul

[Resolve]
DNS=127.0.0.1:8600
DNSSEC=false
Domains=~consul

After creating the resolved configuration, restart systemd-resolved.

# systemctl restart systemd-resolved

# systemctl restart systemd-resolved

Validate the systemd-resolved configuration

Validate that systemd-resolved has restarted and is configured to forward queries to Consul.

# systemctl is-active systemd-resolved
active

# resolvectl domain
Global: ~consul
Link 2 (eth0):

# resolvectl query consul.service.consul
consul.service.consul: 127.0.0.1

-- Information acquired via protocol DNS in 6.6ms.
-- Data is authenticated: no

# systemctl is-active systemd-resolved
active

# resolvectl domain
Global: ~consul
Link 2 (eth0):

# resolvectl query consul.service.consul
consul.service.consul: 127.0.0.1

-- Information acquired via protocol DNS in 6.6ms.
-- Data is authenticated: no

Confirm that /etc/resolv.conf points to the stub-resolv.conf file managed by systemd-resolved, and that the IP address for systemd-resolved's stub resolver is the configured nameserver.

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 37 Aug 20 22:50 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 37 Aug 20 22:50 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0

Ensure that the operating system can resolve DNS queries to the .consul domain.

$ host consul.service.consul
consul.service.consul has address 127.0.0.1

$ host consul.service.consul
consul.service.consul has address 127.0.0.1

Using any local resolver with systemd

By default, the local resolver stub in the resolved.conf file is configured to listen for UDP and TCP requests at 127.0.0.53:53, but you can set the DNSStubListener option to false, which disables the stub. As a result, your system will be able to use any DNS configuration as long as it loads earlier than resolved.

DNSStubListener=false

DNSStubListener=false

Disabling the local resolver stub can also solve other DNS configuration issues. See Troubleshooting for additional information.

BIND setup

In this example, BIND and Consul are running on the same machine.

First, disable DNSSEC so that Consul and BIND can communicate. The following example shows a BIND configuration with DNSSEC disabled.

options {
  listen-on port 53 { 127.0.0.1; };
  listen-on-v6 port 53 { ::1; };
  directory       "/var/named";
  dump-file       "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-query     { localhost; };
  recursion yes;

  dnssec-enable no;
  dnssec-validation no;

  /* Path to ISC DLV key */
  bindkeys-file "/etc/named.iscdlv.key";

  managed-keys-directory "/var/named/dynamic";
};

include "/etc/named/consul.conf";

options {
  listen-on port 53 { 127.0.0.1; };
  listen-on-v6 port 53 { ::1; };
  directory       "/var/named";
  dump-file       "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-query     { localhost; };
  recursion yes;

  dnssec-enable no;
  dnssec-validation no;

  /* Path to ISC DLV key */
  bindkeys-file "/etc/named.iscdlv.key";

  managed-keys-directory "/var/named/dynamic";
};

include "/etc/named/consul.conf";

Zone file

Next, set up a zone for your Consul managed records in consul.conf.

zone "consul" IN {
  type forward;
  forward only;
  forwarders { 127.0.0.1 port 8600; };
};

zone "consul" IN {
  type forward;
  forward only;
  forwarders { 127.0.0.1 port 8600; };
};

Dnsmasq setup

Dnsmasq is typically configured via a dnsmasq.conf or a series of files in the /etc/dnsmasq.d directory. Add the following settings to your dnsmasq configuration file (e.g., /etc/dnsmasq.d/10-consul):

# Enable forward lookup of the 'consul' domain:
server=/consul/127.0.0.1#8600

# Uncomment and modify as appropriate to enable reverse DNS lookups for
# common netblocks found in RFC 1918, 5735, and 6598:
#rev-server=0.0.0.0/8,127.0.0.1#8600
#rev-server=10.0.0.0/8,127.0.0.1#8600
#rev-server=100.64.0.0/10,127.0.0.1#8600
#rev-server=127.0.0.1/8,127.0.0.1#8600
#rev-server=169.254.0.0/16,127.0.0.1#8600
#rev-server=172.16.0.0/12,127.0.0.1#8600
#rev-server=192.168.0.0/16,127.0.0.1#8600
#rev-server=224.0.0.0/4,127.0.0.1#8600
#rev-server=240.0.0.0/4,127.0.0.1#8600

# Enable forward lookup of the 'consul' domain:
server=/consul/127.0.0.1#8600

# Uncomment and modify as appropriate to enable reverse DNS lookups for
# common netblocks found in RFC 1918, 5735, and 6598:
#rev-server=0.0.0.0/8,127.0.0.1#8600
#rev-server=10.0.0.0/8,127.0.0.1#8600
#rev-server=100.64.0.0/10,127.0.0.1#8600
#rev-server=127.0.0.1/8,127.0.0.1#8600
#rev-server=169.254.0.0/16,127.0.0.1#8600
#rev-server=172.16.0.0/12,127.0.0.1#8600
#rev-server=192.168.0.0/16,127.0.0.1#8600
#rev-server=224.0.0.0/4,127.0.0.1#8600
#rev-server=240.0.0.0/4,127.0.0.1#8600

Restart the dnsmasq service after creating the configuration.

You can configure additional settings in dnsmasq that may be useful in your environments. See dnsmasq(8) for additional details.

# Accept DNS queries only from hosts whose address is on a local subnet.
#local-service

# Don't poll /etc/resolv.conf for changes.
#no-poll

# Don't read /etc/resolv.conf. Get upstream servers only from the command
# line or the dnsmasq configuration file (see the "server" directive below).
#no-resolv

# Specify IP address(es) of other DNS servers for queries not handled
# directly by consul. There is normally one 'server' entry set for every
# 'nameserver' parameter found in '/etc/resolv.conf'. See dnsmasq(8)'s
# 'server' configuration option for details.
#server=1.2.3.4
#server=208.67.222.222
#server=8.8.8.8

# Set the size of dnsmasq's cache. The default is 150 names. Setting the
# cache size to zero disables caching.
#cache-size=65536

# Accept DNS queries only from hosts whose address is on a local subnet.
#local-service

# Don't poll /etc/resolv.conf for changes.
#no-poll

# Don't read /etc/resolv.conf. Get upstream servers only from the command
# line or the dnsmasq configuration file (see the "server" directive below).
#no-resolv

# Specify IP address(es) of other DNS servers for queries not handled
# directly by consul. There is normally one 'server' entry set for every
# 'nameserver' parameter found in '/etc/resolv.conf'. See dnsmasq(8)'s
# 'server' configuration option for details.
#server=1.2.3.4
#server=208.67.222.222
#server=8.8.8.8

# Set the size of dnsmasq's cache. The default is 150 names. Setting the
# cache size to zero disables caching.
#cache-size=65536

Unbound setup

Unbound is typically configured in a unbound.conf file or a series of files in the /etc/unbound/unbound.conf.d directory. Add the following settings to your Unbound configuration file (e.g., /etc/unbound/unbound.conf.d/consul.conf):

#Allow insecure queries to local resolvers
server:
  do-not-query-localhost: no
  domain-insecure: "consul"

#Add consul as a stub-zone
stub-zone:
  name: "consul"
  stub-addr: 127.0.0.1@8600

#Allow insecure queries to local resolvers
server:
  do-not-query-localhost: no
  domain-insecure: "consul"

#Add consul as a stub-zone
stub-zone:
  name: "consul"
  stub-addr: 127.0.0.1@8600

You may have to add the following line to the bottom of your /etc/unbound/unbound.conf file for the new configuration to be included.

include: "/etc/unbound/unbound.conf.d/*.conf"

include: "/etc/unbound/unbound.conf.d/*.conf"

iptables setup

The rules must be set on the same host as the Consul instance. Relay hosts should not be on the same host, otherwise the redirects will intercept the traffic.

On Linux systems that support it, incoming requests and requests to the local host can use iptables to forward to ports on the same machine without using a secondary service. The recursors flag is especially important for resolving other domains with iptables. This is because Consul only resolves the .consul TLD by default. Recursors should not include the local host address because the iptables redirects would intercept the requests.

The iptables method is suitable for situations where an external DNS service is already running in your infrastructure and is used as the recursor. It is also a suitable method if you want to use an existing DNS server as your query endpoint and forward requests for the consul domain to the Consul server. In both cases, you may want to query the Consul server without the overhead of running a separate service on the Consul host.

iptables -t nat -A PREROUTING -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600

iptables -t nat -A PREROUTING -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600

macOS setup

On macOS systems, you can use the macOS system resolver to point all .consul requests to Consul. Documentation for this feature is available via: man 5 resolver. You must add a resolver entry in /etc/resolver/ to point at Consul. If you do not have this folder, create it. To configure a resolver entry, you will need sudo/root access. Create a new file /etc/resolver/consul, and add the following text to the file:

nameserver 127.0.0.1
port 8600

nameserver 127.0.0.1
port 8600

The configuration informs the macOS resolver daemon to forward all .consul TLD requests to 127.0.0.1 on port 8600.

Query Consul DNS and BIND

First, add a service registration to your Consul agent. You can use the following example service configuration file and registration command.

{
  "service": {
    "name": "redis",
    "port": 80
  }
}

{
  "service": {
    "name": "redis",
    "port": 80
  }
}

$ curl --data @redis.json http://127.0.0.1:8500/v1/catalog/register

$ curl --data @redis.json http://127.0.0.1:8500/v1/catalog/register

To ensure that Consul DNS and your DNS server are configured correctly, use a dig query for the service you've registered.

$ dig @localhost -p 8600 primary.redis.service.dc1.consul. A

$ dig @localhost -p 8600 primary.redis.service.dc1.consul. A

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

Next, run the same query against your BIND instance and make sure you get a valid result:

$ dig @localhost -p 53 primary.redis.service.dc1.consul. A

$ dig @localhost -p 53 primary.redis.service.dc1.consul. A

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A

;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr  9 17:36:12 2014
;; MSG SIZE  rcvd: 76

If desired, verify reverse DNS using the same method:

$ dig @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR

$ dig @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR

; <<>> DiG 9.10.3-P3 <<>> @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3713
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;133.139.16.172.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
133.139.16.172.in-addr.arpa. 0  IN  PTR consul1.node.dc1.consul.

;; Query time: 3 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jan 31 04:25:39 UTC 2016
;; MSG SIZE  rcvd: 109

; <<>> DiG 9.10.3-P3 <<>> @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3713
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;133.139.16.172.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
133.139.16.172.in-addr.arpa. 0  IN  PTR consul1.node.dc1.consul.

;; Query time: 3 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jan 31 04:25:39 UTC 2016
;; MSG SIZE  rcvd: 109

$ dig @127.0.0.1 -p 8600 +short -x 172.16.139.133
consul1.node.dc1.consul.

$ dig @127.0.0.1 -p 8600 +short -x 172.16.139.133
consul1.node.dc1.consul.

Troubleshooting

If you don't get an answer from your DNS server (e.g., BIND, Dnsmasq) but you do get an answer from Consul, turn on your DNS server's query log to check for errors.

For BIND:

rndc querylog
tail -f /var/log/messages

rndc querylog
tail -f /var/log/messages

The log may show errors like this:

error (no valid RRSIG) resolving
error (no valid DS) resolving

error (no valid RRSIG) resolving
error (no valid DS) resolving

This indicates that DNSSEC is not disabled properly.

If you see errors about network connections, verify that there are no firewall or routing problems between the servers running BIND and Consul.

For Dnsmasq, see the log-queries configuration option and the USR1 signal.

DNS forwarding may fail if you use the default systemd-resolved configuration and attempt to bind to 0.0.0.0. The default configuration uses a DNS stub that listens for UDP and TCP requests at 127.0.0.53. As a result, attempting to bind to 127.0.0.53 conflicts with the running stub. You can disable the stub as described in the Using any local resolver with systemd section to troubleshoot this problem.

Next steps

In this tutorial, you configured DNS forwarding for Consul DNS and your DNS server. Now that you have configured your environment, you can start using Consul for service discovery or service mesh.

• Ensure only healthy service instances are available in Consul DNS.
• Deploy services into a zero trust network with Consul service mesh.