How to fix Docker container not resolving domain names

Friday 28 June 2024

How to fix Docker container not resolving domain names

I had a case where I run a Terraform Docker container but it would fail to reach a provider package repository.

docker-compose.yaml:

---

name: terraform

services:

terraform:

image: hashicorp/terraform:latest

volumes:

- .:/infra

working_dir: /infra

I wanted to execute (terraform) init command (terraform argument here refers to the name of the service, not the terraform executable itself):

$ docker compose run --rm terraform init

[+] Creating 1/1

✔ Network import_demo_default Created 0.1s

[+] Running 5/5

✔ terraform Pulled 11.2s

✔ ec99f8b99825 Pull complete 4.2s

✔ 47bfda048af5 Pull complete 8.4s

✔ 755b9030e6bd Pull complete 8.4s

✔ db586b81a2dc Pull complete 9.4s

Initializing the backend...

Initializing provider plugins...

- Finding kreuzwerker/docker versions matching "3.0.2"...

╷

│ Error: Failed to query available provider packages

│

│ Could not retrieve the list of available versions for provider kreuzwerker/docker: could not connect to registry.terraform.io: failed to request discovery document: Get

│ "https://registry.terraform.io/.well-known/terraform.json": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

╵

I changed entrypoint in order to execute debugging tools. With either

entrypoint: ["wget", "https://registry.terraform.io/.well-known/terraform.json"]

entrypoint: ["ping", "registry.terraform.io"]

$ docker compose run --rm terraform

returned:

bad address 'registry.terraform.io'

...and

entrypoint: ["nslookup", "registry.terraform.io"]

returned:

;; connection timed out; no servers could be reached

To check the DNS servers used I set entrypoint to print the resolv.conf file:

entrypoint: ["cat", "/etc/resolv.conf"]

This returned:

# Generated by Docker Engine.

# This file can be edited; Docker Engine will not make further changes once it

# has been modified.

nameserver 127.0.0.11

search bigcorp.com

options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)

# ExtServers: [10.10.1.255 10.11.5.183]

# Overrides: [nameservers search]

# Option ndots from: internal

By default, Docker provides a DNS server (daemon's embedded DNS resolver) at 127.0. 0.11, so all DNS requests from containers come to it. Daemon then forwards these requests to uplink DNS servers as defined via --dns arguments, /etc/docker/daemon.json or host's /etc/resolv.conf.

From Networking overview | Docker Docs:

Containers use the same DNS servers as the host by default, but you can override this with --dns.

By default, containers inherit the DNS settings as defined in the /etc/resolv.conf configuration file. Containers that attach to the default bridge network receive a copy of this file. Containers that attach to a custom network use Docker's embedded DNS server. The embedded DNS server forwards external DNS lookups to the DNS servers configured on the host.

Using --dns is the same as adding dns attribute to /etc/docker/daemon.json. Same applies for --dns-search. DNS settings in /etc/docker/daemon.json will override those set in the local /etc/resolv.conf file.

My local /etc/resolv.conf file:

$ cat /etc/resolv.conf

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).

# Do not edit.

# This file might be symlinked as /etc/resolv.conf. If you're looking at

# /etc/resolv.conf and seeing this text, you have followed the symlink.

# This is a dynamic resolv.conf file for connecting local clients to the

# internal DNS stub resolver of systemd-resolved. This file lists all

# configured search domains.

# Run "resolvectl status" to see details about the uplink DNS servers

# currently in use.

# Third party programs should typically not access this file directly, but only

# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a

# different way, replace this symlink by a static file or a different symlink.

# See man:systemd-resolved.service(8) for details about the supported modes of

# operation for /etc/resolv.conf.

nameserver 127.0.0.53

options edns0 trust-ad

search Home

In my case, uplink DNS server is my local router:

$ resolvectl status

Global

Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

resolv.conf mode: stub

Link 2 (enp0s31f6)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)

Current Scopes: DNS

Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Current DNS Server: xxxx:a25c:xxxx:0:xxxx:7aff:fe4d:3700

DNS Servers: 192.168.0.1 xxxx:a25c:xxxx:0:xxxx:7aff:fe4d:3700

DNS Domain: Home

Link 4 (br-a7ba833104f5)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (br-d39e3c16b90f)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (docker0)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 7 (br-1d4f7fd2e5cc)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 8 (br-3c8c9487a095)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 9 (br-7bfedc7c4369)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 26 (veth846e490)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 27 (br-c06da6a5a65a)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 30 (br-c1e0d2aed078)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 71 (enxa44cc8e41d0f)

Current Scopes: none

Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

I discovered that I had DNS settings set in /etc/docker/daemon.json:

$ cat /etc/docker/daemon.json

{

"dns": ["10.10.1.255", "10.11.5.183"],

"dns-search": ["bigcorp.com"]

}

As there is no need to use these custom (corporate) DNS servers, I can remove these settings (basically empty) /etc/docker/daemon.json.

To reload the new (empty) config, I had to flush changes and restart Docker:

$ sudo systemctl daemon-reload

$ sudo systemctl restart docker

Let's check how container's /etc/resolv.conf changed:

$ docker compose run --rm terraform

# Generated by Docker Engine.

# This file can be edited; Docker Engine will not make further changes once it

# has been modified.

nameserver 127.0.0.11

search Home

options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)

# ExtServers: [host(127.0.0.53)]

# Overrides: []

# Option ndots from: internal

Switching entrypoint to nslookup:

entrypoint: ["nslookup", "registry.terraform.io"]

...gives now the expected result:

$ docker compose run --rm terraform

Server: 127.0.0.11

Address: 127.0.0.11:53

Non-authoritative answer:

registry.terraform.io canonical name = d3rdzqodp6w8cx.cloudfront.net