Skip to content

Ubuntu 18.04 + Docker 17.12.1-ce break DNS resolution #2187

Closed
@mprobst

Description

@mprobst

This might be a dupe of #1654 (which is closed - but then this is still happening). /CC @sanimej who's been looking at that last year.

The default installation of Docker on Ubuntu 18.04 is broken if Google DNS servers 8.8.8.8 and 8.8.4.4 are not reachable for some reason.

Out of the box Ubuntu runs systemd, which creates a nameserver running on localhost:

$ docker --version
Docker version 17.12.1-ce, build 7390fc6
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]

nameserver 127.0.0.53

Docker ignores the localhost nameserver:

$ docker run  -ti busybox cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]
nameserver 8.8.8.8
nameserver 8.8.4.4

Because on my network, 8.8.8.8 is not reachable for some reason (different bug), name lookups time out:

$ docker run  -ti busybox nslookup www.google.com
Server:    8.8.8.8
Address 1: 8.8.8.8

nslookup: can't resolve 'www.google.com'

For completeness sake, the symptom that lead me to find this is that when building a Go based image and running go get in a build step, I get a Could not resolve host: github.com (took me a while to piece this one together...):

Step 6/9 : RUN go get ./
 ---> Running in f64c6b120d88
# cd .; git clone https://github.com/julienschmidt/httprouter /go/src/github.com/julienschmidt/httprouter
Cloning into '/go/src/github.com/julienschmidt/httprouter'...
fatal: unable to access 'https://github.com/julienschmidt/httprouter/': Could not resolve host: github.com
package github.com/julienschmidt/httprouter: exit status 128

Activity

fcrisciani

fcrisciani commented on Jun 18, 2018

@fcrisciani

@mprobst in case of standalone container, you are in control on which dns to use, if you don't specify anything you will get by default the google ones. if 8.8.8.8 does not work for you, you can specify others with --dns as per https://docs.docker.com/engine/reference/commandline/run

mprobst

mprobst commented on Jun 19, 2018

@mprobst
Author

@fcrisciani my host system has a /etc/resolv.conf that has a working entry for name resolution. Docker ignores this, and then fails at runtime. That does seem like a bug to me. Can you clarify? It does seem very surprising.

If this is how Docker does and will behave, I guess this is an upstream bug in Ubuntu's package setup.

fcrisciani

fcrisciani commented on Jun 19, 2018

@fcrisciani

@mprobst
a container spawned with docker run and not networking options will also create a network namespace, isolating it from the host namespace. this will not allow the container to communicate with any of the host resources, in this case the 127.0.0.53 dns.

comdw

comdw commented on Jun 22, 2018

@comdw

I agree there is an issue here somewhere...

I have docker 18.03.0-ce running on Ubuntu 16.04 with custom DNS entries in /etc/resolv.conf, e.g.: nameserver 172.99.1.7, and in this environment if I look at /etc/resolv.conf inside a container I see the same thing: nameserver 172.99.1.7

Now on a new machine running docker 18.05.0-ce on Ubuntu 18.04 with the same DNS setup but configured with netplan (which is part of 18.04 changes). The /etc/resolv.conf file should not be edited by hand and uses systemd-resolved - it has an internal nameserver 127.0.0.53 entry in /etc/resolv.conf. Now inside the container I don't see this entry copied but instead get the default google ones (8.8.8.8, 8.8.4.4).

Clearly Ubuntu is different w.r.t. DNS in 18.04, but I haven't confirmed whether Docker is also behaving differently, or just defaults to Google DNS because it somehow knows that 127.0.0.53 is not going to work inside the container.

My workaround for now is to use the --dns option when starting containers to manually specify the hosts, but I would rather the DNS inheritance from the host continued to work!

Documentation ref:
https://docs.docker.com/config/containers/container-networking/#dns-services
"By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis"

Related issue?
#2068

diegoquintanav

diegoquintanav commented on Jul 4, 2018

@diegoquintanav

Probably related moby/moby#36153
At that moment it was with ubuntu 16.04, now with 18.04

If it's worth something, I posted in SO with updated information https://stackoverflow.com/questions/51105875/internet-connection-not-working-networkmanager-not-working-after-installing-dock

ptink

ptink commented on Jul 6, 2018

@ptink

I really don't think this issue should have been closed- docker is broken out of the box on any corporate network, and in a non-obvious way that will usually require the hunting down of this issue in order to find a work around.

Would it not be possible for docker to detect the presence of nameserver 127.0.0.53 in resolv.conf in a similar way to how systemd-resolve does and use /run/systemd/resolv/resolv.conf (which contains the actual DNS nameserver rather than the local) instead?

kaneg

kaneg commented on Jul 9, 2018

@kaneg

I also encountered the same issue after upgrading to Ubuntu 18.04.
The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google's public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way.
I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

ctelfer

ctelfer commented on Jul 11, 2018

@ctelfer
Contributor

I looked into this a bit deeper to add some color as to what is going on here. Docker spawns a resolver for all user-defined networks. From moby/moby:

        if !containertypes.NetworkMode(n.Name()).IsUserDefined() {
                createOptions = append(createOptions, libnetwork.CreateOptionDisableResolution())
        }    

and from libnetwork:

func CreateOptionDisableResolution() EndpointOption {
        return func(ep *endpoint) {
                ep.disableResolution = true
        }
}
...
func (ep *endpoint) needResolver() bool {
        ep.Lock()
        defer ep.Unlock()
        return !ep.disableResolution
}
...
func (sb *sandbox) populateNetworkResources(ep *endpoint) error {
...
        if ep.needResolver() {
                sb.startResolver(false)
        }

So for non-user-defined networks (e.g. bridge) docker/libnetwork specifically does not start any sort of proxying resolver. I verified this experimentally by creating a fresh bridge network and running docker run -it --network=mynet --privileged nicolaka/netshoot iptables -L -t nat -n and seeing the IPtables redirects to the proxy resolver.

...
Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:40937
DNAT       udp  --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:57704
...

Running the same command without the --network=mynet (putting the container in the default bridge network) will show an empty set of iptables rules.

Containers without resolver proxying have no way to reach the host's 127.0.0.* network. As @fcrisciani said above, any network traffic directed to such an address will go to the container's network namespace's loopback address which is not plumbed to answer such requests. This is why libnetwork specifically filters all 127.* nameservers from the generated /etc/resolv.conf that goes in each container. Docker engine reads the (configurably-located if I read the code correctly) resolv.conf file for the system, but then filters out those localhost nameservers in the resolv.conf that it generates for the container because it knows the container can't reach them.

This is why @comdw saw the name server's being copied in, but others have not in ubuntu 18.04. A simple workaround if your system is unable to reach 8.8.8.8 / 8.8.4.4 (the defaults provided when no other host resolution seems possible) may well be to place your containers in a user-defined bridge network rather than the default bridge. (Haven't tested this yet on 18.04, though. If someone would that would be helpful.)

Given how annoying this would be in general, the right solution would seem to be a change of policy (or an override) in moby to re-enable proxy resolution for the bridge network. But it doesn't look like something we can fix from libnetwork ... at least not cleanly.

mushkevych

mushkevych commented on Aug 15, 2018

@mushkevych

As this thread is referenced by Google, let me provide "how-to" on resolving the issue with the docker builds on Ubuntu 18.04.

  • find out the DNS servers with command (your IP addressed will be different):
$ nmcli dev show | grep DNS
IP4.DNS[1]:                             10.11.12.13
IP4.DNS[2]:                             10.11.12.14
  • register these IPs with the /etc/resolv.conf:
sudo -s
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail
  • reboot the OS and check whether the IPs have been registered:
cat /etc/resolv.conf   
bmurphy1976

bmurphy1976 commented on Aug 16, 2018

@bmurphy1976

We solved this by adding the following to every docker run command. Using this approach we at least didn't have to modify any system wide files on our file system.

--dns `cat /run/systemd/resolve/resolv.conf | grep nameserver | head -n 1 | awk '{print $2}'`

The fact that we had to do this on all of our Ubuntu 18.04 servers is obnoxious.

34 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @carletes@mprobst@yaleman@bmurphy1976@kenden

        Issue actions

          Ubuntu 18.04 + Docker 17.12.1-ce break DNS resolution · Issue #2187 · moby/libnetwork