Skip to content

DNS should default to local Docker host settings #193

Open
@jpds

Description

@jpds

I'm running Drone within Kubernetes on Azure, and by default it seems keen to use Google DNS for the docker build container:

time="2018-09-11T14:15:45.926616198Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
time="2018-09-11T14:15:45.926647397Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"

This then fails at:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

If I set custom_dns to my kube-dns (10.0.0.10) service, everything works fine. Docker itself defaults to the host DNS settings:

https://docs.docker.com/config/containers/container-networking/#dns-services

I think this plugin should follow the default Docker behavior instead of defaulting to Google.

Activity

jpds

jpds commented on Sep 11, 2018

@jpds
Author

The DinD container on the agent itself has:

$ kubectl exec -it -n testing-drone drone-agent-64dd9fdf59-967rr -c drone-dind -- sh
/ # cat /etc/resolv.conf 
nameserver 10.0.0.10
search testing-drone.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
jpds

jpds commented on Sep 11, 2018

@jpds
Author

The above is with 17.12.0-ce-dind from the Helm chart here: https://github.com/helm/charts/tree/master/stable/drone/

Testing with 18.06.1-ce-dind now.

jpds

jpds commented on Sep 11, 2018

@jpds
Author

This needs #194 to update the Docker version used by the plugin.

bradrydzewski

bradrydzewski commented on Sep 11, 2018

@bradrydzewski
Member

I think this plugin should follow the default Docker behavior instead of defaulting to Google.

This plugin does follow the default docker behavior.

The reason that the agent and Docker container have different dns configurations are because they are attached to different networks. Your agent is using the default network driver (bridge or whatever), however, all drone pipeline steps are attached to a user-defined network:

docker network create foo
docker run --network=foo plugins/docker

The user-defined networks in Docker using an embedded DNS server [1] and therefore have different rules for DNS configuration than the default bridge network. So there really isn't anything we can do in this plugin, but you may be able to alter the host machine configuration in some way to ensure the correct DNS server is used inside the plugin (I'm not sure, though, this is outside my area of expertise).

Alternatively you can use the global environment variables Enterprise feature [2] and set PLUGIN_CUSTOM_DNS as a global environment variable, so that you do not need to set the value in your yaml file.

[1] https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/
[2] http://docs.drone.io/configure-global-environment/

jpds

jpds commented on Sep 11, 2018

@jpds
Author

This plugin does follow the default docker behavior.

Sure, this is what I agree should be happening, but none of my other Kubernetes containers are doing this and suddenly defaulting to going to Google DNS.

The person at #176 says that they had (undefined) issues with yum which echos the issues I'm seeing with apk in the DinD container, so perhaps the upgrade would fix this (I'm currently deploying a image of this plugin in my private repo to test this theory).

c2h5oh

c2h5oh commented on Feb 28, 2019

@c2h5oh

Btw I'm seeing the same without kubernetes. Docker container started in dind (I know it's not the best idea) also default to goodle dns and is unable to resolve pipeline services. - this is on 1.0 RC4

sneak

sneak commented on Feb 12, 2020

@sneak

I'm running into this issue today. It seems to me a major security issue to leak DNS hostname requests unencrypted to a third party server that the user never selected or configured! My LAN DNS server encrypts all DNS traffic to the WAN; using unencrypted DNS to unauthorized servers is incorrect behavior.


root@las1:~# docker images | grep plug
plugins/docker                      latest              29c70b6af6af        2 months ago        182MB
root@las1:~# docker rmi plugins/docker
Untagged: plugins/docker:latest
Untagged: plugins/docker@sha256:1d4743c50cfe06e84962a45337dc7ac200982c00c98d67852b54bf9475d37358
Deleted: sha256:29c70b6af6af2fb454b19ce8f7b51b35a5655c8398b6dd76938f6184ff7fb05a
Deleted: sha256:2b52935f5ac6ecad710c8ec05acfd172929022010e2fce6feee1d9c967f46013
Deleted: sha256:9bceba7972a6ac6471f2515c3dbf869abd8ff74964bb5ecdae340ad4554f7653
Deleted: sha256:f9d5a45945e1f93e2a2e8a81ac440f0309409d836d119de67dbd61572f7e8c2c
Deleted: sha256:01a062ec6974efa55446c81a9276f20d27d3635fe025d071a6637cfc2dfb13dc
Deleted: sha256:6a07e4cab25962d6b819f0508607d0c8d98e91121a83dbfbd6aec202eebded77
Deleted: sha256:e1bc276529a5de677bb7c41f28c76fce259b69fe35de7a8d1e443fc630c185e7
Deleted: sha256:6789d66fedded194adfa52ade2cf79aa4f84a244b58c2e2e57a85c5883f44b2d
Deleted: sha256:649f547f80b238767bc19dea8cf242adc2fa7164d3aff47b5e2a45dd873bdd6e
Deleted: sha256:1ee98fa99e925362ef980e651c5a685ad04cef41dd80df9be59f158cf9e52951
Deleted: sha256:78c8e55f8cb4c661582af874153f88c2587a034ee32d21cb57ac1fef51c6109e
Deleted: sha256:7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8
root@las1:~# date -u
Wed Feb 12 06:01:13 UTC 2020
root@las1:~#

... start build, wait a sec ...

root@las1:~# docker ps | head -2
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                PORTS                                                                      NAMES
4bceb265b144        plugins/docker:latest               "/usr/local/bin/dock…"   7 seconds ago       Up 3 seconds          2375/tcp                                                                   drone-xl5Q9qFh4kAv2TI9KEp8
root@las1:~#


root@las1:~# docker exec -ti drone-xl5Q9qFh4kAv2TI9KEp8 /bin/sh -c "docker run -ti ubuntu /bin/bash -c 'cat /etc/resolv.conf'"
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
5c939e3a4d10: Pull complete
c63719cdbe7a: Pull complete
19a861ea6baf: Pull complete
651c9d2d6c4f: Pull complete
Digest: sha256:8d31dad0c58f552e890d68bbfb735588b6b820a46e459672d96e585871acc110
Status: Downloaded newer image for ubuntu:latest

search CORRECT-DHCP-DOMAIN-NAME-REDACTED
options ndots:0

nameserver 8.8.8.8
nameserver 8.8.4.4

root@las1:~# docker exec -ti drone-xl5Q9qFh4kAv2TI9KEp8 /bin/sh -c "docker info | grep -i version"
Server Version: 18.09.0
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Kernel Version: 4.15.0-76-generic
WARNING: No swap limit support
root@las1:~#
ashwilliams1

ashwilliams1 commented on Feb 12, 2020

@ashwilliams1

This plugin uses the official docker-in-docker images as its base image, which starts docker-in-docker with default settings. The docker-in-docker deaemon is responsible for default DNS settings. If the docker-in-docker daemon is not using the correct settings or is not inheriting the settings from the parent container, and you think it should do so automatically, you should open an issue with the moby / docker team since they maintain the code responsible for DNS.

Alternatively you can configure custom DNS for the docker-in-docker plugin by globally setting the PLUGIN_CUSTOM_DNS variable. You can globally set environment variables for plugins using DRONE_RUNNER_ENVIRON.

tboerger

tboerger commented on Feb 12, 2020

@tboerger

IMHO this is a wontfix for this plugin, we are using the defaults of docker.

If this is an issue for you report it upstream.

mildebrandt

mildebrandt commented on Feb 12, 2020

@mildebrandt

While passing the issue upstream is the easy thing to do, I don't think it's the right thing in this case. People writing build scripts and using this plugin shouldn't need to be concerned about which DNS is being used. They expect each plugin to behave the same....if a network call works in one plugin, it should work in another.

The fact that this is performing a docker-in-docker is an implementation detail that the user should be isolated from knowing. The build engineer just wants to build a container.

If the solution is to set the PLUGIN_CUSTOM_DNS environment variable, that should be front and center in big bold letters as a prerequisite to using this plugin. If not, every new user that runs into this issue has to hop on Google and figure out why their container fails to build due to a server not being found.

Ideally, the plugin would pass its DNS settings into the child container every time.

ashwilliams1

ashwilliams1 commented on Feb 12, 2020

@ashwilliams1

ideally the plugin would pass its DNS settings into the child container every time.

how? If it were straight forward then I wonder why docker is not already doing this?

bradrydzewski

bradrydzewski commented on Feb 12, 2020

@bradrydzewski
Member

While passing the issue upstream is the easy thing to do, I don't think it's the right thing

This assumes there is something we can do in this codebase to solve this problem. It is not yet clear to me that we can solve this issue without docker's help but I am certainly open to suggestions.

For reference this is the existing issue for dns issues in docker-in-docker:
moby/moby#20037

mildebrandt

mildebrandt commented on Feb 12, 2020

@mildebrandt

The issue stems from the fact that drone uses a custom network which kicks-in docker's DNS service which isn't available to dind containers.

I also see that Drone treats certain containers as special already, including this one:
https://github.com/drone/drone/blob/d84b79f027247284d26066041b1521414f68c631/operator/runner/runner.go#L255

So, how about creating another special category which passes the host's DNS settings into a plugin via --dns and related switches? That way the child containers will get the proper settings since the nameserver wouldn't be a local address.

blopker

blopker commented on Feb 12, 2020

@blopker

I'd also be interested in a Docker plugin that uses the host's daemon instead of docker-in-docker. For self hosted deployments this would have a few benefits, like having a persistent layer cache and it would be easier to configure /etc/docker/daemon.json.

16 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @tboerger@sneak@c2h5oh@blopker@bradrydzewski

        Issue actions

          DNS should default to local Docker host settings · Issue #193 · drone-plugins/drone-docker