Skip to content

Cannot receive UDP traffic in Docker container in case of UDP server being located at Docker host #15127

Open
@igorastds

Description

@igorastds

Description of problem:

When issuing UDP connections from docker container to docker host, no incoming UDP packets are received in container.
UDP works for accessing external resources.
TCP works for accessing docker host.

docker version:

Mostly tested on

Client:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   b900aaa
 Built:        Fri Jul 17 15:15:47 UTC 2015

Still actual in:

 Go version:   go1.4.2
 Git commit:   9d3ad6d
 Built:        Wed Jul 29 16:26:04 UTC 2015

docker info:

Containers: 283
Images: 687
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.2███████
Operating System: Gentoo/Linux
CPUs: 8
Total Memory: 7.746 GiB
Name: ███████
ID: 7WYK:23Z2:J7KR:JFF2:QMLD:E5Z6:FF5O:LAAA:XOW4:56ZR:BOSD:WWAI

uname -a: Linux ███████ 4.1.2-███████ #2 SMP Mon Jul 20 14:13:03 CEST 2015 x86_64 Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz GenuineIntel GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.): physical

How reproducible:

Steps to Reproduce:

  1. you need to start any UDP echo server (nmap's ncat works just fine: ncat -e /bin/cat -k -u -l 5151, also I rechecked with udpqotd, yeah that one from old Perl Cookbook) on Docker host outside of Docker (not tested starting server in docker container)
  2. you need to test it with "netcat -u ip port" to see if you receive incoming traffic
  3. you need to start "netcat -u ip port" in your favorite minimal-shell container and try accessing host, then verify your results by testing various TCP services and UDP echo services located outside of your docker host.

Actual Results:

  • No incoming traffic received from UDP server when running in on docker host
  • Incoming traffic successfully received from UDP server when running in on outside host

Expected Results:

  • Incoming traffic received into docker from UDP server running in on docker host; it doesn't looks like a security problem because host is still accessible via TCP.

Additional info:

  • test container baseimage ubuntu:14.04
  • firewalls are off, accessing TCP ports at docker host works
  • no --net or flags given . Routing/traces seem to be ok. No --userland-proxy option is set in daemon config.
  • no sysctl changes done; reboot was performed to verify results.
  • UPD: confirms on docker 1.5, docker 1.6.
  • UPD: port is accessible, when accessing host by docker0 ip, but I need to access it by host host IP.

Activity

chpio

chpio commented on Jan 16, 2016

@chpio

same here with docker v1.9.1 ("a34a1d5-dirty") @ Arch

👍 please fix this bug

chpio

chpio commented on Jan 17, 2016

@chpio

is there a workaround for this?


maybe it's an iptables bug. i have no other iptables rules, besides the docker rules, 2 networks (docker0 + 1 other). My host has the ip 172.16.0.1/16.

# iptables-save 
# Generated by iptables-save v1.4.21 on Sun Jan 17 10:28:50 2016
*nat
:PREROUTING ACCEPT [8794:1217326]
:INPUT ACCEPT [8773:1215606]
:OUTPUT ACCEPT [17899:2751833]
:POSTROUTING ACCEPT [17898:2752201]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-bcfe92869fd5 -j MASQUERADE
COMMIT
# Completed on Sun Jan 17 10:28:50 2016
# Generated by iptables-save v1.4.21 on Sun Jan 17 10:28:50 2016
*filter
:INPUT ACCEPT [1364822:334302629]
:FORWARD ACCEPT [13:988]
:OUTPUT ACCEPT [1227632:7107850879]
:DOCKER - [0:0]
-A FORWARD -s 172.19.0.0/16 -d 172.17.0.0/16 -j DROP
-A FORWARD -s 172.17.0.0/16 -d 172.19.0.0/16 -j DROP
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-bcfe92869fd5 -j DOCKER
-A FORWARD -o br-bcfe92869fd5 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br-bcfe92869fd5 ! -o br-bcfe92869fd5 -j ACCEPT
-A FORWARD -i br-bcfe92869fd5 -o br-bcfe92869fd5 -j ACCEPT
COMMIT
# Completed on Sun Jan 17 10:28:50 2016
LK4D4

LK4D4 commented on Oct 18, 2016

@LK4D4
Contributor

ping @mrjana @sanimej @aboch
Could you pls try and close or fix issue?

timothyklim

timothyklim commented on May 13, 2017

@timothyklim

Unbelievable, bug still in current docker version and no one cares from docker dev team (@thaJeztah).

mavenugo

mavenugo commented on May 13, 2017

@mavenugo
Contributor

@timothyklim sorry about that. this fell out of radar. We will take a look at it soon.

self-assigned this
on May 13, 2017
timothyklim

timothyklim commented on Jun 21, 2017

@timothyklim

1 month later and still no progress

chrisxaustin

chrisxaustin commented on Jun 28, 2017

@chrisxaustin

I'm having a [possibly] related issue.

My app receives UDP syslog messages from most sources with the original source address preserved, but one device has the source rewritten to the docker0 interface's IP (seen by running tcpdump on docker0).

I tried disabling the userland proxy as an experiment. The ones that were working before still are, but that one source never makes it to docker0 now, which is obviously concerning.

Ubuntu 16.04.2, docker 1.12.6, docker-compose 1.8.0, launched w/ docker-compose up

EDIT: Running conntrack -D -p udp seems to have fixed it for me locally, so potentially unrelated.

cizixs

cizixs commented on Aug 21, 2017

@cizixs
Contributor

The problem happens because host has multiple network interfaces(eth0 and docker0 in this case), and when replying UDP, server choose the wrong IP as the source address.

Running tcpdump to capture packets on the wire gives:

14:42:11.861765 IP 172.17.0.3.37218 > 172.16.13.13.5151: UDP, length 6
14:42:11.863471 IP 172.17.0.1.5151 > 172.17.0.3.37218: UDP, length 6
14:42:11.863520 IP 172.17.0.3 > 172.17.0.1: ICMP 172.17.0.3 udp port 37218 unreachable, length 42
  • The first packet means client sends a UDP request to server, with source address 172.17.0.3(container IP address), and destination address 172.16.13.13(host eth0 address)
  • The second packets is the response, however, source address becomes 172.17.0.1(docker0 ip address), instead of 172.16.13.13 as we expected
  • The third packet is sent by client, telling server that the second packet is INVALID

As we can see the real cause is: the second packet chooses the wrong source address! And somehow, client does not recognize it, thinks it is INVALID, does not pass it to the application, but instead replies with a ICMP packet.

Choosing packet source address is kernel's work, for UDP, by default it will use the IP address of the interface the packet will be sent to. And as to why kernel drops the second packet, is because netfilter records valid UDP connection(since UDP is connectionless, it only means netfilter stores <sourcess IP, source Port, destination IP, destination Port>), and drops those that are invalid(not in connection table).

As for the first problem, socket can actually control which source address should be used with a combination of sendmsg and IP_PKTINFO socket option. See how powerDNS does it: https://blog.powerdns.com/2012/10/08/on-binding-datagram-udp-sockets-to-the-any-addresses/

In conclusion, this is not a problem caused by docker(although it does create a docker0 which is the beginning of it) . And the solution comes in many ways:

lagorsse

lagorsse commented on Jul 24, 2018

@lagorsse

I don't know if my issue is link or not but I can't send or receiver udp message in the docker container, works fine with tcp.

nc -l -u -p 8888
nc -u 127.0.0.1 8888

I am using
EXPOSE 8888/udp
or -p 8888:8888/udp

Netcat works fine with tcp so I don't get why it doesn't work over udp.
I am using osx sierra.

REPTILEHAUS

REPTILEHAUS commented on Aug 31, 2018

@REPTILEHAUS

Also hitting this issue.. any fix for this ?

running osx Sierra too

donovankim

donovankim commented on Nov 28, 2018

@donovankim

35 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @LK4D4@victorli@YodaDaCoda@timothyklim@chrisxaustin

      Issue actions

        Cannot receive UDP traffic in Docker container in case of UDP server being located at Docker host · Issue #15127 · moby/moby