Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core DNS is not resolving external ip #2324

Closed
alok87 opened this issue Nov 19, 2018 · 16 comments
Closed

Core DNS is not resolving external ip #2324

alok87 opened this issue Nov 19, 2018 · 16 comments

Comments

@alok87
Copy link

alok87 commented Nov 19, 2018

We have a kubernetes service that points to an external service ip. External service IP is the ec2 instance internal IP address where the mysql service is running at 3306

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                       AGE
mysql                ExternalName   <none>           172.31.21.116   3306/TCP                      20m
  • When using CORE DNS, the external IP service do not resolve
/ # nslookup mysql.letmetry
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'mysql.letmetry'
  • When using KUBE DNS, the external IP service resolves
/ # nslookup mysql.letmetry.svc.cluster.local.
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mysql.letmetry.svc.cluster.local.
Address 1: 172.31.21.116 ip-172-31-21-116.ap-south-1.compute.internal

Please suggest why this is happening.
The logs of CORE DNS when this request is made -

100.96.109.30:41248 - [19/Nov/2018:14:56:25 +0000] 4 "AAAA IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 110.391µs
100.96.109.30:42034 - [19/Nov/2018:14:56:25 +0000] 5 "AAAA IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 91.647µs
100.96.109.30:41174 - [19/Nov/2018:14:56:25 +0000] 6 "AAAA IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 93.808µs
100.96.109.30:34985 - [19/Nov/2018:14:56:25 +0000] 7 "AAAA IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.458123ms
100.96.109.30:45287 - [19/Nov/2018:14:56:25 +0000] 8 "A IN mysql.letmetry. udp 32 false 512" NXDOMAIN qr,rd,ra 108 102.671µs
100.96.109.30:36956 - [19/Nov/2018:14:56:25 +0000] 9 "A IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 91.6µs
100.96.109.30:51749 - [19/Nov/2018:14:56:25 +0000] 10 "A IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 92.572µs
100.96.109.30:54959 - [19/Nov/2018:14:56:25 +0000] 11 "A IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 87.971µs
100.96.109.30:42924 - [19/Nov/2018:14:56:25 +0000] 12 "A IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.420079ms

System info

  • It is a default installation of Core DNS with kops. No custom configuration was made. We just tried to replace kube dns with core dns using kops and are stuck here.
  • image: gcr.io/google_containers/coredns:1.1.3
  • kubernetes version: 1.9.8
  • kops version: 1.10.0
  • CoreFile
Corefile: |-
    .:53 {
        errors
        log
        health
        kubernetes cluster.local. in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
    }
@chrisohaver
Copy link
Member

Should work. If you use the fully qualified name with CoreDNS (as you did with kube-dns), do you get an answer?

@chrisohaver
Copy link
Member

What does the definition of the external name service look like?

kubectl -n letmetry get service mysql -o yaml

@alok87
Copy link
Author

alok87 commented Nov 19, 2018

  • kube dns works without FQDN
/ # nslookup mysql.letmetry
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mysql.letmetry
Address 1: 172.31.21.116 ip-172-31-21-116.ap-south-1.compute.internal
  • Output of kubectl -n letmetry get service mysql -o yaml
$ kubectl -n letmetry get service mysql -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-11-19T16:42:39Z
  name: mysql
  namespace: letmetry
  resourceVersion: "105997025"
  selfLink: /api/v1/namespaces/letmetry/services/mysql
  uid: 205f30f2-ec1a-11e8-8bba-02ebb8238b60
spec:
  externalName: 172.31.21.116
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}
  • Core dns does not work with FQDN also

@chrisohaver
Copy link
Member

OK, the issue is, I believe, that the external name field is supposed to hold a DNS name, not an IP. For example, per the Cluster DNS spec, this field is expected to be a DNS name because it must be used to construct a CNAME record (which can only contain a DNS name target).

If you want to point a cluster service to an external IP, you can do so with a headless service and static endpoint.

@chrisohaver
Copy link
Member

Another explanation here, with an example of a headless service and static endpoint.

@alok87
Copy link
Author

alok87 commented Nov 19, 2018

thanks @chrisohaver let me try that and get back !
Also, if it works for kube-dns then it should work for core-dns as well to maintain the backward compatibility.

@chrisohaver
Copy link
Member

chrisohaver commented Nov 19, 2018

Also, if it works for kube-dns then it should work for core-dns as well to maintain the backward compatibility.

Perhaps. Although strictly speaking, kube-dns is violating the Cluster DNS Spec in behaving that way.

IMO, it's not intended that the ExternalName field should be able to contain an IP address. All documentation I can find assumes it will be a name, not an IP. The API should probably not allow IPs in that field.

Diving into the k8s API validation code, it looks like the Service validation is letting IPv4s slip through because it thinks they are domain names (after all, an IPV4 address is a series of valid dns labels delimited by dots). For example, looking at the code, an IPv6 address would fail validation because they contain colons. In other words, the API appears to be accepting an IPv4 address as a domain name, not as an IP. This is the regex the API uses ... [a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (used in IsDNS1123Subdomain() which is used when validating the externalName field for ExternalName services).

@johnbelamaric
Copy link
Member

We should release note this in K8s (@rajansandeep ?) as a change in behavior, but CoreDNS is working according to the spec. It's actually a bug in kube-dns - an accident of the skydns code and the missing validations.

@nikopen
Copy link

nikopen commented Nov 26, 2018

Got into this "issue" from the ingress-nginx controller perspective - version 0.18.0 or later outputs the following line:

2018/11/19 15:53:42 [error] 110379#110379: *24398057 [lua] dns.lua:61: resolve(): server returned error code: 3: name error, context: ngx.timer

This happens when any externalName is an IP, and it tries to query the nameserver with the IP as an A record.

Both CoreDNS and ingress-nginx are working according to spec, and this detail should be in the official Kubernetes docs for clarity.

@nvenky
Copy link

nvenky commented Apr 17, 2019

Got into this "issue" from the ingress-nginx controller perspective - version 0.18.0 or later outputs the following line:

2018/11/19 15:53:42 [error] 110379#110379: *24398057 [lua] dns.lua:61: resolve(): server returned error code: 3: name error, context: ngx.timer

This happens when any externalName is an IP, and it tries to query the nameserver with the IP as an A record.

Both CoreDNS and ingress-nginx are working according to spec, and this detail should be in the official Kubernetes docs for clarity.

@nikopen - We are facing the same issue. We created the k8s cluster with Kubeadm and deployed nginx-ingress through helm chart. Can you please let me know how you fixed this issue?

@nikopen
Copy link

nikopen commented Apr 17, 2019

@nvenky this happens when you have some Services with externalName as an IP instead of a hostname, as mentioned above.

Turn the IPs into hostnames to fix this.

@sylr
Copy link
Contributor

sylr commented Apr 30, 2019

I too am using externalname services with internal IP addresses.

The fact that coredns is not able as kubernetes to resolve them is causing me problems.

@chrisohaver
Copy link
Member

I too am using externalname services with internal IP addresses.

In Kubernetes, IP addresses are not supposed to be allowed in the in ExternalName services. Kubernetes only permits FQDNs, as such an IP would be interpreted as a FQDN not an IP, resulting in unexpected behavior. IMO Kubernetes field validation could be improved here, by screening out numeric TLDs.

sylr pushed a commit to sylr/coredns that referenced this issue Apr 30, 2019
Fix coredns#2324

Signed-off-by: Sylvain Rabot <s.rabot@lectra.com>
sylr pushed a commit to sylr/coredns that referenced this issue Apr 30, 2019
Fix coredns#2324

Signed-off-by: Sylvain Rabot <s.rabot@lectra.com>
@sylr
Copy link
Contributor

sylr commented Apr 30, 2019

I understand but this is something kube-dns users who were not aware of the spec and who build something on top of externalname resolving an IP will appreciate.

@nikopen
Copy link

nikopen commented Apr 30, 2019

@sylr This is the Kubernetes spec and CoreDNS follows it as intended. Docs have been equally updated to make this detail visible.

If you want to push for this change, a good place for that would be SIG-Network in Kubernetes. If accepted and implemented there, only then I assume CoreDNS would propagate the change.

@sylr
Copy link
Contributor

sylr commented Apr 30, 2019

This is the Kubernetes spec and CoreDNS follows it as intended.

Not respecting the spec and going beyond it are not the same thing. You'll only make the transition smoother for kube-dns users like myself who unwillingly got into this situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants