Skip to content

Ingress not working when kubernetes.io/ingress.class used #2349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
web3rover opened this issue Apr 14, 2018 · 11 comments
Closed

Ingress not working when kubernetes.io/ingress.class used #2349

web3rover opened this issue Apr 14, 2018 · 11 comments
Assignees
@antoineco

Comments

@web3rover
Copy link

NGINX Ingress controller version:
NGINX Ingress Controller version: 0.12.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

What happened:

I want to have multiple ingress controllers. Each pointing to a different backend service. I am trying to use kubernetes.io/ingress.class class annotation but it's giving 502 error. If I remove kubernetes.io/ingress.class then it works all fine.

Here is my deployment file:

{
                                                "apiVersion":"apps/v1beta1",
                                                "kind":"Deployment",
                                                "metadata":{
                                                    "name":instanceId + "-firewall"
                                                },
                                                "spec":{
                                                    "replicas":1,
                                                    "template":{
                                                        "metadata":{
                                                            "labels":{
                                                                "app":"ingress-nginx-" + instanceId
                                                            }
                                                        },
                                                        "spec":{
                                                            "terminationGracePeriodSeconds":60,
                                                            "containers":[
                                                                {
                                                                    "image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.12.0",
                                                                    "name":"ingress-nginx",
                                                                    "imagePullPolicy":"Always",
                                                                    "ports":[
                                                                        {
                                                                            "name":"http",
                                                                            "containerPort":80,
                                                                            "protocol":"TCP"
                                                                        }
                                                                    ],
                                                                    "livenessProbe":{
                                                                        "httpGet":{
                                                                            "path":"/healthz",
                                                                            "port":10254,
                                                                            "scheme":"HTTP"
                                                                        },
                                                                        "initialDelaySeconds":30,
                                                                        "timeoutSeconds":5
                                                                    },
                                                                    "env":[
                                                                        {
                                                                            "name":"POD_NAME",
                                                                            "valueFrom":{
                                                                                "fieldRef":{
                                                                                    "fieldPath":"metadata.name"
                                                                                }
                                                                            }
                                                                        },
                                                                        {
                                                                            "name":"POD_NAMESPACE",
                                                                            "valueFrom":{
                                                                                "fieldRef":{
                                                                                    "fieldPath":"metadata.namespace"
                                                                                }
                                                                            }
                                                                        }
                                                                    ],
                                                                    "args":[
                                                                        "/nginx-ingress-controller",
                                                                        "--ingress-class=" + instanceId + "-firewall",
                                                                        "--default-backend-service=$(POD_NAMESPACE)/default-http-backend",
                                                                        "-v=4"
                                                                    ]
                                                                }
                                                            ]
                                                        }
                                                    }
                                                }

And here is my ingress:

{
                                                                        "apiVersion":"extensions/v1beta1",
                                                                        "kind":"Ingress",
                                                                        "metadata":{
                                                                            "name":"ingress-" + instanceId,
                                                                            "annotations":{
                                                                                "nginx.ingress.kubernetes.io/rewrite-target":"/",
                                                                                "ingress.kubernetes.io/ssl-redirect":"false",
                                                                                "nginx.ingress.kubernetes.io/auth-type":"basic",
                                                                                "nginx.ingress.kubernetes.io/auth-secret":"basic-auth-" + instanceId,
                                                                                "nginx.ingress.kubernetes.io/auth-realm":"Authentication Required",
                                                                                "kubernetes.io/ingress.class": instanceId + "-firewall"
                                                                            }
                                                                        },
                                                                        "spec":{
                                                                            "rules":[
                                                                                {
                                                                                    "http":{
                                                                                        "paths":[
                                                                                            {
                                                                                                "path":"/jsonRPC",
                                                                                                "backend":{
                                                                                                    "serviceName": instanceId,
                                                                                                    "servicePort":8545
                                                                                                }
                                                                                            }
                                                                                        ]
                                                                                    }
                                                                                }
                                                                            ]
                                                                        }
                                                                    }

Here is the logs:

UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"ip-172-31-24-33_b7cb02d3-3dbc-11e8-b95e-022243d57c40","leaseDurationSeconds":15,"acquireTime":"2018-04-11T19:16:02Z","renewTime":"2018-04-14T14:32:51Z","leaderTransitions":3},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713838       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:d88843d3-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:313521,Generation:0,CreationTimestamp:2018-04-11 19:09:23 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"ip-172-31-24-33_b7cb02d3-3dbc-11e8-b95e-022243d57c40","leaseDurationSeconds":15,"acquireTime":"2018-04-11T19:16:02Z","renewTime":"2018-04-14T14:32:51Z","leaderTransitions":3},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713861       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:d8344a25-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:313519,Generation:0,CreationTimestamp:2018-04-11 19:09:23 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"ip-172-31-24-33_b893a60f-3dbc-11e8-a6ae-022243d57c40","leaseDurationSeconds":15,"acquireTime":"2018-04-11T19:16:03Z","renewTime":"2018-04-14T14:32:50Z","leaderTransitions":2},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713884       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:d8344a25-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:313519,Generation:0,CreationTimestamp:2018-04-11 19:09:23 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"ip-172-31-24-33_b893a60f-3dbc-11e8-a6ae-022243d57c40","leaseDurationSeconds":15,"acquireTime":"2018-04-11T19:16:03Z","renewTime":"2018-04-14T14:32:50Z","leaderTransitions":2},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713902       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:mongo,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/mongo,UID:847a31ea-3dbd-11e8-b0ef-022243d57c40,ResourceVersion:1578,Generation:0,CreationTimestamp:2018-04-11 19:21:21 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{name: mongo,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.18  0xc420379f50 ObjectReference{Kind:Pod,Namespace:default,Name:mongo-controller-bwjvp,UID:7f5f40c3-3dbd-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1576,FieldPath:,}}] [] [{ 27017 TCP}]}],}
I0414 14:32:52.713931       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:mongo,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/mongo,UID:847a31ea-3dbd-11e8-b0ef-022243d57c40,ResourceVersion:1578,Generation:0,CreationTimestamp:2018-04-11 19:21:21 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{name: mongo,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.18  0xc420379f50 ObjectReference{Kind:Pod,Namespace:default,Name:mongo-controller-bwjvp,UID:7f5f40c3-3dbd-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1576,FieldPath:,}}] [] [{ 27017 TCP}]}],}
I0414 14:32:52.713960       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-controller-leader,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/ingress-controller-leader,UID:5b848feb-3fe7-11e8-b0ef-022243d57c40,ResourceVersion:310415,Generation:0,CreationTimestamp:2018-04-14 13:25:54 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"qassooyf-firewall-64687f8869-chhnv","leaseDurationSeconds":30,"acquireTime":"2018-04-14T13:59:37Z","renewTime":"2018-04-14T14:02:08Z","leaderTransitions":2},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713979       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-controller-leader,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/ingress-controller-leader,UID:5b848feb-3fe7-11e8-b0ef-022243d57c40,ResourceVersion:310415,Generation:0,CreationTimestamp:2018-04-14 13:25:54 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"qassooyf-firewall-64687f8869-chhnv","leaseDurationSeconds":30,"acquireTime":"2018-04-14T13:59:37Z","renewTime":"2018-04-14T14:02:08Z","leaderTransitions":2},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.713996       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/kubernetes,UID:d768ce46-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:15,Generation:0,CreationTimestamp:2018-04-11 19:09:21 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{172.31.24.33  <nil> nil}] [] [{https 6443 TCP}]}],}
I0414 14:32:52.714017       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/kubernetes,UID:d768ce46-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:15,Generation:0,CreationTimestamp:2018-04-11 19:09:21 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{172.31.24.33  <nil> nil}] [] [{https 6443 TCP}]}],}
I0414 14:32:52.714038       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:aixiapbc-firewall,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/aixiapbc-firewall,UID:af3b3a45-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313486,Generation:0,CreationTimestamp:2018-04-14 14:32:39 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.714055       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:aixiapbc-firewall,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/aixiapbc-firewall,UID:af3b3a45-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313486,Generation:0,CreationTimestamp:2018-04-14 14:32:39 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.714071       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-dns,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-dns,UID:e0dfc494-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1195,Generation:0,CreationTimestamp:2018-04-11 19:09:37 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,k8s-app: kube-dns,kubernetes.io/cluster-service: true,kubernetes.io/name: KubeDNS,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.13  0xc420379f60 ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-dns-5c6c5b55b-x8vzq,UID:eb76e558-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1194,FieldPath:,}}] [] [{dns 53 UDP} {dns-tcp 53 TCP}]}],}
I0414 14:32:52.714104       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-dns,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-dns,UID:e0dfc494-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1195,Generation:0,CreationTimestamp:2018-04-11 19:09:37 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,k8s-app: kube-dns,kubernetes.io/cluster-service: true,kubernetes.io/name: KubeDNS,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.13  0xc420379f60 ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-dns-5c6c5b55b-x8vzq,UID:eb76e558-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1194,FieldPath:,}}] [] [{dns 53 UDP} {dns-tcp 53 TCP}]}],}
I0414 14:32:52.714140       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes-dashboard,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kubernetes-dashboard,UID:e16ae51e-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1160,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{cdk-addons: true,k8s-app: kubernetes-dashboard,kubernetes.io/cluster-service: true,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.15  0xc420379f70 ObjectReference{Kind:Pod,Namespace:kube-system,Name:kubernetes-dashboard-7d5dcdb6d9-627z5,UID:eb77ceac-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1158,FieldPath:,}}] [] [{ 8443 TCP}]}],}
I0414 14:32:52.714170       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes-dashboard,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kubernetes-dashboard,UID:e16ae51e-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1160,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{cdk-addons: true,k8s-app: kubernetes-dashboard,kubernetes.io/cluster-service: true,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.15  0xc420379f70 ObjectReference{Kind:Pod,Namespace:kube-system,Name:kubernetes-dashboard-7d5dcdb6d9-627z5,UID:eb77ceac-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1158,FieldPath:,}}] [] [{ 8443 TCP}]}],}
I0414 14:32:52.714196       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:monitoring-influxdb,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb,UID:e14519b5-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1162,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: InfluxDB,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.14  0xc420379f80 ObjectReference{Kind:Pod,Namespace:kube-system,Name:monitoring-influxdb-grafana-v4-6d67c7f4f-8trdp,UID:eba06c78-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1161,FieldPath:,}}] [] [{api 8086 TCP} {http 8083 TCP}]}],}
I0414 14:32:52.714227       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:monitoring-influxdb,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb,UID:e14519b5-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1162,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: InfluxDB,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.14  0xc420379f80 ObjectReference{Kind:Pod,Namespace:kube-system,Name:monitoring-influxdb-grafana-v4-6d67c7f4f-8trdp,UID:eba06c78-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1161,FieldPath:,}}] [] [{api 8086 TCP} {http 8083 TCP}]}],}
I0414 14:32:52.714256       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:aixiapbc,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/aixiapbc,UID:adf1c029-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313515,Generation:0,CreationTimestamp:2018-04-14 14:32:37 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.193  0xc420379f90 ObjectReference{Kind:Pod,Namespace:default,Name:aixiapbc-59494d49b4-2mgb2,UID:ad94f386-3ff0-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:313513,FieldPath:,}}] [] [{constellation 9001 TCP} {readfile 6382 TCP} {rpc 8545 TCP} {eth 23000 TCP}]}],}
I0414 14:32:52.714284       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:aixiapbc,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/aixiapbc,UID:adf1c029-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313515,Generation:0,CreationTimestamp:2018-04-14 14:32:37 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.193  0xc420379f90 ObjectReference{Kind:Pod,Namespace:default,Name:aixiapbc-59494d49b4-2mgb2,UID:ad94f386-3ff0-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:313513,FieldPath:,}}] [] [{constellation 9001 TCP} {readfile 6382 TCP} {rpc 8545 TCP} {eth 23000 TCP}]}],}
I0414 14:32:52.714312       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:default-http-backend,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/default-http-backend,UID:f9f49549-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1144,Generation:0,CreationTimestamp:2018-04-11 19:10:19 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{k8s-app: default-http-backend,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.12  0xc420379fa0 ObjectReference{Kind:Pod,Namespace:default,Name:default-http-backend-6wlqd,UID:eb74cec6-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1142,FieldPath:,}}] [] [{ 80 TCP}]}],}
I0414 14:32:52.714335       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:default-http-backend,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/default-http-backend,UID:f9f49549-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1144,Generation:0,CreationTimestamp:2018-04-11 19:10:19 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{k8s-app: default-http-backend,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.12  0xc420379fa0 ObjectReference{Kind:Pod,Namespace:default,Name:default-http-backend-6wlqd,UID:eb74cec6-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1142,FieldPath:,}}] [] [{ 80 TCP}]}],}
I0414 14:32:52.714360       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:metrics-server,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/metrics-server,UID:e1800878-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1178,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: Metrics-server,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.16  0xc420379fb0 ObjectReference{Kind:Pod,Namespace:kube-system,Name:metrics-server-v0.2.1-7486f5bd67-sfpnr,UID:eb8b650d-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1176,FieldPath:,}}] [] [{ 443 TCP}]}],}
I0414 14:32:52.714391       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:metrics-server,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/metrics-server,UID:e1800878-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1178,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: Metrics-server,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.16  0xc420379fb0 ObjectReference{Kind:Pod,Namespace:kube-system,Name:metrics-server-v0.2.1-7486f5bd67-sfpnr,UID:eb8b650d-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1176,FieldPath:,}}] [] [{ 443 TCP}]}],}
I0414 14:32:52.714422       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:monitoring-grafana,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/monitoring-grafana,UID:e124b512-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1164,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: Grafana,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.14  0xc420379fc0 ObjectReference{Kind:Pod,Namespace:kube-system,Name:monitoring-influxdb-grafana-v4-6d67c7f4f-8trdp,UID:eba06c78-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1161,FieldPath:,}}] [] [{ 3000 TCP}]}],}
I0414 14:32:52.714456       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:monitoring-grafana,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/monitoring-grafana,UID:e124b512-3dbb-11e8-bfe1-022243d57c40,ResourceVersion:1164,Generation:0,CreationTimestamp:2018-04-11 19:09:38 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,cdk-addons: true,kubernetes.io/cluster-service: true,kubernetes.io/name: Grafana,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.1.28.14  0xc420379fc0 ObjectReference{Kind:Pod,Namespace:kube-system,Name:monitoring-influxdb-grafana-v4-6d67c7f4f-8trdp,UID:eba06c78-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1161,FieldPath:,}}] [] [{ 3000 TCP}]}],}
I0414 14:32:52.714485       7 nginx.go:338] Event CREATE received - object &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-controller-leader-qsbmgxdv-firewall,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/ingress-controller-leader-qsbmgxdv-firewall,UID:76ecf5e8-3feb-11e8-b0ef-022243d57c40,ResourceVersion:309989,Generation:0,CreationTimestamp:2018-04-14 13:55:17 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"qsbmgxdv-firewall-5889bbb6b9-h4z7r","leaseDurationSeconds":30,"acquireTime":"2018-04-14T13:55:17Z","renewTime":"2018-04-14T13:58:25Z","leaderTransitions":1},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.714508       7 queue.go:70] queuing item &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-controller-leader-qsbmgxdv-firewall,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/ingress-controller-leader-qsbmgxdv-firewall,UID:76ecf5e8-3feb-11e8-b0ef-022243d57c40,ResourceVersion:309989,Generation:0,CreationTimestamp:2018-04-14 13:55:17 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"qsbmgxdv-firewall-5889bbb6b9-h4z7r","leaseDurationSeconds":30,"acquireTime":"2018-04-14T13:55:17Z","renewTime":"2018-04-14T13:58:25Z","leaderTransitions":1},},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0414 14:32:52.714526       7 nginx.go:338] Event CREATE received - object &Ingress{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-aixiapbc,GenerateName:,Namespace:default,SelfLink:/apis/extensions/v1beta1/namespaces/default/ingresses/ingress-aixiapbc,UID:aff33abb-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313490,Generation:1,CreationTimestamp:2018-04-14 14:32:41 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{ingress.kubernetes.io/ssl-redirect: false,kubernetes.io/ingress.class: aixiapbc-firewall,nginx.ingress.kubernetes.io/auth-realm: Authentication Required,nginx.ingress.kubernetes.io/auth-secret: basic-auth-aixiapbc,nginx.ingress.kubernetes.io/auth-type: basic,nginx.ingress.kubernetes.io/rewrite-target: /,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:IngressSpec{Backend:nil,TLS:[],Rules:[{ {HTTPIngressRuleValue{Paths:[{/jsonRPC {aixiapbc {0 8545 }}}],}}}],},Status:IngressStatus{LoadBalancer:k8s_io_api_core_v1.LoadBalancerStatus{Ingress:[],},},}
I0414 14:32:52.714572       7 queue.go:70] queuing item &Ingress{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-aixiapbc,GenerateName:,Namespace:default,SelfLink:/apis/extensions/v1beta1/namespaces/default/ingresses/ingress-aixiapbc,UID:aff33abb-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313490,Generation:1,CreationTimestamp:2018-04-14 14:32:41 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{ingress.kubernetes.io/ssl-redirect: false,kubernetes.io/ingress.class: aixiapbc-firewall,nginx.ingress.kubernetes.io/auth-realm: Authentication Required,nginx.ingress.kubernetes.io/auth-secret: basic-auth-aixiapbc,nginx.ingress.kubernetes.io/auth-type: basic,nginx.ingress.kubernetes.io/rewrite-target: /,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:IngressSpec{Backend:nil,TLS:[],Rules:[{ {HTTPIngressRuleValue{Paths:[{/jsonRPC {aixiapbc {0 8545 }}}],}}}],},Status:IngressStatus{LoadBalancer:k8s_io_api_core_v1.LoadBalancerStatus{Ingress:[],},},}
I0414 14:32:52.714610       7 queue.go:111] syncing
I0414 14:32:52.714649       7 controller.go:1113] getting endpoints for service default/default-http-backend and port &ServicePort{Name:,Protocol:TCP,Port:80,TargetPort:80,NodePort:0,}
I0414 14:32:52.714674       7 controller.go:1159] endpoints found: [{10.1.28.12 80 0 0 &ObjectReference{Kind:Pod,Namespace:default,Name:default-http-backend-6wlqd,UID:eb74cec6-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1142,FieldPath:,}}]
I0414 14:32:52.714694       7 controller.go:668] creating upstream default-aixiapbc-8545
I0414 14:32:52.714702       7 controller.go:768] obtaining port information for service default/aixiapbc
I0414 14:32:52.714709       7 controller.go:1113] getting endpoints for service default/aixiapbc and port &ServicePort{Name:rpc,Protocol:TCP,Port:8545,TargetPort:8545,NodePort:30328,}
I0414 14:32:52.714717       7 controller.go:1159] endpoints found: [{10.1.28.193 8545 0 0 &ObjectReference{Kind:Pod,Namespace:default,Name:aixiapbc-59494d49b4-2mgb2,UID:ad94f386-3ff0-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:313513,FieldPath:,}}]
I0414 14:32:52.714750       7 controller.go:404] secret  does not contain 'ca.crt', mutual authentication not enabled - ingress rule default/ingress-aixiapbc.
I0414 14:32:52.714757       7 controller.go:467] adding location /jsonRPC in ingress rule default/ingress-aixiapbc upstream default-aixiapbc-8545
I0414 14:32:52.714766       7 controller.go:216] obtaining information about stream services of type TCP located in configmap
I0414 14:32:52.714771       7 controller.go:216] obtaining information about stream services of type UDP located in configmap
I0414 14:32:52.714776       7 controller.go:183] backend reload required
I0414 14:32:52.714782       7 stat_collector.go:34] changing prometheus collector from  to default
I0414 14:32:52.714831       7 nginx.go:543] adjusting ServerNameHashBucketSize variable to 32
I0414 14:32:52.714837       7 nginx.go:555] number of worker processes: 4
I0414 14:32:52.714894       7 util.go:64] system fs.file-max=1638086
I0414 14:32:52.714900       7 nginx.go:560] maximum number of open file descriptors : 408497
I0414 14:32:52.714930       7 util.go:48] system net.core.somaxconn=128 (using system default)
I0414 14:32:52.716506       7 template.go:92] NGINX configuration: {"ProxySetHeaders":{},"AddHeaders":{},"MaxOpenFiles":408497,"BacklogSize":511,"Backends":[{"name":"default-aixiapbc-8545","service":{"metadata":{"name":"aixiapbc","namespace":"default","selfLink":"/api/v1/namespaces/default/services/aixiapbc","uid":"adf1183f-3ff0-11e8-b0ef-022243d57c40","resourceVersion":"313464","creationTimestamp":"2018-04-14T14:32:37Z"},"spec":{"ports":[{"name":"rpc","protocol":"TCP","port":8545,"targetPort":8545,"nodePort":30328},{"name":"constellation","protocol":"TCP","port":9001,"targetPort":9001,"nodePort":32529},{"name":"eth","protocol":"TCP","port":23000,"targetPort":23000,"nodePort":31915},{"name":"readfile","protocol":"TCP","port":6382,"targetPort":6382,"nodePort":30217}],"selector":{"app":"quorum-node-aixiapbc"},"clusterIP":"10.152.183.197","type":"NodePort","sessionAffinity":"None","externalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}},"port":8545,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.1.28.193","port":"8545","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"default","name":"aixiapbc-59494d49b4-2mgb2","uid":"ad94f386-3ff0-11e8-b0ef-022243d57c40","resourceVersion":"313513"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}},{"name":"upstream-default-backend","service":{"metadata":{"name":"default-http-backend","namespace":"default","selfLink":"/api/v1/namespaces/default/services/default-http-backend","uid":"f9f38b17-3dbb-11e8-bfe1-022243d57c40","resourceVersion":"255","creationTimestamp":"2018-04-11T19:10:19Z","labels":{"k8s-app":"default-http-backend"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"default-http-backend\"},\"name\":\"default-http-backend\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":80,\"protocol\":\"TCP\",\"targetPort\":80}],\"selector\":{\"app\":\"default-http-backend\"}}}\n"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"default-http-backend"},"clusterIP":"10.152.183.135","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.1.28.12","port":"80","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"default","name":"default-http-backend-6wlqd","uid":"eb74cec6-3dbc-11e8-b0ef-022243d57c40","resourceVersion":"1142"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}}],"PassthroughBackends":null,"Servers":[{"hostname":"_","sslPassthrough":false,"sslCertificate":"/ingress-controller/ssl/default-fake-certificate.pem","sslFullChainCertificate":"","sslExpireTime":"0001-01-01T00:00:00Z","sslPemChecksum":"b5fd8058397b8cadcfa0f5ee07709033df8230d9","locations":[{"path":"/jsonRPC","isDefBackend":false,"ingress":{"metadata":{"name":"ingress-aixiapbc","namespace":"default","selfLink":"/apis/extensions/v1beta1/namespaces/default/ingresses/ingress-aixiapbc","uid":"aff33abb-3ff0-11e8-b0ef-022243d57c40","resourceVersion":"313490","generation":1,"creationTimestamp":"2018-04-14T14:32:41Z","annotations":{"ingress.kubernetes.io/ssl-redirect":"false","kubernetes.io/ingress.class":"aixiapbc-firewall","nginx.ingress.kubernetes.io/auth-realm":"Authentication Required","nginx.ingress.kubernetes.io/auth-secret":"basic-auth-aixiapbc","nginx.ingress.kubernetes.io/auth-type":"basic","nginx.ingress.kubernetes.io/rewrite-target":"/"}},"spec":{"rules":[{"http":{"paths":[{"path":"/jsonRPC","backend":{"serviceName":"aixiapbc","servicePort":8545}}]}}]},"status":{"loadBalancer":{}}},"backend":"default-aixiapbc-8545","service":{"metadata":{"name":"aixiapbc","namespace":"default","selfLink":"/api/v1/namespaces/default/services/aixiapbc","uid":"adf1183f-3ff0-11e8-b0ef-022243d57c40","resourceVersion":"313464","creationTimestamp":"2018-04-14T14:32:37Z"},"spec":{"ports":[{"name":"rpc","protocol":"TCP","port":8545,"targetPort":8545,"nodePort":30328},{"name":"constellation","protocol":"TCP","port":9001,"targetPort":9001,"nodePort":32529},{"name":"eth","protocol":"TCP","port":23000,"targetPort":23000,"nodePort":31915},{"name":"readfile","protocol":"TCP","port":6382,"targetPort":6382,"nodePort":30217}],"selector":{"app":"quorum-node-aixiapbc"},"clusterIP":"10.152.183.197","type":"NodePort","sessionAffinity":"None","externalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}},"port":8545,"upstream-vhost":"","basicDigestAuth":{"type":"basic","realm":"Authentication Required","file":"/etc/ingress-controller/auth/default-ingress-aixiapbc.passwd","secured":true,"fileSha":"e9b8ff9df4601f7da0a60f077bfbf367397d2e38","secret":"default/basic-auth-aixiapbc"},"corsConfig":{"corsEnabled":false,"corsAllowOrigin":"*","corsAllowMethods":"GET, PUT, POST, DELETE, PATCH, OPTIONS","corsAllowHeaders":"DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization","corsAllowCredentials":true,"corsMaxAge":1728000},"externalAuth":{"url":"","host":"","signinUrl":"","method":"","requestRedirect":""},"rateLimit":{"connections":{"name":"","limit":0,"burst":0,"sharedSize":0},"rps":{"name":"","limit":0,"burst":0,"sharedSize":0},"rpm":{"name":"","limit":0,"burst":0,"sharedSize":0},"limit-rate":0,"limit-rate-after":0,"name":"","id":"","whitelist":null},"redirect":{"url":"","code":0,"fromToWWW":false},"rewrite":{"target":"/","addBaseUrl":false,"baseUrlScheme":"","sslRedirect":true,"forceSSLRedirect":false,"appRoot":""},"whitelist":{},"proxy":{"bodySize":"1m","connectTimeout":5,"sendTimeout":60,"readTimeout":60,"bufferSize":"4k","cookieDomain":"off","cookiePath":"off","nextUpstream":"error timeout invalid_header http_502 http_503 http_504","proxyRedirectFrom":"off","proxyRedirectTo":"","requestBuffering":"on","proxyBuffering":"off"},"usePortInRedirects":false,"configurationSnippet":"","connection":{"header":"","enabled":false},"logs":{"accessLog":true}},{"path":"/","isDefBackend":true,"ingress":null,"backend":"upstream-default-backend","service":{"metadata":{"name":"default-http-backend","namespace":"default","selfLink":"/api/v1/namespaces/default/services/default-http-backend","uid":"f9f38b17-3dbb-11e8-bfe1-022243d57c40","resourceVersion":"255","creationTimestamp":"2018-04-11T19:10:19Z","labels":{"k8s-app":"default-http-backend"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"default-http-backend\"},\"name\":\"default-http-backend\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":80,\"protocol\":\"TCP\",\"targetPort\":80}],\"selector\":{\"app\":\"default-http-backend\"}}}\n"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"default-http-backend"},"clusterIP":"10.152.183.135","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"upstream-vhost":"","basicDigestAuth":{"type":"","realm":"","file":"","secured":false,"fileSha":"","secret":""},"corsConfig":{"corsEnabled":false,"corsAllowOrigin":"","corsAllowMethods":"","corsAllowHeaders":"","corsAllowCredentials":false,"corsMaxAge":0},"externalAuth":{"url":"","host":"","signinUrl":"","method":"","requestRedirect":""},"rateLimit":{"connections":{"name":"","limit":0,"burst":0,"sharedSize":0},"rps":{"name":"","limit":0,"burst":0,"sharedSize":0},"rpm":{"name":"","limit":0,"burst":0,"sharedSize":0},"limit-rate":0,"limit-rate-after":0,"name":"","id":"","whitelist":null},"redirect":{"url":"","code":0,"fromToWWW":false},"rewrite":{"target":"","addBaseUrl":false,"baseUrlScheme":"","sslRedirect":false,"forceSSLRedirect":false,"appRoot":""},"whitelist":{},"proxy":{"bodySize":"1m","connectTimeout":5,"sendTimeout":60,"readTimeout":60,"bufferSize":"4k","cookieDomain":"off","cookiePath":"off","nextUpstream":"error timeout invalid_header http_502 http_503 http_504","proxyRedirectFrom":"off","proxyRedirectTo":"","requestBuffering":"on","proxyBuffering":"off"},"usePortInRedirects":false,"configurationSnippet":"","connection":{"header":"","enabled":false},"logs":{"accessLog":false}}],"certificateAuth":{"secret":"","caFilename":"","pemSha":"","verify_client":"","validationDepth":0,"errorPage":"","passCertToUpstream":false,"AuthTLSError":""},"serverSnippet":""}],"TCPBackends":[],"UDPBackends":[],"HealthzURI":"/healthz","CustomErrors":false,"Cfg":{"app-root":"","custom-http-errors":[],"proxy-body-size":"1m","proxy-connect-timeout":5,"proxy-read-timeout":60,"proxy-send-timeout":60,"proxy-buffer-size":"4k","proxy-cookie-path":"off","proxy-cookie-domain":"off","proxy-next-upstream":"error timeout invalid_header http_502 http_503 http_504","proxy-redirect-from":"off","proxy-redirect-to":"","proxy-request-buffering":"on","Resolver":["10.152.183.156"],"skip-access-log-urls":[],"ssl-redirect":true,"force-ssl-redirect":false,"use-port-in-redirects":false,"upstream-max-fails":0,"upstream-fail-timeout":0,"upstream-hash-by":"","whitelist-source-range":[],"limit-rate":0,"limit-rate-after":0,"proxy-buffering":"off","allow-backend-server-header":false,"access-log-path":"/var/log/nginx/access.log","error-log-path":"/var/log/nginx/error.log","enable-dynamic-tls-records":true,"enable-modsecurity":false,"enable-owasp-modsecurity-crs":false,"client-header-buffer-size":"1k","client-header-timeout":60,"client-body-buffer-size":"8k","client-body-timeout":60,"disable-ipv6-dns":false,"enable-underscores-in-headers":false,"ignore-invalid-headers":true,"vts-status-zone-size":"10m","vts-default-filter-key":"$geoip_country_code country::*","retry-non-idempotent":false,"error-log-level":"notice","http2-max-field-size":"4k","http2-max-header-size":"16k","hsts":true,"hsts-include-subdomains":true,"hsts-max-age":"15724800","keep-alive":75,"keep-alive-requests":100,"large-client-header-buffers":"4 8k","log-format-upstream":"%v - [$the_real_ip] - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status","log-format-stream":"[$time_local] $protocol $status $bytes_sent $bytes_received $session_time","max-worker-connections":16384,"map-hash-bucket-size":64,"proxy-real-ip-cidr":["0.0.0.0/0"],"server-name-hash-max-size":1024,"server-name-hash-bucket-size":32,"proxy-headers-hash-max-size":512,"proxy-headers-hash-bucket-size":64,"server-tokens":true,"ssl-ciphers":"ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256","ssl-ecdh-curve":"auto","ssl-protocols":"TLSv1.2","ssl-session-cache":true,"ssl-session-cache-size":"10m","ssl-session-tickets":true,"ssl-session-timeout":"10m","ssl-buffer-size":"4k","use-gzip":true,"use-geoip":true,"brotli-level":4,"brotli-types":"application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component","use-http2":true,"gzip-types":"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component","worker-processes":"4","worker-shutdown-timeout":"10s","load-balance":"least_conn","variables-hash-bucket-size":128,"variables-hash-max-size":2048,"upstream-keepalive-connections":32,"limit-conn-zone-variable":"$binary_remote_addr","proxy-stream-timeout":"600s","proxy-stream-responses":1,"forwarded-for-header":"X-Forwarded-For","enable-opentracing":false,"zipkin-collector-host":"","zipkin-collector-port":9411,"zipkin-service-name":"nginx","jaeger-collector-host":"","jaeger-collector-port":6831,"jaeger-service-name":"nginx","jaeger-sampler-type":"const","jaeger-sampler-param":"1","http-snippet":"","server-snippet":"","location-snippet":"","http-redirect-code":308,"reuse-port":false,"hide-headers":null,"limit-req-status-code":503,"enable-syslog":false,"syslog-host":"","syslog-port":514,"no-tls-redirect-locations":"/.well-known/acme-challenge"},"IsIPV6Enabled":true,"IsSSLPassthroughEnabled":false,"RedirectServers":{},"ListenPorts":{"HTTP":80,"HTTPS":443,"Status":18080,"Health":10254,"Default":8181,"SSLProxy":442},"PublishService":null,"DynamicConfigurationEnabled":false}
I0414 14:32:52.720668       7 leaderelection.go:184] successfully acquired lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:32:52.720708       7 status.go:196] new leader elected: aixiapbc-firewall-866774d6bc-jp7hj
I0414 14:32:52.720722       7 status.go:184] I am the new status update leader
I0414 14:32:52.750451       7 nginx.go:658] NGINX configuration diff
I0414 14:32:52.750510       7 nginx.go:659] --- /etc/nginx/nginx.conf	2018-03-19 12:21:40.000000000 +0000

I0414 14:32:52.773461       7 controller.go:192] ingress backend successfully reloaded...
I0414 14:32:52.773519       7 queue.go:105] skipping kube-system/heapster sync (1523716372714606837 > 1523716372713762794)
I0414 14:32:52.773548       7 queue.go:105] skipping kube-system/kube-controller-manager sync (1523716372714606837 > 1523716372713836393)
I0414 14:32:52.773569       7 queue.go:105] skipping kube-system/kube-scheduler sync (1523716372714606837 > 1523716372713881780)
I0414 14:32:52.773587       7 queue.go:105] skipping default/mongo sync (1523716372714606837 > 1523716372713929051)
I0414 14:32:52.773600       7 queue.go:105] skipping default/ingress-controller-leader sync (1523716372714606837 > 1523716372713977378)
I0414 14:32:52.773617       7 queue.go:105] skipping default/kubernetes sync (1523716372714606837 > 1523716372714015788)
I0414 14:32:52.773627       7 queue.go:105] skipping default/aixiapbc-firewall sync (1523716372714606837 > 1523716372714053834)
I0414 14:32:52.773632       7 queue.go:105] skipping kube-system/kube-dns sync (1523716372714606837 > 1523716372714102345)
I0414 14:32:52.773638       7 queue.go:105] skipping kube-system/kubernetes-dashboard sync (1523716372714606837 > 1523716372714168182)
I0414 14:32:52.773656       7 queue.go:105] skipping kube-system/monitoring-influxdb sync (1523716372714606837 > 1523716372714225040)
I0414 14:32:52.773666       7 queue.go:105] skipping default/aixiapbc sync (1523716372714606837 > 1523716372714282178)
I0414 14:32:52.773671       7 queue.go:105] skipping default/default-http-backend sync (1523716372714606837 > 1523716372714333172)
I0414 14:32:52.773689       7 queue.go:105] skipping kube-system/metrics-server sync (1523716372714606837 > 1523716372714389126)
I0414 14:32:52.773699       7 queue.go:105] skipping kube-system/monitoring-grafana sync (1523716372714606837 > 1523716372714453757)
I0414 14:32:52.773704       7 queue.go:105] skipping default/ingress-controller-leader-qsbmgxdv-firewall sync (1523716372714606837 > 1523716372714506580)
I0414 14:32:52.773723       7 queue.go:105] skipping default/ingress-aixiapbc sync (1523716372714606837 > 1523716372714570365)
I0414 14:33:00.228263       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:07.738663       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:15.245660       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:22.764948       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:30.272418       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:37.782668       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:45.288999       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:33:52.720901       7 queue.go:70] queuing item sync status
I0414 14:33:52.720945       7 queue.go:111] syncing sync status
I0414 14:33:52.733481       7 status.go:356] updating Ingress default/ingress-aixiapbc status to [{ }]
I0414 14:33:52.738974       7 store.go:465] updating annotations information for ingres default/ingress-aixiapbc
I0414 14:33:52.739051       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-aixiapbc", UID:"aff33abb-3ff0-11e8-b0ef-022243d57c40", APIVersion:"extensions", ResourceVersion:"313629", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/ingress-aixiapbc
I0414 14:33:52.739229       7 main.go:109] No default affinity was found for Ingress ingress-aixiapbc
I0414 14:33:52.739369       7 nginx.go:338] Event UPDATE received - object &Ingress{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-aixiapbc,GenerateName:,Namespace:default,SelfLink:/apis/extensions/v1beta1/namespaces/default/ingresses/ingress-aixiapbc,UID:aff33abb-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313629,Generation:1,CreationTimestamp:2018-04-14 14:32:41 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{ingress.kubernetes.io/ssl-redirect: false,kubernetes.io/ingress.class: aixiapbc-firewall,nginx.ingress.kubernetes.io/auth-realm: Authentication Required,nginx.ingress.kubernetes.io/auth-secret: basic-auth-aixiapbc,nginx.ingress.kubernetes.io/auth-type: basic,nginx.ingress.kubernetes.io/rewrite-target: /,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:IngressSpec{Backend:nil,TLS:[],Rules:[{ {HTTPIngressRuleValue{Paths:[{/jsonRPC {aixiapbc {0 8545 }}}],}}}],},Status:IngressStatus{LoadBalancer:k8s_io_api_core_v1.LoadBalancerStatus{Ingress:[{ }],},},}
I0414 14:33:52.739461       7 queue.go:70] queuing item &Ingress{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ingress-aixiapbc,GenerateName:,Namespace:default,SelfLink:/apis/extensions/v1beta1/namespaces/default/ingresses/ingress-aixiapbc,UID:aff33abb-3ff0-11e8-b0ef-022243d57c40,ResourceVersion:313629,Generation:1,CreationTimestamp:2018-04-14 14:32:41 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{ingress.kubernetes.io/ssl-redirect: false,kubernetes.io/ingress.class: aixiapbc-firewall,nginx.ingress.kubernetes.io/auth-realm: Authentication Required,nginx.ingress.kubernetes.io/auth-secret: basic-auth-aixiapbc,nginx.ingress.kubernetes.io/auth-type: basic,nginx.ingress.kubernetes.io/rewrite-target: /,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:IngressSpec{Backend:nil,TLS:[],Rules:[{ {HTTPIngressRuleValue{Paths:[{/jsonRPC {aixiapbc {0 8545 }}}],}}}],},Status:IngressStatus{LoadBalancer:k8s_io_api_core_v1.LoadBalancerStatus{Ingress:[{ }],},},}
I0414 14:33:52.739520       7 queue.go:111] syncing default/ingress-aixiapbc
I0414 14:33:52.739546       7 controller.go:1113] getting endpoints for service default/default-http-backend and port &ServicePort{Name:,Protocol:TCP,Port:80,TargetPort:80,NodePort:0,}
I0414 14:33:52.739563       7 controller.go:1159] endpoints found: [{10.1.28.12 80 0 0 &ObjectReference{Kind:Pod,Namespace:default,Name:default-http-backend-6wlqd,UID:eb74cec6-3dbc-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:1142,FieldPath:,}}]
I0414 14:33:52.739584       7 controller.go:668] creating upstream default-aixiapbc-8545
I0414 14:33:52.739592       7 controller.go:768] obtaining port information for service default/aixiapbc
I0414 14:33:52.739604       7 controller.go:1113] getting endpoints for service default/aixiapbc and port &ServicePort{Name:rpc,Protocol:TCP,Port:8545,TargetPort:8545,NodePort:30328,}
I0414 14:33:52.739612       7 controller.go:1159] endpoints found: [{10.1.28.193 8545 0 0 &ObjectReference{Kind:Pod,Namespace:default,Name:aixiapbc-59494d49b4-2mgb2,UID:ad94f386-3ff0-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:313513,FieldPath:,}}]
I0414 14:33:52.739661       7 controller.go:404] secret  does not contain 'ca.crt', mutual authentication not enabled - ingress rule default/ingress-aixiapbc.
I0414 14:33:52.739673       7 controller.go:467] adding location /jsonRPC in ingress rule default/ingress-aixiapbc upstream default-aixiapbc-8545
I0414 14:33:52.739682       7 controller.go:216] obtaining information about stream services of type TCP located in configmap
I0414 14:33:52.739686       7 controller.go:216] obtaining information about stream services of type UDP located in configmap
I0414 14:33:52.739701       7 controller.go:170] skipping backend reload (no changes detected)
I0414 14:33:52.796414       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:00.303136       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:07.817270       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:15.327970       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:22.838043       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:30.345579       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:37.857127       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall
I0414 14:34:45.364854       7 leaderelection.go:199] successfully renewed lease default/ingress-controller-leader-aixiapbc-firewall

I am making request to /jsonRPC and it's giving 502. I don't see anything wrong in the logs.
@antoineco
Copy link
Contributor

It seems like the backend has 1 endpoint and the location is added properly, so the ingress class matches:

] obtaining port information for service default/aixiapbc
] getting endpoints for service default/aixiapbc and port &ServicePort{Name:rpc,Protocol:TCP,Port:8545,TargetPort:8545,NodePort:30328,}
] endpoints found: [{10.1.28.193 8545 0 0 &ObjectReference{Kind:Pod,Namespace:default,Name:aixiapbc-59494d49b4-2mgb2,UID:ad94f386-3ff0-11e8-b0ef-022243d57c40,APIVersion:,ResourceVersion:313513,FieldPath:,}}]
...
] adding location /jsonRPC in ingress rule default/ingress-aixiapbc upstream default-aixiapbc-8545

Could you attach the generated nginx config? (/etc/nginx/nginx.conf)

@web3rover
Copy link
Author

web3rover commented Apr 19, 2018 •
edited
Loading

@antoineco here is the nginx config

daemon off;

worker_processes 4;
pid /run/nginx.pid;

worker_rlimit_nofile 261120;
worker_shutdown_timeout 10s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;

    real_ip_recursive   on;
    set_real_ip_from    0.0.0.0/0;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    sendfile            on;

    aio                 threads;
    aio_write           on;

    tcp_nopush          on;
    tcp_nodelay         on;

    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout  75s;
    keepalive_requests 100;

    client_header_buffer_size       1k;
    client_header_timeout           60s;
    large_client_header_buffers     4 8k;
    client_body_buffer_size         8k;
    client_body_timeout             60s;

    http2_max_field_size            4k;
    http2_max_header_size           16k;

    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   32;
    map_hash_bucket_size            64;

    proxy_headers_hash_max_size     512;
    proxy_headers_hash_bucket_size  64;

    variables_hash_bucket_size      64;
    variables_hash_max_size         2048;

    underscores_in_headers          off;
    ignore_invalid_headers          on;
    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    # Custom headers for response

    server_tokens on;

    # disable warnings
    uninitialized_variable_warn off;

    # Additional available variables:
    # $namespace
    # $ingress_name
    # $service_name
    log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver 172.31.0.2 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    map $http_x_forwarded_for $the_real_ip {
        default          $remote_addr;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
        default           $http_x_forwarded_port;
        ''                $server_port;
    }

    map $http_x_forwarded_host $best_http_host {
        default          $http_x_forwarded_host;
        ''               $this_host;
    }

    map $pass_server_port $pass_port {
        443              443;
        default          $pass_server_port;
    }

    # Obtain best http host
    map $http_host $this_host {
        default          $http_host;
        ''               $host;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    ssl_ecdh_curve auto;

    proxy_ssl_session_reuse on;

    upstream upstream-default-backend {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;

        keepalive 32;

        server 10.1.28.12:80 max_fails=0 fail_timeout=0;
    }

    server {
        server_name _;
        listen 80 default_server reuseport backlog=511;
        listen [::]:80 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        listen 443  default_server reuseport backlog=511 ssl http2;
        listen [::]:443  default_server reuseport backlog=511 ssl http2;
        # PEM sha: 18dc9494af264e0f1eb52361d3a16c2ac5a033da
        ssl_certificate                         /ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/default-fake-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

        location /ujkltfsx {

            set $proxy_upstream_name "upstream-default-backend";

            set $namespace      "default";
            set $ingress_name   "ingress-ujkltfsx";
            set $service_name   "ujkltfsx";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://upstream-default-backend;
        }
        location / {

            set $proxy_upstream_name "upstream-default-backend";

            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://upstream-default-backend;
        }

        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }
    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/ingress/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
        listen 18080 default_server reuseport backlog=511;
        listen [::]:18080 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        location /healthz {
            access_log off;
            return 200;
        }

        location /nginx_status {
            set $proxy_upstream_name "internal";

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass          http://upstream-default-backend;
        }
    }
}

stream {
    log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # TCP services

    # UDP services
}

@antoineco
Copy link
Contributor

Thanks @Narayanprusty, looking into this.

@ralphrodkey
Copy link

I'm seeing the same issue with the latest stable chart (nginx-ingress-0.18.0). When I specify something like --set controller.ingressClass=nginx-internal, I get output from the ingress controller like this:

ignoring add for ingress XXXXXXX based on annotation kubernetes.io/ingress.class with value

When I remove the custom ingressClass, everything works as expected.

@antoineco
Copy link
Contributor

antoineco commented May 3, 2018 •
edited
Loading

@ralphrodkey that's a different issue, you created your Ingress object without setting the corresponding ingress class annotation.

ignoring add for ingress XXXXXXX based on annotation kubernetes.io/ingress.class with value <-- (notice the empty value)

The class set in the controller and in the Ingress objects must match. In your case: kubernetes.io/ingress.class: nginx-internal.

@antoineco
Copy link
Contributor

@Narayanprusty I tried to reproduce this without success. I think either your backend was not ready and NGINX returned a 502, or the backend itself returned that error. Besides the nginx.conf you posted does not correspond to the error you mentioned; the example uses the default backend, while the logs reference a service called "aixiapbc".

I started a controller with --ingress-class=foo and created the following Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-issue2349
  annotations:
    kubernetes.io/ingress.class: foo
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: echoheaders
          servicePort: 80
        path: /jsonRPC

Works like a charm:

$ curl -kL http://localhost/jsonRPC

Hostname: echoheaders-5d8965b6cc-lj5z5
(...)
$ curl -kL http://localhost/ 
      
default backend - 404

I suggest you give it another try on the latest ingress-nginx version and troubleshoot the upstream corresponding to your service.

I believe this can be closed, ping @aledbf.

@aledbf
Copy link
Member

aledbf commented May 3, 2018

Agree with @antoineco. Closing.

@Narayanprusty when open an issue please send the stored yaml object and not a template, that way is a lot easier for us try to replicate the setup.

@aledbf aledbf closed this as completed May 3, 2018
@ralphrodkey
Copy link

@antoineco Sorry, I can see that I didn't mention that I was updating the annotation on my deployment to match what the ingress controller was expecting.

This fails:

  • Install nginx-ingress-0.18.0 with -set controller.ingressClass=nginx-internal
  • Create a deployment with kubernetes.io/ingress.class: nginx-internal
  • The ingress controller logs ignoring add for ingress XXXXXXX based on annotation kubernetes.io/ingress.class with value

This works:

  • Install nginx-ingress-0.18.0
  • Create a deployment with kubernetes.io/ingress.class: nginx
  • The ingress controller registers the pod as expected

@antoineco
Copy link
Contributor

@ralphrodkey the annotation must be set on the Ingress objects, not Deployments (see my example above).
with value <empty string> means the annotation is not set properly.

@ralphrodkey
Copy link

Well, there's nothing like an embarrassingly public mistake to keep you humble. Thanks, @antoineco. It works exactly as expected when you do it the right way.

@antoineco
Copy link
Contributor

Glad it helped! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@antoineco antoineco

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aledbf @ralphrodkey @antoineco @web3rover