Skip to content

China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument #3860

New issue
New issue
Closed
Closed
China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument#3860
Labels
area/networkingnetworking issuescause/firewall-or-proxyWhen firewalls or proxies seem to be interferingkind/bugCategorizes issue or PR as related to a bug.priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.
@hect1c

Description

@hect1c
hect1c
opened on Mar 12, 2019 · edited by tstromberg

Hey,

I'm trying to do a quick-start local setup on my MacOS Machine:

Run the following command

minikube start --vm-driver=hyperkit -v=9

And I get the following output / debug

`minikube v0.35.0 on darwin (amd64)
🔥 Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
...
...
About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %s "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
" | sudo tee /etc/sysconfig/crio.minikube
SSH cmd err, output: :
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '

...
📶 "minikube" IP address is 192.168.64.6
🐳 Configuring Docker as the container runtime ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:62493
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
✨ Preparing Kubernetes environment ...
(minikube) Calling .GetURL
🚜 Pulling images required by Kubernetes v1.13.4 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.4": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.4 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.4
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new`

Again my OS is MacOS High Sierra Version 10.13.6

I am able to pull the images using docker pull so don't think it's a network issue, however it seems unable to download or access the kubeadm, which I think is something I may need to install separately but having found any guides or documentation on this. I may be missing something but greatly appreciate your assistance on this ahead of time.

Activity

huguesalary

huguesalary commented on Mar 12, 2019

@huguesalary
huguesalary
on Mar 12, 2019
Contributor

Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443

This is quite a weird IP. What does nslookup k8s.gcr.io return? It seems to me you have an internal DNS issue.

hect1c

hect1c commented on Mar 13, 2019

@hect1c
hect1c
on Mar 13, 2019 · edited by hect1c
Author

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

huguesalary

huguesalary commented on Mar 13, 2019

@huguesalary
huguesalary
on Mar 13, 2019
Contributor

That’s what I thought, the issue is with the DNS returning an invalid IP. I’d have thought that through a VPN you’d get the correct ip.

Not sure what your options are since you are in China. For me the k8s.gcr.io resolves to 74.125.20.82.

tstromberg
added
cause/firewall-or-proxyWhen firewalls or proxies seem to be interfering
on Mar 13, 2019
tstromberg
changed the title [-]minikube start --vm-driver=hyperkit fails on pulling kube images[/-] [+]China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument[/+] on Mar 13, 2019
tstromberg
added
area/networkingnetworking issues
on Mar 13, 2019
tstromberg

tstromberg commented on Mar 13, 2019

@tstromberg
tstromberg
on Mar 13, 2019
Contributor

Your DNS is responding with an address that apparently is not available. You can try this to possibly work-around the issue by fetching the images outside of the VM using:

minikube start --cache-images

Or by passing information to Docker so that it can use an HTTP proxy: https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

Let me know what you find out! Once #3714 is merged, you'll be able to specify --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

tstromberg
added
kind/bugCategorizes issue or PR as related to a bug.
needs-solution-messageIssues where where offering a solution for an error would be helpful
priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.
on Mar 13, 2019
daniel-farina

daniel-farina commented on Mar 17, 2019

@daniel-farina
daniel-farina
on Mar 17, 2019

Deleting the local config fixed this issue for me:

rm -rf ~/.minikube

I run it again and all errors were gone.

kingonion

kingonion commented on Apr 2, 2019

@kingonion
kingonion
on Apr 2, 2019 · edited by kingonion

In china, because of the GFW, you can't pull images from k8s.gcr.io without a VPN. But you can pull the needed images from registry.cn-hangzhou.aliyuncs.com/google_containers and tag them. If the needed images exist on the local system, the starting process will go on even with pulling error. I wrote a shell script that can help you install minikube on Linux OS easily. And you can also make a offline installation package by using the script. The project home is https://github.com/Kingonion/minikube-on-linux

gbraad

gbraad commented on Apr 2, 2019

@gbraad
gbraad
on Apr 2, 2019 via email
Contributor
maxnilz

maxnilz commented on Apr 17, 2019

@maxnilz
maxnilz
on Apr 17, 2019 · edited by maxnilz

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

Hi, I have the same problem with you, after some digging, I found the docker pull request will first go to docker daemon via REST endpoint /images/create, and the docker daemon will take over the pulling work. In order to make docker daemon to use the proxy, we need to set http_proxy & https_proxy to the place where dockerd running, i.e, root user.

Here is my scenario:
OS: I'm using ubuntu18.04 desktop
Where the docker daemon is running: the docker daemon run in root user
What supervisor host the docker daemon: systemd
Do I set the https_proxy & http_proxy: yes, I set them for the normal user, but didn't set them for root user.

How I solve this:
Because I just want to set http_proxy & https_proxy for docker daemon(dockerd) only for root, I set the proxy info into systemd service unit, like this(for my computer, the config file is located here: /lib/systemd/system/docker.service:

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
Environment="http_proxy=http://127.0.0.1:7777"
Environment="https_proxy=http://127.0.0.1:7777"
ExecStart=/usr/bin/dockerd -D -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Hope this will help somebody who got the same error like me.

singleye

singleye commented on Apr 24, 2019

@singleye
singleye
on Apr 24, 2019 · edited by singleye

I had the same issue on 'MacOS 10.14.4 Majave'

Pull and tag images doesn't work for me

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

$ docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                                                         v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
k8s.gcr.io/kube-controller-manager                                            v1.14.0             b95b1efa0436        4 weeks ago         158MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.14.0             b95b1efa0436        4 weeks ago         158MB
k8s.gcr.io/kube-scheduler                                                     v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.14.0             ecf910f40d6e        4 weeks ago         210MB
k8s.gcr.io/kube-apiserver                                                     v1.14.0             ecf910f40d6e        4 weeks ago         210MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.3.1               eb516548c180        3 months ago        40.3MB
k8s.gcr.io/coredns                                                            1.3.1               eb516548c180        3 months ago        40.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/etcd                                                               3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/pause                                                              3.1                 da86e6ba6ca1        16 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        16 months ago       742kB

$ minikube start
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...r     y    d         r          r   s           7                   1                   4 m      a          2
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to compe_te ...                               3                   d                   1  m      a         7
✨  Preparing Kubernetes environment ...                                      9                   1                   1  m      a         2
🚜  Pulling images required by Kubernetes v1.14.0 ...
❌  Unable to pull images, which may be OK: runnigz cmd: sudo kubeadm cgnfig images pull --config /var/lib/kubeadm.yaml: command faierdv sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting heade
rs)
, error: exit status 1
: Process exited with status 1
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd scheduler controller dns           v                   5                   4 w     a           8
📯  Updating kube-proxy configuration ...
🤔  Verifying component health ......
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

But specify '--image-repository' works for me

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...          "                        6 w     a           E      (     6 w     a                                         f
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to complete ...                  6 w     a           E      (   6 w     a                                           v
✨  Preparing Kubernetes 4nvironment ..."                        6 w     a           E      (   6 w     a                                           h
🚜  Pulling images required by Kubernetes v1.14.0 ...
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd skhrduler controller dnsw     a           E      (   6 w     a                                           g
📯  Updating kube-proxy configuration ...
🤔  Verifying component health .....
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

BTW, I'm using minikube v1.0.0

$ minikube version
minikube version: v1.0.0

6 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/networkingnetworking issuescause/firewall-or-proxyWhen firewalls or proxies seem to be interferingkind/bugCategorizes issue or PR as related to a bug.priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @gbraad@tstromberg@huguesalary@hantsy@singleye

        Issue actions
          China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument · Issue #3860 · kubernetes/minikube