Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument #3860

Closed
hect1c opened this issue Mar 12, 2019 · 13 comments
Closed

China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument #3860

hect1c opened this issue Mar 12, 2019 · 13 comments
Labels
area/networking networking issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.

Comments

@hect1c
Copy link

hect1c commented Mar 12, 2019 •
edited by tstromberg

Hey,

I'm trying to do a quick-start local setup on my MacOS Machine:

Run the following command

minikube start --vm-driver=hyperkit -v=9

And I get the following output / debug

`minikube v0.35.0 on darwin (amd64)
🔥 Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
...
...
About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %s "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
" | sudo tee /etc/sysconfig/crio.minikube
SSH cmd err, output: :
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '

...
📶 "minikube" IP address is 192.168.64.6
🐳 Configuring Docker as the container runtime ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:62493
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
✨ Preparing Kubernetes environment ...
(minikube) Calling .GetURL
🚜 Pulling images required by Kubernetes v1.13.4 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.4": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.4 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.4
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new`

Again my OS is MacOS High Sierra Version 10.13.6

I am able to pull the images using docker pull so don't think it's a network issue, however it seems unable to download or access the kubeadm, which I think is something I may need to install separately but having found any guides or documentation on this. I may be missing something but greatly appreciate your assistance on this ahead of time.
@huguesalary
Copy link
Contributor

Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443

This is quite a weird IP. What does nslookup k8s.gcr.io return? It seems to me you have an internal DNS issue.

@hect1c
Copy link
Author

hect1c commented Mar 13, 2019 •
edited

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

@huguesalary
Copy link
Contributor

That’s what I thought, the issue is with the DNS returning an invalid IP. I’d have thought that through a VPN you’d get the correct ip.

Not sure what your options are since you are in China. For me the k8s.gcr.io resolves to 74.125.20.82.

@tstromberg tstromberg added the cause/firewall-or-proxy When firewalls or proxies seem to be interfering label Mar 13, 2019
@tstromberg tstromberg changed the title minikube start --vm-driver=hyperkit fails on pulling kube images China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument Mar 13, 2019
@tstromberg tstromberg added the area/networking networking issues label Mar 13, 2019
@tstromberg
Copy link
Contributor

Your DNS is responding with an address that apparently is not available. You can try this to possibly work-around the issue by fetching the images outside of the VM using:

minikube start --cache-images

Or by passing information to Docker so that it can use an HTTP proxy: https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

Let me know what you find out! Once #3714 is merged, you'll be able to specify --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

@tstromberg tstromberg added kind/bug Categorizes issue or PR as related to a bug. needs-solution-message Issues where where offering a solution for an error would be helpful priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Mar 13, 2019
@daniel-farina
Copy link

Deleting the local config fixed this issue for me:

rm -rf ~/.minikube

I run it again and all errors were gone.

@kingonion
Copy link

kingonion commented Apr 2, 2019 •
edited

In china, because of the GFW, you can't pull images from k8s.gcr.io without a VPN. But you can pull the needed images from registry.cn-hangzhou.aliyuncs.com/google_containers and tag them. If the needed images exist on the local system, the starting process will go on even with pulling error. I wrote a shell script that can help you install minikube on Linux OS easily. And you can also make a offline installation package by using the script. The project home is https://github.com/Kingonion/minikube-on-linux

@gbraad
Copy link
Contributor

gbraad commented Apr 2, 2019 via email

@maxnilz
Copy link

maxnilz commented Apr 17, 2019 •
edited

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

Hi, I have the same problem with you, after some digging, I found the docker pull request will first go to docker daemon via REST endpoint /images/create, and the docker daemon will take over the pulling work. In order to make docker daemon to use the proxy, we need to set http_proxy & https_proxy to the place where dockerd running, i.e, root user.

Here is my scenario:
OS: I'm using ubuntu18.04 desktop
Where the docker daemon is running: the docker daemon run in root user
What supervisor host the docker daemon: systemd
Do I set the https_proxy & http_proxy: yes, I set them for the normal user, but didn't set them for root user.

How I solve this:
Because I just want to set http_proxy & https_proxy for docker daemon(dockerd) only for root, I set the proxy info into systemd service unit, like this(for my computer, the config file is located here: /lib/systemd/system/docker.service:

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
Environment="http_proxy=http://127.0.0.1:7777"
Environment="https_proxy=http://127.0.0.1:7777"
ExecStart=/usr/bin/dockerd -D -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Hope this will help somebody who got the same error like me.

@singleye
Copy link

singleye commented Apr 24, 2019 •
edited

I had the same issue on 'MacOS 10.14.4 Majave'

Pull and tag images doesn't work for me

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

$ docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                                                         v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
k8s.gcr.io/kube-controller-manager                                            v1.14.0             b95b1efa0436        4 weeks ago         158MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.14.0             b95b1efa0436        4 weeks ago         158MB
k8s.gcr.io/kube-scheduler                                                     v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.14.0             ecf910f40d6e        4 weeks ago         210MB
k8s.gcr.io/kube-apiserver                                                     v1.14.0             ecf910f40d6e        4 weeks ago         210MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.3.1               eb516548c180        3 months ago        40.3MB
k8s.gcr.io/coredns                                                            1.3.1               eb516548c180        3 months ago        40.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/etcd                                                               3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/pause                                                              3.1                 da86e6ba6ca1        16 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        16 months ago       742kB

$ minikube start
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...r     y    d         r          r   s           7                   1                   4 m      a          2
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to compe_te ...                               3                   d                   1  m      a         7
✨  Preparing Kubernetes environment ...                                      9                   1                   1  m      a         2
🚜  Pulling images required by Kubernetes v1.14.0 ...
❌  Unable to pull images, which may be OK: runnigz cmd: sudo kubeadm cgnfig images pull --config /var/lib/kubeadm.yaml: command faierdv sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting heade
rs)
, error: exit status 1
: Process exited with status 1
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd scheduler controller dns           v                   5                   4 w     a           8
📯  Updating kube-proxy configuration ...
🤔  Verifying component health ......
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

But specify '--image-repository' works for me

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...          "                        6 w     a           E      (     6 w     a                                         f
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to complete ...                  6 w     a           E      (   6 w     a                                           v
✨  Preparing Kubernetes 4nvironment ..."                        6 w     a           E      (   6 w     a                                           h
🚜  Pulling images required by Kubernetes v1.14.0 ...
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd skhrduler controller dnsw     a           E      (   6 w     a                                           g
📯  Updating kube-proxy configuration ...
🤔  Verifying component health .....
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

BTW, I'm using minikube v1.0.0

$ minikube version
minikube version: v1.0.0

@tstromberg tstromberg removed the needs-solution-message Issues where where offering a solution for an error would be helpful label May 14, 2019
@tstromberg
Copy link
Contributor

v1.0.1 now includes a user-friendly message to resolve the original requesters issue. Thank you for opening the bug report!

@BingerCoder
Copy link

rm -rf ~/.minikube
and with --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
works for me ~

@jiran2
Copy link

jiran2 commented Dec 17, 2019

rm -rf ~/.minikube
and with --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
works for me ~

worked thanks

@hantsy
Copy link

hantsy commented Mar 9, 2020 •
edited

Used the mirror repo(registry.cn-hangzhou.aliyuncs.com/google_containers), still failed, there is checksum still using Google service. saving checksum file: getting storage object: Get https://www.googleapis.com/storage/v1/b/minikube-preloaded-volume-tarballs/o/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4?alt=json&prettyPrint=false&projection=full:

 minikube start --driver=hyperv --memory=4096m --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
* minikube v1.8.1 on Microsoft Windows 10 Pro 10.0.19041 Build 19041
* Using the hyperv driver based on user configuration
* Using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
* Downloading VM boot image ...
* Creating hyperv VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
* Downloading preloaded images tarball for k8s v1.17.3 ...
E0310 01:54:28.001249   12292 cache.go:116] Error caching images:  **saving checksum file: getting storage object: Get https://www.googleapis.com/storage/v1/b/minikube-preloaded-volume-tarballs/o/preloaded-images-k8s-v1-v1.17.3-docker-overlay2.tar.lz4?alt=json&prettyPrint=false&projection=full:** dial tcp 216.58.200.42:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
E0310 01:54:28.020668   12292 config.go:71] Failed to preload container runtime Docker: copying file: sudo test -d \ && sudo scp -t \ && sudo touch -d "2020-03-10 01:10:26.0875112 +0800" \/preloaded.tar.lz4: Process exited with status 1
output: , falling back to caching images

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking networking issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests