GitLab Helm chart is ready for Kubernetes 1.16 apiversion deprecations

Summary

Kubernetes 1.16 no longer will be serving extensions/* and apps/beta* API versions, all resources should stop using them

Relevant links

• https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
• https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/

@mendini @twk3 we should confirm the release cycle for this, and when any extension/beta API was moved into this state.

added For Scheduling backstage [DEPRECATED] groupdistribution priority2 severity3 labels

@ljlane @mendeni @joshlambert 1.16 is scheduled to be released here in September. We should probably include this update in our October release, so 12.4.

added Deliverable Scheduled labels and removed For Scheduling label

Setting devopsenablement based on groupdistribution.

added devopssystems label

Thanks. Will this impact upgrades? Thinking about people still on older versions, or that sometimes these fields can be immutable.

Both kubectl and helm handle apiVersion change gracefully.

An example of the grace that @redbaron1 brings up: deploy the charts onto K8S v1.15, then helm get xx and compare the API versions represented. Most of the time, these are auto-upgraded. However, when v1.16 come out, some of the older APIs will be full deprecated, resulting in the loss of automatic translation and resulting failure.

AFAIK, it is safe to change apiVersion in charts, tiller won't recognise these as new resource and will simply submit any other changes, should there be any, using new apiVersion

Kubernetes v1.16.0 has been released https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/ ... We can expect to see any impact reported pretty quickly.

changed the description

assigned to @twk3

changed milestone to %12.4

Hi, I can confirm there is an issue with k8s 1.16.

@matej.kern This is not an item for these Helm charts. You'll want to file an issue with gitlab-org/gitlab

Thanks for the info @WarheadsSE. I filed an issue #32722.

changed title from Be ready for Kubernetes 1.16 apiversion deprecations to GitLab Helm chart is ready for Kubernetes 1.16 apiversion deprecations

mentioned in merge request !975 (merged)

• Update for the apis directly in this repo are in !975 (merged)
• gitlab-runner also has updates ready in master, from gitlab-runner!167 (merged) but I have a followup to improve the upgrade here: gitlab-runner!181 (merged)
• grafana has been updated upstream, Our upgrade MR here: !976 (merged)
• prometheus has not yet been updated upstream, but there is a PR being worked on: https://github.com/helm/charts/pull/17268
• postgres is blocked by: #1535 (closed)
• certmanager is blocked by: #1568 (closed)

mentioned in merge request !976 (merged)

mentioned in issue #1602 (closed)

Sorry for the duplicate issue. Thank you for working on this. I really need this fix. :D

Im having problem with this in my enviroment even when trying 2.3.3 or 2.3.4 is it not commited to master? =)

Error: validation failed: [unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1", unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"]

Just as a temporary workaround (until this is fixed) you can add:

spec:
    ...
    - --runtime-config=apps/v1beta1=true,apps/v1beta2=true,extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true

... to /etc/kubernetes/manifests/kube-apiserver.yaml. Then,

kubectl -n kube-system delete pod kube-apiserver-whatever kube-controller-manager-whatever

This will re-enable the removed APIs and it will work. There is no telling what will happen when we re-enable them, though...

In any case, deployment with helm works fine afterwards.

That is a very useful tip for anyone that has upgraded their cluster and can't rollback. Thank you for providing this, much appreciated.

For those waiting on this. While we have updated our own templates already, which will be included in the October 22nd release. Our dependencies like certmanager and postgres will take us more time to update, and will require a major chart release. We are hoping to have this fully complete for our November 22nd release.

@twk3 Any updates? Since the issue still exists even after the November 22nd release.

We still need to complete our certmanager investigation, then actually schedule the work to do a major release of the chart. We are now expecting to complete this early in the new year.

We've completed the bulk of the research tasks for completing all the upgrades. And we have scheduled version 3.0 of our chart for January 22nd, aligned with GitLab 12.7

Upgrading to version 3 will require some manual steps for users on our included postgres database, and will only support upgrades from the 2.6 release chart (being released this weekend).

@renich It did install helm, and with that install the Ingress controller, but everything else failed afterward :

• Jupyterhub
• Gitlab Runner
• Cert-Manager
• Prometheus

All with pretty much the same kind of output :

+ helm init --upgrade
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /root/.helm.
Error: error installing: the server could not find the requested resource

My guess is that this is related to https://github.com/helm/helm/issues/6374. Essentially, from what I gather, everything helm-related is broken in kubernetes 1.16 until helm is updated, which is not the case yet.

@AlexandreOuellet What you're experiencing is with the Kubernetes integration features of gitlab/gitlab-foss, not with these GitLab helm charts. Please locate or open an issue within that project

Learned today we also need to bump the kubectl version we use, to be 1.13 in order to have a kubectl version that stops using the old apis: https://github.com/kubernetes/kubernetes/pull/70039

We have #1583 (closed) open to do this change

Does this relate to the issue I'm running into?

By following the handbook I'm getting:

Release "gitlab" does not exist. Installing it now.
Error: validation failed: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"

on the helm upgrade... command.

This is precise what you're hitting.

mentioned in issue #1583 (closed)

added missed:12.4 label

marked this issue as related to #1654 (closed)

marked #1654 (closed) as a duplicate of this issue

mentioned in issue #1654 (closed)

changed milestone to %12.5

added missed-deliverable label

Release "production" does not exist. Installing it now.

Error: validation failed: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"

ERROR: Job failed: exit code 1

I'm having the same issue as David H.

stable/prometheus has been updated to change the API versions as of dac63a339b5203e26ca912a7cb590d8c385c4b44. We can now evaluate this for breaking changes compared to how we have it included.

We now await further work on

• stable/postgresql (#1535 (closed), #1536 (closed))
• stable/cert-manager (#1568 (closed))
• stable/redis (#1519 (closed)) (breaking change, our in-chart redis is a fork but API ready.)

james@server-01:~$ helm upgrade --install gitlab gitlab/gitlab   --timeout 600   --set global.hosts.domain=foo.lan   --set global.hosts.externalIP=69.6.6.35   --set certmanager-issuer.email=james@foo.com --set global.edition=ce --set certmanager.rbac.create=false --set nginx-ingress.rbac.createRole=false --set prometheus.rbac.create=false --set gitlab-runner.rbac.create=false
...
Error: validation failed: [unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta1", unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"]

james@server-01:~$ kubectl logs -n kube-system tiller-deploy-684c9f98f5-fnngb -f
...
[tiller] 2019/11/07 19:55:00 performing install for gitlab
[tiller] 2019/11/07 19:55:00 executing 8 crd-install hooks for gitlab
[tiller] 2019/11/07 19:55:00 hooks complete for crd-install gitlab
[tiller] 2019/11/07 19:55:01 failed install perform step: validation failed: [unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta1", unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"]

What's requested for a fresh install until this is resolved? Thanks so much.

This blows my mind that Kubernetes 1.16 would be released and break so much shit. Basically every Helm guide out there doesn't work anymore. Facepalm.

The breaking API has been deprecated for a long time now. It's just that everybody decided to ignore it.

mentioned in issue #1352 (closed)

@WarheadsSE should we add a deprecation notice in the docs? @ehenley noticed that https://docs.gitlab.com/charts/development/minikube/index.html doesn't work with 1.16.

The recommended version in GDK appears to be 1.15.4 https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/prometheus.md#start-minikube.

doesn't work with 1.16.

I don't think a deprecation notice in the docs makes any sense at the moment. There is nothing to deprecate. Perhaps a limitation, until such time as we have all components working with 1.16 our of the box (e.g. closing this issue).

mentioned in merge request !1039 (merged)

Looking at the prometheus update. Looks like we need to do it in two steps to avoid breaking changes. Working on it here: !1039 (merged)

We will go ahead and update to the latest non-breaking this release, then be able to update fully in the major release.

added to epic &6 (closed)

mentioned in issue #1702 (closed)

added missed:12.5 label

changed milestone to %12.6

mentioned in issue #1776 (closed)

changed milestone to %12.7