Description

Provide a brief description of your issue here. For example:
Installed OpenShift 3.1.1 into Redhat 7.6 and got error: Could find csr for nodes

On a multi master install, if the first master goes down we can no: N/A only 1 master
longer scaleup the cluster with new nodes or masters: N/A

Version

Please put the following version information in the code block
indicated below.

• Your ansible version per ansible --version
  ansible 2.6.14
  config file = /usr/share/ansible/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

If you're running from playbooks installed via RPM

• The output of rpm -q openshift-ansible
  ansible-2.6.14-1.el7ae.noarch
  Place the output between the code block below:

VERSION INFORMATION HERE PLEASE

Steps To Reproduce

Step1: Prepare VMs:
I have 4 Redhat7.6 VMs
Followed the doc https://docs.openshift.com/container-platform/3.11/install/index.html
to install setup hosts, Inventory File (/etc/ansible/hosts):

Create an OSEv3 group that contains the masters, nodes,

[OSEv3:children]
masters
nodes
etcd

Set variables common for all OSEv3 hosts

[OSEv3:vars]
os_firewall_use_firewalld=True

SSH user, this user should allow ssh based auth without requiring a password

ansible_ssh_user=root

If ansible_ssh_user is not root, ansible_become must be set to true

ansible_become=false
openshift_master_default_subdomain=apps.fyre.ibm.com
openshift_deployment_type=openshift-enterprise
oreg_url=registry.redhat.io/openshift3/ose-${component}:${version}
oreg_auth_user=
oreg_auth_password=xxxxxxxxxxxxxxxxxxxxxx

uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider

#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]

host group for masters

[masters]
scaorh-master.fyre.ibm.com

host group for etcd

[etcd]
scaorh-master.fyre.ibm.com

host group for nodes, includes region info

[nodes]
scaorh-master.fyre.ibm.com openshift_node_group_name='node-config-master'
scaorh-worker1.fyre.ibm.com openshift_node_group_name='node-config-compute'
scaorh1-worker2.fyre.ibm.com openshift_node_group_name='node-config-compute'
scaorh2-infranode.fyre.ibm.com openshift_node_group_name='node-config-infra'

Step 2: deploy:
cd /usr/share/ansible/openshift-ansible
Run:
ansible-playbook -i /etc/ansible/hosts playbooks/prerequisites.yml

ansible-playbook -i /etc/ansible/hosts playbooks/deploy_cluster.yml
Got error:
TASK [Approve node certificates when bootstrapping] ***********************************************************
Sunday 17 March 2019 12:36:15 -0700 (0:00:00.137) 0:30:15.928 **********
FAILED - RETRYING: Approve node certificates when bootstrapping (30 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (29 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (28 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (27 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (26 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (25 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (24 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (23 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (22 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (21 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (20 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (19 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (18 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (17 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (16 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (15 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (14 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (13 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (12 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (11 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (10 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (9 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (8 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (7 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (6 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (5 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (4 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (3 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (2 retries left).
FAILED - RETRYING: Approve node certificates when bootstrapping (1 retries left).
fatal: [scaorh-master.fyre.ibm.com]: FAILED! => {"all_subjects_found": ["subject=/O=system:nodes/CN=system:node:scaorh-master.fyre.ibm.com\n", "subject=/O=system:nodes/CN=system:node:scaorh-master.fyre.ibm.com\n", "subject=/O=system:nodes/CN=system:node:scaorh-master.fyre.ibm.com\n", "subject=/O=system:nodes/CN=system:node:scaorh-master.fyre.ibm.com\n", "subject=/O=system:nodes/CN=system:node:scaorh1-worker2.fyre.ibm.com\n", "subject=/O=system:nodes/CN=system:node:scaorh-worker1.fyre.ibm.com\n"], "attempts": 30, "changed": false, "client_approve_results": [], "client_csrs": {"node-csr-8e-uSNcl4xSbMe02CoIcaelY5mjC1eqCIXaXEu4Vjco": "scaorh1-worker2.fyre.ibm.com", "node-csr-J-1_iIVS5-hgaQz5xGifBwWTf5l4CcXgvOzvKs7yufU": "scaorh-worker1.fyre.ibm.com"}, "msg": "Could not find csr for nodes: scaorh2-infranode.fyre.ibm.com", "oc_get_nodes": {"apiVersion": "v1", "items": [{"apiVersion": "v1", "kind": "Node", "metadata": {"annotations": {"node.openshift.io/md5sum": "6ada87691866d0068b8c8cfe0df773b2", "volumes.kubernetes.io/controller-managed-attach-detach": "true"}, "creationTimestamp": "2019-03-17T19:26:30Z", "labels": {"beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "scaorh-master.fyre.ibm.com", "node-role.kubernetes.io/master": "true"}, "name": "scaorh-master.fyre.ibm.com", "namespace": "", "resourceVersion": "2860", "selfLink": "/api/v1/nodes/scaorh-master.fyre.ibm.com", "uid": "90c98d93-48ea-11e9-bf0d-00163e01f117"}, "spec": {}, "status": {"addresses": [{"address": "172.16.241.23", "type": "InternalIP"}, {"address": "scaorh-master.fyre.ibm.com", "type": "Hostname"}], "allocatable": {"cpu": "16", "hugepages-1Gi": "0", "hugepages-2Mi": "0", "memory": "32676344Ki", "pods": "250"}, "capacity": {"cpu": "16", "hugepages-1Gi": "0", "hugepages-2Mi": "0", "memory": "32778744Ki", "pods": "250"}, "conditions": [{"lastHeartbeatTime": "2019-03-17T19:39:14Z", "lastTransitionTime": "2019-03-17T19:26:30Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk"}, {"lastHeartbeatTime": "2019-03-17T19:39:14Z", "lastTransitionTime": "2019-03-17T19:26:30Z", "message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure"}, {"lastHeartbeatTime": "2019-03-17T19:39:14Z", "lastTransitionTime": "2019-03-17T19:26:30Z", "message": "kubelet has no disk pressure", "reason": "KubeletHasNoDiskPressure", "status": "False", "type": "DiskPressure"}, {"lastHeartbeatTime": "2019-03-17T19:39:14Z", "lastTransitionTime": "2019-03-17T19:26:30Z", "message": "kubelet has sufficient PID available", "reason": "KubeletHasSufficientPID", "status": "False", "type": "PIDPressure"}, {"lastHeartbeatTime": "2019-03-17T19:39:14Z", "lastTransitionTime": "2019-03-17T19:26:30Z", "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", "reason": "KubeletNotReady", "status": "False", "type": "Ready"}], "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"names": ["registry.redhat.io/openshift3/ose-node@sha256:8d28f961c74f033b3df9ed0d7a2a1bfb5e6ebb0611cb6b018f7e623961f7ea52", "registry.redhat.io/openshift3/ose-node:v3.11"], "sizeBytes": 1171108452}, {"names": ["registry.redhat.io/openshift3/ose-control-plane@sha256:200a14df0fdf3c467588f5067ab015cd316e49856114ba7602d4ca9e5f42b0f3", "registry.redhat.io/openshift3/ose-control-plane:v3.11"], "sizeBytes": 808610884}, {"names": ["registry.redhat.io/rhel7/etcd@sha256:be1c3e3f002ac41c35f2994f1c0cb3bd28a8ff59674941ca1a6223a8b72c2758", "registry.redhat.io/rhel7/etcd:3.2.22"], "sizeBytes": 259048769}, {"names": ["registry.redhat.io/openshift3/ose-pod@sha256:f27c68d225803ca3a97149083b5211ccc3def3230f8147fd017eef5b11d866d5", "registry.redhat.io/openshift3/ose-pod:v3.11", "registry.redhat.io/openshift3/ose-pod:v3.11.88"], "sizeBytes": 238366131}], "nodeInfo": {"architecture": "amd64", "bootID": "bdeaf185-56b0-4cff-b344-2fe95351d324", "containerRuntimeVersion": "docker://1.13.1", "kernelVersion": "3.10.0-957.5.1.el7.x86_64", "kubeProxyVersion": "v1.11.0+d4cacc0", "kubeletVersion": "v1.11.0+d4cacc0", "machineID": "cbb00030e5204543a0474ffff17ec26f", "operatingSystem": "linux", "osImage": "OpenShift Enterprise", "systemUUID": "E21E048B-6EB8-4685-A3EA-57F5CF1F2BF3"}}}], "kind": "List", "metadata": {"resourceVersion": "", "selfLink": ""}}, "raw_failures": [], "rc": 0, "server_approve_results": [], "server_csrs": null, "state": "unknown", "unwanted_csrs": [{"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2019-03-17T19:36:13Z", "generateName": "csr-", "name": "csr-58dj9", "namespace": "", "resourceVersion": "2555", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-58dj9", "uid": "ecbad18b-48eb-11e9-bf0d-00163e01f117"}, "spec": {"groups": ["system:nodes", "system:authenticated"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlR6Q0I5Z0lCQURCSU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14THpBdEJnTlZCQU1USm5ONQpjM1JsYlRwdWIyUmxPbk5qWVc5eWFDMXRZWE4wWlhJdVpubHlaUzVwWW0wdVkyOXRNRmt3RXdZSEtvWkl6ajBDCkFRWUlLb1pJemowREFRY0RRZ0FFS1VZbGZFai9WUlFQL09ETFpORDFMYXh4VnNGc0RaSllTeDBkOGdEUityWVcKaC9rUUhFL0QvVHE4SHIwOENRT2pQaGlkbHFGWkZjcExkQlpMSVdQcWdLQk1NRW9HQ1NxR1NJYjNEUUVKRGpFOQpNRHN3T1FZRFZSMFJCREl3TUlJYWMyTmhiM0pvTFcxaGMzUmxjaTVtZVhKbExtbGliUzVqYjIyQ0FJY0VyQkR4CkY0Y0VDUjdDbzRjRXJCRUFBVEFLQmdncWhrak9QUVFEQWdOSUFEQkZBaUJMRmVrbmRjVm4zSGlYNGVwN0ZOMi8KTi9WYm5VbXlINmhTb1VOUFowTWE1Z0loQU5zdGU4QUNSR1BnWGNIS3YzT0g3cnNEWk92N1FuVm5XOFNOUWZUTwpzMm9rCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:node:scaorh-master.fyre.ibm.com"}, "status": {}}, {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2019-03-17T19:26:52Z", "generateName": "csr-", "name": "csr-lzvjj", "namespace": "", "resourceVersion": "949", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-lzvjj", "uid": "9e264342-48ea-11e9-bf0d-00163e01f117"}, "spec": {"groups": ["system:masters", "system:cluster-admins", "system:authenticated"], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQkJEQ0JxZ0lCQURCSU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14THpBdEJnTlZCQU1USm5ONQpjM1JsYlRwdWIyUmxPbk5qWVc5eWFDMXRZWE4wWlhJdVpubHlaUzVwWW0wdVkyOXRNRmt3RXdZSEtvWkl6ajBDCkFRWUlLb1pJemowREFRY0RRZ0FFdm1CRmppdm9qMlBkWDJyRmM0eE5rVERSYjROclVWSGRCRDFNRk50OHV2L1AKdTZ3aUdVbTZpdTRqOVdrb2Y1TS9LOUE2eGRBdVRlUzU2WkRRaEdNSllxQUFNQW9HQ0NxR1NNNDlCQU1DQTBrQQpNRVlDSVFDS3o4dVBqcSt0ZzJwNkNxdC9NZks0OGQ2cjFFWUNEeHRhcmFjMlRpN3I1QUloQU4yeUY2QVlUcU5LCmhNVlJKSTJIMzIxVWN0R08zRi9wbTltL1IreDhYMTFuCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=", "usages": ["digital signature", "key encipherment", "client auth"], "username": "system:admin"}, "status": {"certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNoVENDQVcyZ0F3SUJBZ0lVSEFMc0FQQXNlYXFmUUhpUytMM2hIWHFEWDZzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd
PLAY RECAP ****************************************************************************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
scaorh-master.fyre.ibm.com : ok=487 changed=238 unreachable=0 failed=1
scaorh-worker1.fyre.ibm.com : ok=109 changed=66 unreachable=0 failed=0
scaorh1-worker2.fyre.ibm.com : ok=109 changed=66 unreachable=0 failed=0
scaorh2-infranode.fyre.ibm.com : ok=101 changed=19 unreachable=0 failed=0

INSTALLER STATUS **********************************************************************************************
Initialization : Complete (0:00:25)
Health Check : Complete (0:00:55)
Node Bootstrap Preparation : Complete (0:13:04)
etcd Install : Complete (0:02:25)
Master Install : Complete (0:07:07)
Master Additional Install : Complete (0:06:11)
Node Join : In Progress (0:03:06)
This phase can be restarted by running: playbooks/openshift-node/join.yml
Sunday 17 March 2019 12:39:16 -0700 (0:03:01.349) 0:33:17.277 **********

cockpit : Install cockpit-ws ------------------------------------------------------------------------- 316.13s
openshift_node : install needed rpm(s) --------------------------------------------------------------- 237.61s
Approve node certificates when bootstrapping --------------------------------------------------------- 181.35s
openshift_node : Install iSCSI storage plugin dependencies ------------------------------------------- 120.08s
openshift_node : Install node, clients, and conntrack packages --------------------------------------- 103.55s
etcd : Install etcd ----------------------------------------------------------------------------------- 83.24s
openshift_control_plane : Wait for all control plane pods to become ready ----------------------------- 70.09s
Run health checks (install) - EL ---------------------------------------------------------------------- 54.79s
openshift_control_plane : Wait for control plane pods to appear --------------------------------------- 54.14s
openshift_node : Install Ceph storage plugin dependencies --------------------------------------------- 47.59s
openshift_node : Install dnsmasq ---------------------------------------------------------------------- 46.75s
openshift_ca : Install the base package for admin tooling --------------------------------------------- 45.79s
openshift_node : Install GlusterFS storage plugin dependencies ---------------------------------------- 43.07s
openshift_excluder : Install openshift excluder - yum ------------------------------------------------- 39.41s
openshift_excluder : Install docker excluder - yum ---------------------------------------------------- 24.91s
openshift_cli : Install clients ----------------------------------------------------------------------- 24.76s
openshift_node_group : Wait for the sync daemonset to become ready and available ---------------------- 11.54s
openshift_manageiq : Configure role/user permissions -------------------------------------------------- 10.10s
nickhammond.logrotate : nickhammond.logrotate | Install logrotate -------------------------------------- 9.12s
openshift_node : Install NFS storage plugin dependencies ----------------------------------------------- 8.84s

Failure summary:

1. Hosts: scaorh-master.fyre.ibm.com
   Play: Approve any pending CSR requests from inventory nodes
   Task: Approve node certificates when bootstrapping
   Message: Could not find csr for nodes: scaorh2-infranode.fyre.ibm.com

Expected Results

Describe what you expected to happen.

Example command and output or error messages

Observed Results

Describe what is actually happening.

Example command and output or error messages

For long output or logs, consider using a gist

Additional Information

Provide any additional information which may help us diagnose the
issue.

• Your operating system and version, ie: RHEL 7.2, Fedora 23 ($ cat /etc/redhat-release)
• Your inventory file (especially any non-standard configuration parameters)
• Sample code, etc
  Red Hat Enterprise Linux Server release 7.6 (Maipo)

EXTRA INFORMATION GOES HERE