CHECK [memory_availability] fail when run ansible playbook on OKD 3.11

[dearteno]

CHECK [memory_availability : ip-10-0-1-238.ap-southeast-1.compute.internal] ****
fatal: [ip-10-0-1-238.ap-southeast-1.compute.internal]: FAILED! => {"changed": true, "checks": {"disk_availability": {}, "docker_image_availability": {"changed": true, "failed": true, "failures": [["OpenShiftCheckException", "One or more required container images are not available:\n quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"]], "msg": "One or more required container images are not available:\n quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"}, "docker_storage": {"changed": true, "data_pct_used": 0.023671361645169142, "data_threshold": 90.0, "data_total": "34.18 GB", "data_total_bytes": 36700495544.32, "data_used": "19.92 MB", "data_used_bytes": 20887633.92, "metadata_pct_used": 0.00011372055802915023, "metadata_threshold": 90.0, "metadata_total": "88.08 MB", "metadata_total_bytes": 92358574.08, "metadata_used": "57.34 kB", "metadata_used_bytes": 58716.16, "msg": "Thinpool usage is within thresholds.", "vg_free": "48.00g", "vg_free_bytes": 51539607552.0}, "memory_availability": {}, "package_availability": {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", "bash-completion", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", "flannel", "glusterfs-fuse", "httpd-tools", "iptables", "iptables-services", "iscsi-initiator-utils", "libselinux-python", "nfs-utils", "ntp", "openssl", "origin", "origin-clients", "origin-hyperkube", "origin-node", "pyparted", "python-httplib2", "yum-utils"]}}}, "package_version": {"changed": false, "invocation": {"module_args": {"package_list": [{"check_multi": false, "name": "origin", "version": "3.11"}, {"check_multi": false, "name": "origin-master", "version": "3.11"}, {"check_multi": false, "name": "origin-node", "version": "3.11"}], "package_mgr": "yum"}}}}, "msg": "One or more checks failed", "playbook_context": "install"}

  1. Hosts:    ip-10-0-1-182.ap-southeast-1.compute.internal
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required container images are not available:
                   quay.io/coreos/etcd:v3.2.22
               Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>```

[dwmkerr]

Will take a look into this shortly. Any special changes you made to the setup that made it non-standard?

[dearteno]

no, just make infrastructure and make openshift.

[dwmkerr]

Have you tried rerunning make openshift? It can be a little flakey on the ansible side, but is idempotent so safe to run and rerun

…

On Fri, 5 Apr 2019 at 12:21 PM, Pichate Ins ***@***.***> wrote: no, just make infrastructure and make openshift. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#85 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1nSATxXHb6rbEk7XbbboA8-GUl-fY0ks5vds9EgaJpZM4cc_F9> .

[dearteno]

i try to run many time. but still got same error

[dwmkerr]

I have completely cleaned and retried locally, everything works:

I think you are encountering the below:

openshift/openshift-ansible#11369

I have some other things I'm looking into (see below) but for now I think it is the issue above. Will update this comment as tI debug more. There's not been any activity on that issue for a while - have you tried again recently? It might have been that the image was just unavailable for a period of time...

---

Potential things to try

1. try deleting your local .terraform folder (after running terraform destroy) and then re-run terraform init && terraform get so that you get the absolute latest version of the AWS providers.
2. Let me know which region you are running in, if not the default

Similar Issues

Searching for similar issues, I've found these:

• One or more required container images are not available: registry.access.redhat.com/openshift3/registry-console openshift/openshift-ansible#6002
• OpenShift 3.9 origin install on aws fails memory_availability check due to skopeo not finding etcd on registry.access.redhat.com openshift/openshift-ansible#7808
• Problem with getting images during installation openshift/openshift-ansible#6300
• https://bugzilla.redhat.com/show_bug.cgi?id=1480195

Summarising:

1. The first suggests the issue is due to transient failures or outages in the redhat container registries it is trying to pull images from. Highly unlikely to be your issue as it is failing consistently.
2. The second was the wrong registry
3. The third was proxy settings for a registry

In all cases it is a failure to download a specific image. In your case, the specific image which is failing to download is:

As indicated by:

["OpenShiftCheckException", "One or more required container images are not available:
 quay.io/coreos/etcd:v3.2.22
Checked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"]]

So right now it looks like an issue with quay.io/coreos/etcd:v3.2.22

[dearteno]

I was testing on ap-southeast-1 got the same issue but the ap-southeast-2 was working smoothly

[dwmkerr]

That is bizarre.... Let me see if I can repro in the same way

[ashishxooa]

I am also getting the same issue, it worked yesterday, I destroyed everything and then when I tried it again, getting the issue even after several retries, deleted the tf state files and tried everything from beginning still getting the same issue.

[efess]

I was getting this as well. The issue may be related to this issue:

openshift/openshift-ansible#10741

The workaround to disable the availability check worked for me and I was able to install this cluster successfully.

[FridMehdi]

@dwmkerr hello any workaround for this issue ? I have the same problem

[lejeczek]

I too see the same with openshift-ansible-3.11.37-1.git.0.3b8b341.el7.noarch on Centos 7.
First is:
CHECK [memory_availability : ....
followed by:
Message: One or more checks failed Details: check "docker_image_availability": One or more required container images are not available: docker.io/cockpit/kubernetes:latest, docker.io/openshift/origin-deployer:v3.11, docker.io/openshift/origin-docker-registry:v3.11, docker.io/openshift/origin-haproxy-router:v3.11, docker.io/openshift/origin-pod:v3.11 Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>

check "docker_storage": Could not find imported module support code for docker_info. Looked for either AnsibleDockerClient.py or docker_common.py Traceback (most recent call last): File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check result = check.run() File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_storage.py", line 53, in run docker_info = self.execute_module("docker_info", {}) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/__init__.py", line 211, in execute_module result = self._execute_module(module_name, module_args, self.tmp, self.task_vars) File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 815, in _execute_module (module_style, shebang, module_data, module_path) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars) File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 209, in _configure_module **become_kwargs) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1258, in modify_module environment=environment) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1095, in _find_module_utils py_module_cache, zf) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 764, in recursive_finder raise AnsibleError(' '.join(msg)) AnsibleError: Could not find imported module support code for docker_info. Looked for either AnsibleDockerClient.py or docker_common.py
I'm on bare-metal and behind a proxy but I put proxy vars in all the places I know of. Commands: docker, skopeo succeed when ran manually.

regards, L.