2.1.4 - failed to create task for container: failed to create shim task: OCI runtime create failed

[lightson]

I've been using plausible successully well over a year now. Today I had reason to reboot my server and, whoof, suddenly it won't start.

Has anyone run into the below before? I can't figure it out for the life of me. Prior to rebooting today there was no issue, and it's always been fine in the past after reboots.

I run docker-compose up -d as normal, but then I get

---

Starting data_plausible_events_db_1 ... error
Starting data_plausible_db_1 ...

ERROR: for data_plausible_events_db_1 Cannot start service plausible_events_db: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peerStarting data_plausible_db_1 ... error

ERROR: for data_plausible_db_1 Cannot start service plausible_db: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown

ERROR: for plausible_events_db Cannot start service plausible_events_db: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown

ERROR: for plausible_db Cannot start service plausible_db: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown

ERROR: Encountered errors while bringing up the project.

[ruslandoga]

👋 @lightson

Have you tried "turning it off and on"? :)

Like sudo systemctl restart docker followed by another docker compose up -d

[lightson]

will give it a go. Probably more of an issue with docker than plausible, but it's so irritating.

[lightson]

Alas, no change. Irritating!

[ruslandoga]

If updating Docker runtime didn't work out, you can try to get the logs for the Docker daemon and check if there is any information about the error. For example, journalctl -u docker --since "2024-11-01"

[lightson]

Seems I'm having an issue with runc.

Nov 05 12:22:57 localhost systemd[1]: Stopped Docker Application Container Engine.
Nov 05 12:22:57 localhost systemd[1]: docker.service: Start request repeated too quickly.
Nov 05 12:22:57 localhost systemd[1]: docker.service: Failed with result 'exit-code'.
Nov 05 12:22:57 localhost systemd[1]: Failed to start Docker Application Container Engine.
Nov 05 12:24:01 localhost systemd[1]: Starting Docker Application Container Engine...
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.239417667Z" level=info msg="Starting up"
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.240995057Z" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously"
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.442850183Z" level=warning msg="Failed to run [runc features]: """ error="signal: segmentation fault (core dumped)"
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.630387544Z" level=warning msg="Failed to run [runc features]: """ error="signal: segmentation fault (core dumped)"
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.657765652Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.680794139Z" level=info msg="Loading containers: start."
Nov 05 12:24:01 localhost dockerd[87805]: time="2024-11-05T12:24:01.705723434Z" level=info msg="Firewalld: docker zone already exists, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.003314360Z" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.025516247Z" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.138345493Z" level=info msg="Firewalld: interface br-d8fb16d16f40 already part of docker zone, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.161145094Z" level=info msg="Firewalld: interface br-d8fb16d16f40 already part of docker zone, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.373787605Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.459951161Z" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569005211Z" level=warning msg="error locating sandbox id 10bd7a9f01affdc175a473321a5044f92bdd80151bed3bbb3314c4f98b52e6d2: sandbox 10bd7a9f01affdc175a473321a5044f92bdd80151bed3bbb3314c4f98b52e6d2 not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569064433Z" level=warning msg="error locating sandbox id 617ba9951dc1475aeeebd579c57d67fc2edcbf1fb00a72c6614dfc59a7ad7286: sandbox 617ba9951dc1475aeeebd579c57d67fc2edcbf1fb00a72c6614dfc59a7ad7286 not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569094263Z" level=warning msg="error locating sandbox id 4a8f71354c39d4420f9856d58b1403dc14c2f0c829ec3989beb83b4114a7fb40: sandbox 4a8f71354c39d4420f9856d58b1403dc14c2f0c829ec3989beb83b4114a7fb40 not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569131554Z" level=warning msg="error locating sandbox id a7d31ce014ee19376d8cbe05113882ebb94f7ef419d1edc6017eda874e1a4bb9: sandbox a7d31ce014ee19376d8cbe05113882ebb94f7ef419d1edc6017eda874e1a4bb9 not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569157215Z" level=warning msg="error locating sandbox id 49cb16c3c1dfbd6237b1b04fffdd4eb9c1f303730641e1a625447025791e1f23: sandbox 49cb16c3c1dfbd6237b1b04fffdd4eb9c1f303730641e1a625447025791e1f23 not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569176435Z" level=warning msg="error locating sandbox id 3b673f64f6196d963c38d6803fac34d60bd91a92a5c07e12a970148fe29d1edf: sandbox 3b673f64f6196d963c38d6803fac34d60bd91a92a5c07e12a970148fe29d1edf not>
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.569400041Z" level=info msg="Loading containers: done."
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.764723451Z" level=warning msg="Failed to retrieve default runtime version" error="failed to retrieve runc version: signal: segmentation fault (core dumped)"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.774091739Z" level=warning msg="WARNING: No swap limit support"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.774284084Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.774385437Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.774500710Z" level=info msg="Docker daemon" commit=41ca978 containerd-snapshotter=false storage-driver=overlay2 version=27.3.1
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.774662204Z" level=info msg="Daemon has completed initialization"
Nov 05 12:24:02 localhost dockerd[87805]: time="2024-11-05T12:24:02.798569983Z" level=info msg="API listen on /run/docker.sock"
Nov 05 12:24:02 localhost systemd[1]: Started Docker Application Container Engine.
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.079456831Z" level=warning msg="Failed to retrieve default runtime version" error="failed to retrieve runc version: signal: segmentation fault (core dumped)" spanID=f9bbe370c9a0cb12 traceID=c5e53bd2bf>
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.758286077Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.758377059Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.796522372Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.797303922Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.855735501Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:17 localhost dockerd[87805]: time="2024-11-05T12:24:17.855813723Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:36 localhost dockerd[87805]: time="2024-11-05T12:24:36.123108908Z" level=warning msg="Failed to retrieve default runtime version" error="failed to retrieve runc version: signal: segmentation fault (core dumped)" spanID=d598234f6a3953bf traceID=c11492db19>
Nov 05 12:24:39 localhost dockerd[87805]: time="2024-11-05T12:24:39.668376192Z" level=warning msg="Failed to retrieve default runtime version" error="failed to retrieve runc version: signal: segmentation fault (core dumped)" spanID=ac4f241fbb803df8 traceID=fb9aeaf4ac>
Nov 05 12:24:39 localhost dockerd[87805]: time="2024-11-05T12:24:39.780008008Z" level=info msg="Firewalld: interface br-5f0fdc7a53bf already part of docker zone, returning"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.102568463Z" level=warning msg="Failed to retrieve default runtime version" error="failed to retrieve runc version: signal: segmentation fault (core dumped)" spanID=e3b9599d1a15cd10 traceID=fb9aeaf4ac>
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.880546250Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.882636143Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.936671810Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.937047930Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.958567599Z" level=error msg="stream copy error: reading from a closed fifo"
Nov 05 12:24:40 localhost dockerd[87805]: time="2024-11-05T12:24:40.959768309Z" level=error msg="stream copy error: reading from a closed fifo"
lines 130-194/194 (END)

[lightson]

Seems it's my docker version. I downgraded and now docker spawns fine, I see clickhouse-serv in TOP, but still get a 503. Probably due to other changes I've made. Progress though, and clearly nothing wrong with plausible itself.