Starting dockerd failed

[garaujo23]

Hi there, I have an issue where if a camera unexpectedly reboots (power or memory) then on reboot I get the following errors:

2022-08-05T10:39:30.172+10:00 axis-b8a44f3be4f1 [ ERR     ] dockerdwrapperwithcompose[1353]: Couldn't identify the file system of the SD card at /var/spool/storage/SD_DISK
2022-08-05T10:39:30.179+10:00 axis-b8a44f3be4f1 [ ERR     ] dockerdwrapperwithcompose[1353]: Starting dockerd failed

I am using Docker Daemon with Compose version 1.2.3, the camera is P3265-LV, Axis OS version 10.9.4. After reboot if I stop the app and start it again, it works perfectly fine. I have a 126gb sd card installed and formatted as per below:

[garaujo23]

Found a new issue with the docker compose version 1.2.3 and sd card, see logs below, showing the failure to start and then on restart today its working fine.

2023-01-22T13:01:15.554+11:00 axis-b8a44f3be4f1 [ ERR     ] dockerdwrapperwithcompose[1386]: Cannot store data on the SD card, no storage exists at /var/spool/storage/SD_DISK
2023-01-22T13:01:15.611+11:00 axis-b8a44f3be4f1 [ ERR     ] dockerdwrapperwithcompose[1386]: Couldn't identify the file system of the SD card at /var/spool/storage/SD_DISK
2023-01-22T13:01:15.611+11:00 axis-b8a44f3be4f1 [ ERR     ] dockerdwrapperwithcompose[1386]: Starting dockerd failed
2023-02-07T09:57:55.296+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869365]: Started logging.
2023-02-07T09:57:55.324+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869365]: Starting dockerd in unsecured mode using SD card as storage with IPC socket.
2023-02-07T09:57:56.016+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:56.015153440+11:00" level=info msg="Starting up"
2023-02-07T09:57:56.025+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:56.024238080+11:00" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
2023-02-07T09:57:56.025+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:56.024382000+11:00" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
2023-02-07T09:57:57.035+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.033618400+11:00" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2023-02-07T09:57:57.055+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.055291480+11:00" level=info msg="parsed scheme: \"unix\"" module=grpc
2023-02-07T09:57:57.055+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.055410080+11:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2023-02-07T09:57:57.055+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.055511840+11:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2023-02-07T09:57:57.055+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.055575160+11:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-02-07T09:57:57.140+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.139776280+11:00" level=info msg="parsed scheme: \"unix\"" module=grpc
2023-02-07T09:57:57.140+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.139869280+11:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2023-02-07T09:57:57.140+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.139967240+11:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2023-02-07T09:57:57.140+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.140021240+11:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-02-07T09:57:57.281+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.281337440+11:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
2023-02-07T09:57:57.414+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.413456680+11:00" level=warning msg="Your kernel does not support CPU realtime scheduler"
2023-02-07T09:57:57.414+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.413566920+11:00" level=warning msg="Your kernel does not support cgroup blkio weight"
2023-02-07T09:57:57.414+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.413603720+11:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
2023-02-07T09:57:57.414+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:57.414198600+11:00" level=info msg="Loading containers: start."
2023-02-07T09:57:58.452+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.450735760+11:00" level=info msg="Removing stale sandbox 578d19df653df35e9fe7fe53e4e992c16cab8f92f5abb27ba91b78de8b93acc2 (a81d9476197e0a1ccf1ab03f9f63e7916e96597e075f140efa0e31f3d5c65b5a)"
2023-02-07T09:57:58.455+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.454757080+11:00" level=warning msg="Failed to delete conntrack state for 172.18.0.2: invalid argument"
2023-02-07T09:57:58.472+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.472351880+11:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 112e88c537124861d74be7ca8ccd0acaa8a8454cc0bcb3c56300039b7bb97539 6b1523d0dd07aceaf94b49619f654a5bfa281aa7e4bb8dc24fd8edb10ee72052], retrying...."
2023-02-07T09:57:58.811+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.810954800+11:00" level=info msg="Removing stale sandbox 9cf614aacdd0fc65d531912779a69c786e08ea1374730a6d89e1314ed91dba87 (95f365f742722ac09c4f30abfce0a754c6840c24c3c96f40162debb44f7eed4d)"
2023-02-07T09:57:58.815+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.814855480+11:00" level=warning msg="Failed to delete conntrack state for 172.18.0.3: invalid argument"
2023-02-07T09:57:58.825+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.824408160+11:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 112e88c537124861d74be7ca8ccd0acaa8a8454cc0bcb3c56300039b7bb97539 cb5fc7c42532804f459ce73c506415bb41db39b532855dd2988abf8f0791fd4b], retrying...."
2023-02-07T09:57:58.950+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:58.949749240+11:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2023-02-07T09:57:59.059+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:59.059374760+11:00" level=info msg="Loading containers: done."
2023-02-07T09:57:59.471+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:59.468703400+11:00" level=info msg="Docker daemon" commit=unsupported graphdriver(s)=overlay2 version=dev
2023-02-07T09:57:59.473+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:59.471954280+11:00" level=info msg="Daemon has completed initialization"
2023-02-07T09:57:59.539+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:59.539291360+11:00" level=info msg="API listen on [::]:2375"
2023-02-07T09:57:59.570+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:57:59.569499760+11:00" level=info msg="API listen on /var/run/docker.sock"
2023-02-07T09:58:39.157+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.048688920+11:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
2023-02-07T09:58:39.157+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.048800680+11:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
2023-02-07T09:58:39.157+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.126065840+11:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
2023-02-07T09:58:39.157+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.126169400+11:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
2023-02-07T09:58:39.231+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.193100320+11:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
2023-02-07T09:58:39.231+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:39.193228240+11:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
2023-02-07T09:58:42.895+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:42.894193160+11:00" level=info msg="ignoring event" container=954b4c1b8d3a7627af38dee0d8f533186a82be8da8094f50bee793a9631aea67 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2023-02-07T09:58:42.978+11:00 axis-b8a44f3be4f1 [ INFO    ] dockerdwrapperwithcompose[2869376]: time="2023-02-07T09:58:42.977542720+11:00" level=warning msg="Failed to delete conntrack state for 172.18.0.4: invalid argument"

[mnarodovitch]

camera:
AXIS M4308-PLE Panoramic Camera

os:
AXIS OS version 11.3.70

logs after restart

# journalctl | grep SD_DISK
Nov 23 15:14:52 axis-b8a44f552720 disks[1532]: Disk SD_DISK starting mount w/ id 1
Nov 23 15:14:58 axis-b8a44f552720 dockerdwrapperwithcompose[1790]: Cannot store data on the SD card, no storage exists at /var/spool/storage/SD_DISK
Nov 23 15:14:58 axis-b8a44f552720 dockerdwrapperwithcompose[1790]: Couldn't identify the file system of the SD card at /var/spool/storage/SD_DISK
Nov 23 15:15:03 axis-b8a44f552720 disks[1532]: Disk 'SD_DISK' mounted  on '/var/spool/storage/SD_DISK'
Nov 23 15:15:03 axis-b8a44f552720 disks[1532]: Disk available status changed for SD_DISK. Status = 'Available'
Nov 23 15:15:03 axis-b8a44f552720 indexer[929]: Attaching database 'SD_DISK' w/ version 3.0 (2023-03-16T15:52:53.716134Z)
Nov 23 15:15:03 axis-b8a44f552720 indexer[929]: /var/spool/storage/SD_DISK/index.db attached as SD_DISK

systemd

# systemctl cat sdkdockerdwrapperwithcompose.service 
# /etc/systemd/system/sdkdockerdwrapperwithcompose.service
[Unit]
Description=Docker Daemon with Compose
After=acap-pre.target 
Requires=acap-pre.target 
[Service]
Slice=extension-acap.slice
ExecStart=/usr/local/packages/dockerdwrapperwithcompose/dockerdwrapperwithcompose 
Restart=no
User=root
Group=root
WorkingDirectory=/usr/local/packages/dockerdwrapperwithcompose
[Install]
WantedBy=acap.target
[Unit]
BindsTo=containerd.service
After=network-online.target containerd.service var-spool-storage-SD_DISK.mount
Wants=network-online.target
[Service]
Environment=PATH=/usr/local/packages/dockerdwrapperwithcompose:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

# systemctl show sdkdockerdwrapperwithcompose.service | grep After
RemainAfterExit=no
After=basic.target network-online.target extension-acap.slice containerd.service multi-user.target systemd-journald.socket usr.mount -.mount var-spool-storage-SD_DISK.mount sysinit.target usr-local.mount

# systemctl show sdkdockerdwrapperwithcompose.service | grep Wants
Wants=network-online.target