Docker container needs a restart when camera signal was lost for a short time #461

AlexHolly · 2021-10-01T21:27:21Z

Version https://github.com/opendatacam/opendatacam/tree/db78e881097619616a5cb1e94b68abc3178c6798

When I unplug the camera for a short time (happens for both IP and USB) I have to restart the docker container otherwise the signal is never found again.

Anyone else has this issue or knows a solution?

vsaw · 2021-10-01T22:00:52Z

It may be related to #368

Could you provide log data?

AlexHolly · 2021-10-01T22:19:01Z

I also noticed the behavior described in #368 it also happens when I plug it in right after starting the container. I will try to provide a log tomorrow.

AlexHolly · 2021-10-02T11:30:37Z

Steps:

Connect IP Camera
Start Docker Container
Disconnect IP Camera
Connect IP Camera

Log:

// Recognizing that stream is not available anymore is also not happening and I am not able to "Start Recording"

Start recording
Start recording
Start recording
Start recording
Start recording
Start recording
Start recording
Start recording
Start recording
Start recording
Start recording

// Here I connected the camera again

[NULL @ 0x564e1eac4500] missing picture in access unit

Stream closed.
input video stream closed. 
Close socket: out = 0, in = 0 
MJPG_sender: close clinet: 0 
Close socket: out = 0, in = 0 
MJPG_sender: close clinet: 0 
Close socket: out = 0, in = 0 
MJPG_sender: close clinet: 0 
Close socket: out = 0, in = 0 
MJPG_sender: close clinet: 0 
Close socket: out = 0, in = 0 
MJPG_sender: close acceptor: 0 

Close socket: out = 0, in = 0 
JSON_sender: close clinet: 0 
Close socket: out = 0, in = 0 
JSON_sender: close acceptor: 0 

==== HTTP Stream closed by darknet, reset UI ====
==== If you are running on a file, it is restarting  because you reached the end ====
==== If you are running on a camera, it might have crashed for some reason and we are trying to restart ====
Restart YOLO

// Many more CUDA lines

...


[yolo] params: iou loss: ciou (4), iou_norm: 0.07, cls_norm: 1.00, scale_x_y: 1.05
Total BFLOPS 60.137 
avg_outputs = 500162 
 Allocate additional workspace_size = 26.22 MB 
Demo
net.optimized_memory = 0 
mini_batch = 1, batch = 8, time_steps = 1, train = 0 
nms_kind: greedynms (1), beta = 0.600000 
nms_kind: greedynms (1), beta = 0.600000 
nms_kind: greedynms (1), beta = 0.600000 
Loading weights from yolov4.weights...Done! Loaded 162 layers from weights-file 
[tcp @ 0x55ca32fafa00] Connection to tcp://192.168.1.1:554?timeout=0 failed: No route to host

 seen 64, trained: 32032 K-images (500 Kilo-batches_64) 
video file: rtsp://secret:secret@192.168.1.1
[ WARN:0] global /var/local/git/opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module source reported: Could not open resource for reading and writing.
[ WARN:0] global /var/local/git/opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /var/local/git/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

// Manually open http://localhost:8080/ in browser

already started
Something went wrong: connect ECONNREFUSED 172.19.0.4:8070
Too much retries, YOLO took more than 3 min to start, likely an error
-1
Something went wrong: connect ECONNREFUSED 172.19.0.4:8070
Too much retries, YOLO took more than 3 min to start, likely an error
-1

// Manually open http://localhost:8080/ in browser

already started
Something went wrong: connect ECONNREFUSED 172.19.0.4:8070
Too much retries, YOLO took more than 3 min to start, likely an error
-1

vsaw · 2021-10-07T18:33:31Z

Thanks for the logs. I can't tell if Darknet is running or not. Would you mind trying one more thing: After you reconnected the camera and you have the Darknet logs on the console, can you try accessing ports 8070 or 8090 to access Darknet directly?

haviduck · 2021-10-09T22:57:23Z

maybe due to 8080 already being yoinked at time of error and mjpeg being ad hoc?

AlexHolly · 2021-10-10T07:12:28Z

It looks like both connections are not available anymore. On 8090 I was getting this error message for a second an then it was not reachable anymore:

SyntaxError: JSON.parse: end of data when ',' or ']' was expected at line 481 column 2 of the JSON data

vsaw · 2021-10-10T09:09:33Z

Thank you @AlexHolly. That would indicate that Darknet is not starting up any more and would be a different error then #368. Last question, which Version of ODC are you running?

AlexHolly · 2021-10-10T09:19:22Z

It should be this one 7c79544

AlexHolly · 2021-10-20T10:57:45Z

I tested it again. Maybe I missed this message previously. When I disconnect the IP Camera for ~2 seconds I get this message.

==== HTTP Stream closed by darknet, reset UI ====
==== If you are running on a file, it is restarting  because you reached the end 
======== If you are running on a camera, it might have crashed for some reason and we are trying to restart ====
problem with request:  Error: read ECONNRESET   
at TCP.onStreamRead (internal/stream_base_commons.js:209:20) {  errno: 'ECONNRESET',  code: 'ECONNRESET',  syscall: 'read'}

Any idea where I should look to fix this?

haviduck · 2021-10-20T16:39:20Z

that looks like what i talked about. connreset for the 1-1 mjpeg conn and then it will die.

vsaw · 2021-10-21T16:26:59Z

@AlexHolly When you disconnect the camera, the Darknet process stopps (which is expected) and then ODC complains about that the Video stream stopped (which is also expected).

What should happen then is that ODC tries to start the Darknet process again and once Darknet is running it should connect to Darknet. Obviously this is not happening.

What I don't know is if this is because Darknet is not running, OR because ODC fails to re-connect to Darknet.

What you can do to help me debug this is:

Disconnect the camera
Wait untill you see the error messages you showed above
reconnect the USB-Camera
Wait a few seconds
try to open http://<IP of your Jetson>:8090 in your browser and tell me what you see

If you see something in your browser that means that Darknet restarted successfully and ODC is not connecting to it. If you don't see anything it means that Darknet did not restart. Here we'd need additional logs from you to understand why it did not restart.

Let me know if this works otherwise feel free to ask again :-)

rantgithub · 2021-10-21T16:37:19Z

Actually

We have encounter this problem on the past, but this is a darknet issue.

If you do manually a test, and stop/interrupt the camera video (regardless of what is the feed) and reconnect the feed darknet remains disconnected

therefore, ODC will remains without input from darknet and nothing works.

A simple thing is to restart darknet and all works again.

Having a logic to check the feed and restart darknet when this happens is a workaround for this problem.

AlexHolly · 2021-10-21T16:43:21Z

@vsaw I tested exactly this and you can see the result here.

It looks like both connections are not available anymore. On 8090 I was getting this error message for a second an then it was not reachable anymore:

SyntaxError: JSON.parse: end of data when ',' or ']' was expected at line 481 column 2 of the JSON data

@rantgithub Can I restart darknet with a 'exec' command from ODC?

vsaw · 2021-10-21T18:30:03Z

@AlexHolly Sorry I mussed have missed this then. This indicates that the Darknet process is not restarting. I can't really tell why without additional logs. Do you have any logs that show Darknet's or gstreamer output?

AlexHolly · 2021-10-24T15:47:41Z

@vsaw It looks like after connecting the camera again, darknet (8070) is reachable for a few seconds but mjpeg (8090) is not.

As a workaround I added a script to restart the node.js process in case of such an error. mjpeg then works but there are still some timing issues with /start and /recording/start

vsaw · 2021-10-24T18:21:48Z

@AlexHolly I currently don't have any HW so I can not investigate the issue any further. I can however offer you another workaround:

Darknet inside docker can be configured to accept UDP Streams. You can then create a quick shell script to stream the USB-Camera via UDP to Darknet inside docker. This allows you to restart the UDP Stream independently of Darknet and Docker. See also https://gstreamer.freedesktop.org/documentation/udp/udpsink.html?gi-language=c#udpsink and https://gstreamer.freedesktop.org/documentation/udp/udpsrc.html?gi-language=c#udpsrc

AlexHolly · 2021-11-03T12:57:47Z

@vsaw How can I configure Darknet Docker to accept UDP. I can only see this PR but it was never merged AlexeyAB/darknet#1976

vsaw · 2021-11-04T19:16:38Z

As Darknet can take images from any Gstreamer pipeline we don't need to hack Darknet. Instead we will configure the GST pipeline.

The process has three steps.

Step 1 Preparing ODC

I usually use something like this in my ODC configs

{
  "VIDEO_INPUT": "jpeg_remote_cam",
  "VIDEO_INPUTS_PARAMS": {
    "jpeg_remote_cam": "udpsrc port=7001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! videoconvert ! videoscale ! appsink"
  },
  ...
}

Step 2 - Configuring Port Forwarding

Now when you start the Docker container make sure that you forward UDP Port 7001 to ODC. You can skip this step if you don't use Docker.

See the Docker Documentation on port Forwarding, but it should work a little something like this:

docker run -p 127.0.0.1:7001:7001/udp opendatacam/opendatacam:v3.0.2-nano

Step 3 - Stream video via UDP

Now you should have ODC running in the background waiting for a UDP stream. You can stream your USB-Webcam to ODC when you run the following command in a different terminal

gst-launch-1.0 \
    v4l2src device=/dev/video0 ! video/x-raw, framerate=30/1, width=640, height=360 ! \
    nvjpegenc ! \
    rtpjpegpay ! \
    queue leaky=downstream max-size-buffers=1 ! \
    udpsink host=127.0.0.1 port=7001;

It may be the commands contain some typos or other errors but in general this is I've been using it. The great advantage is you can stop and start the command from Step 3 any time you want. From the point of ODC this is like having a frozen picture, but it does not crash ODC 😉

AlexHolly · 2021-11-30T12:09:00Z

@vsaw Do you know how to make it run on a Desktop? The plugin nvjpegenc is missing?

https://forums.developer.nvidia.com/t/nvjpegenc-on-dgpu-platforms/84126

> gst-inspect-1.0 --version
gst-inspect-1.0 version 1.16.2
GStreamer 1.16.2
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0

> gst-inspect-1.0 nvjpegdec
No such element or plugin 'nvjpegdec'

vsaw · 2021-11-30T13:38:52Z

You can run it without the nv prefix. Will probably be slower frame rate but should work.

AlexHolly · 2021-11-30T18:20:37Z

@vsaw It works like you described, but when I unplug the camera it will have a different video source (/dev/video0 or video1) and I have to rerun gst-launch-1.0. It is also not returning any error/signal to restart it automatically :(

gst-launch-1.0 \
    v4l2src device=/dev/video0 ! video/x-raw, framerate=30/1, width=640, height=480 ! \
    jpegenc ! \
    rtpjpegpay ! \
    queue leaky=downstream max-size-buffers=1 ! \
    udpsink host=127.0.0.1 port=7001;

My main use case is an Ip-Cam. Here I get at least a return value after a few seconds I will try tomorrow to automate the whole process and check if it is reliable:

gst-launch-1.0 \
    rtspsrc location=rtsp://admin:admin@192.168.0.100 ! \
    decodebin ! \
    videoconvert ! \
    jpegenc ! \
    rtpjpegpay ! \
    queue max-size-buffers=1 ! \
    udpsink host=127.0.0.1 port=7001;

Config:

udpsrc port=7001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! videoconvert ! videoscale ! appsink

AlexHolly · 2021-12-01T11:35:39Z

@vsaw I created a systemd service for the gst-launch-1.0 looks good so far. One small issue. Now after "two minutes" of camera disconnect the ODC live preview shows a grey background and is not able to repair without a refresh (F5). It shows the detected areas but no image. It works when the camera is back before the "two minutes". Is there a way to change this?

Error:

Close socket: out = -1, in = -1

vsaw · 2021-12-01T12:49:00Z

This is a bug in MJPEG Proxy. It is just "cosmetic" the counting logic is not affected.

We've discussed fixing it in #492
Feel free to give it a try if you like!

AlexHolly · 2021-12-01T13:15:49Z

Ok, I followed the discussion on #492. If it is just cleaning up by adding 99999 retries without the config. I could give it a try this weekend.

Thank you for your help, feel free to close this issue.

vsaw · 2021-12-01T13:55:09Z

If I remember correctly that's not enough. We discussed listening to the close event in the MJPEG Proxy. However I don't know what happens in your case with the UDP Stream config.

vsaw added to reproduce bug labels Nov 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker container needs a restart when camera signal was lost for a short time #461

Docker container needs a restart when camera signal was lost for a short time #461

AlexHolly commented Oct 1, 2021

vsaw commented Oct 1, 2021

AlexHolly commented Oct 1, 2021

AlexHolly commented Oct 2, 2021

vsaw commented Oct 7, 2021

haviduck commented Oct 9, 2021

AlexHolly commented Oct 10, 2021

vsaw commented Oct 10, 2021

AlexHolly commented Oct 10, 2021

AlexHolly commented Oct 20, 2021

haviduck commented Oct 20, 2021

vsaw commented Oct 21, 2021

rantgithub commented Oct 21, 2021

AlexHolly commented Oct 21, 2021

vsaw commented Oct 21, 2021

AlexHolly commented Oct 24, 2021

vsaw commented Oct 24, 2021

AlexHolly commented Nov 3, 2021

vsaw commented Nov 4, 2021

AlexHolly commented Nov 30, 2021

vsaw commented Nov 30, 2021

AlexHolly commented Nov 30, 2021

AlexHolly commented Dec 1, 2021

vsaw commented Dec 1, 2021

AlexHolly commented Dec 1, 2021

vsaw commented Dec 1, 2021

Docker container needs a restart when camera signal was lost for a short time #461

Docker container needs a restart when camera signal was lost for a short time #461

Comments

AlexHolly commented Oct 1, 2021

vsaw commented Oct 1, 2021

AlexHolly commented Oct 1, 2021

AlexHolly commented Oct 2, 2021

vsaw commented Oct 7, 2021

haviduck commented Oct 9, 2021

AlexHolly commented Oct 10, 2021

vsaw commented Oct 10, 2021

AlexHolly commented Oct 10, 2021

AlexHolly commented Oct 20, 2021

haviduck commented Oct 20, 2021

vsaw commented Oct 21, 2021

rantgithub commented Oct 21, 2021

AlexHolly commented Oct 21, 2021

vsaw commented Oct 21, 2021

AlexHolly commented Oct 24, 2021

vsaw commented Oct 24, 2021

AlexHolly commented Nov 3, 2021

vsaw commented Nov 4, 2021

Step 1 Preparing ODC

Step 2 - Configuring Port Forwarding

Step 3 - Stream video via UDP

AlexHolly commented Nov 30, 2021

vsaw commented Nov 30, 2021

AlexHolly commented Nov 30, 2021

AlexHolly commented Dec 1, 2021

vsaw commented Dec 1, 2021

AlexHolly commented Dec 1, 2021

vsaw commented Dec 1, 2021