Problem with random crashing

[nikp00]

With both version 3.0.1 and 3.0.2, I'm having trouble with random crashing. I did some debugging but was unable to reliably reproduce the behavior. The crashes usually happen on startup but in a few cases also during operation.

OpenDataCam is running on a Jetson Nano inside Docker.

If found that after the crash the web GUI at 8080 is unavailable, but the raw camera stream is available at 8090.

...
...
...
already started
Send request to connect to YOLO JSON Stream
Error: Request failed with status code 400
    at createError (/var/local/opendatacam/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/var/local/opendatacam/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/var/local/opendatacam/node_modules/axios/lib/adapters/http.js:236:11)
    at IncomingMessage.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  config: {
    url: 'http://104.26.9.160:443/counter/areas',
    method: 'get',
...
...
...

The full log is available in the attached file.

(Unrelated to this issue I also saw cases where the Darknet API responds with multiple frames in one go and the parser throws an error because the JSON is invalid. I don't know if this is a Darknet issue or OpenDataCam.)

odc_crash.log

[vsaw]

Are you trying to run ODC behind a reverse proxy or Cloudflare Tunnel?

That is unfortunately not easily done as ODC does not play nice with reverse proxies. I suspect a bug in urlHelper.js. I've tried fixing it like this

module.exports = {
  getURLData(req) {
    let protocol = 'http';
    if (req.headers['x-forwarded-proto'] === 'https') {
      protocol = 'https';
    }

    let ret;
    const parsedUrl = req.get('Host').split(':');
    if (parsedUrl.length > 1) {
      ret = {
        address: parsedUrl[0],
        port: parsedUrl[1],
        protocol,
      };
    } else {
      ret = {
        address: req.get('Host'),
        port: 80,
        protocol: req.protocol,
      };
    }
    return ret;
  },
};

Which worked in my setup but I haven't tested it yet, which is why it's not out there yet. If you are able to confirm that this code-snippet fixes your issue I'm happy to create an official fixed docker version for you.

[nikp00]

Thank you for the quick reply.

No, I'm not running ODC behind any proxy or Cloudlfare tunnel (I also don't understand form where did Axios get that server: 'cloudflare'). ODC is being run from the offcial docker-compose file for the Jetson Nano.

I will try the fix that you provided and report back.

[vsaw]

No, I'm not running ODC behind any proxy or Cloudlfare tunnel (I also don't understand form where did Axios get that server: 'cloudflare'). ODC is being run from the offcial docker-compose file for the Jetson Nano.

In this case I'm not sure the fix will work for you. From what I see somehow ODC changed the port to 443 (standard HTTPS port) which is why I assumed a reverse proxy or TLS endpoint of some sort.

Could you describe your setup one more time for me? Do you have any special networking setup (VPN, Proxy, etc?) running?

[nikp00]

ODC is running on a Jetson Nano using the provided docker-compose.yml. When deployed the Jetson is connected using LTE, although the same behavior was observed when connected with a normal LAN. That's it, no other special network configuration

[vsaw]

What you describe should just work. I've operated devices like this before. However, the Cloudfront and HTTPS traces in the logs make me skeptical that something in your setup is interfering with the networking.

I've updated the development container with the fixes I posted earlier. You could give it a try and let me know if this changes anything.