Agent causes system to hang after starting up, Ubuntu 20.04.1 LTS

[robertfromont]

On Ubuntu 20.04.1 LTS system, starting the agent causes the system to freeze (no mouse or keyboard input accepted) after a delay (which varies, from several seconds to several minutes).
This happens whether or not the Arduino is plugged in.
Using the aduino package installed via apt works fine, just not the arduino-create-agent - version 1.1.

[matthijskooijman]

Hm, weird behavior, I wouldn't really know what an application could do to make your system freeze like that.

The arduino package from apt is an (rather old) version of the java/offline IDE, which is completely unrelated to the agent, so that's not really relevant comparison.

How do you start the agent? Manually on the commandline? Any output produced? Maybe something in journalctl output?

[robertfromont]

Thanks for the quick reply!

(I only mention it working with the apt package to rule out a hardware problem)

It was starting automatically at login when the problem appeared. I've since disabled the Start Application entry, and tried starting it manually, with the same result.

The installer output is:

INFO[0000] Version:1.1.251                              
INFO[0000] Hostname: roberts-razer                      
INFO[0000] Garbage collection is on using Standard mode, meaning we just let Golang determine when to garbage collect. 
INFO[0000] You specified a serial port regular expression filter: usb|acm|com 
Terminated
INFO[0000] Version:1.1.251                              
INFO[0000] Hostname: roberts-razer                      
INFO[0000] Garbage collection is on using Standard mode, meaning we just let Golang determine when to garbage collect. 
INFO[0000] You specified a serial port regular expression filter: usb|acm|com 
INFO[0000] Your serial ports:                           
INFO[0000] 	There are no serial ports to list.          
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /                         --> main.homeHandler (2 handlers)
[GIN-debug] GET    /certificate.crt          --> main.certHandler (2 handlers)
[GIN-debug] DELETE /certificate.crt          --> main.deleteCertHandler (2 handlers)
[GIN-debug] POST   /upload                   --> main.uploadHandler (2 handlers)
[GIN-debug] GET    /socket.io/               --> main.(*WsServer).ServeHTTP-fm (2 handlers)
[GIN-debug] POST   /socket.io/               --> main.(*WsServer).ServeHTTP-fm (2 handlers)
[GIN-debug] WS     /socket.io/               --> main.(*WsServer).ServeHTTP-fm (2 handlers)
[GIN-debug] WSS    /socket.io/               --> main.(*WsServer).ServeHTTP-fm (2 handlers)
[GIN-debug] GET    /info                     --> main.infoHandler (2 handlers)
[GIN-debug] POST   /killbrowser              --> main.killBrowserHandler (2 handlers)
[GIN-debug] POST   /pause                    --> main.pauseHandler (2 handlers)
[GIN-debug] POST   /update                   --> main.updateHandler (2 handlers)
[GIN-debug] GET    /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] POST   /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] PUT    /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] PATCH  /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] HEAD   /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] OPTIONS /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] DELETE /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] CONNECT /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] TRACE  /v2/*path                 --> github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.WrapH.func1 (2 handlers)
[GIN-debug] Listening and serving HTTP on 127.0.0.1:8991

(Arduino_Create_Bridge:24255): libappindicator-WARNING **: 12:18:53.170: Using '/tmp' paths in SNAP environment will lead to unreadable resources

(Arduino_Create_Bridge:24255): libappindicator-WARNING **: 12:18:53.170: Using '/tmp' paths in SNAP environment will lead to unreadable resources

The journalctl output for the few seconds before hanging is:

Sep 20 12:21:18 roberts-razer Fortitray.desktop[3431]: check_message_call_back
Sep 20 12:21:18 roberts-razer Fortitray.desktop[3431]: check_message_call_back
Sep 20 12:21:18 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-62 noise=9999 txrate=780000
Sep 20 12:21:23 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-59 noise=9999 txrate=780000
Sep 20 12:21:28 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-59 noise=9999 txrate=6000
Sep 20 12:21:31 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-61 noise=9999 txrate=780000
Sep 20 12:21:40 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-61 noise=9999 txrate=780000
Sep 20 12:21:45 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-62 noise=9999 txrate=780000
Sep 20 12:21:50 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-62 noise=9999 txrate=780000
Sep 20 12:22:01 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-59 noise=9999 txrate=6000
Sep 20 12:22:11 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-62 noise=9999 txrate=6000
Sep 20 12:22:12 roberts-razer gnome-shell[2825]: ../clutter/clutter/clutter-actor.c:10556: The clutter_actor_set_allocation() function can only be called from within the implementation of the ClutterAct>
Sep 20 12:22:13 roberts-razer wpa_supplicant[1187]: wlp72s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-61 noise=9999 txrate=780000
Sep 20 12:22:14 roberts-razer gnome-shell[2825]: JS ERROR: TypeError: cancellable.cancel is not a function
                                                 cancelRefreshPropertyOnProxy@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/util.js:81:25
                                                 destroy@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/appIndicator.js:256:14
                                                 _remove@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:182:25
                                                 _itemVanished@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:177:18
                                                 _registerItem/this._nameWatcher[id]<@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:102:24
Sep 20 12:22:15 roberts-razer gnome-shell[2825]: ../clutter/clutter/clutter-actor.c:10556: The clutter_actor_set_allocation() function can only be called from within the implementation of the ClutterAct>
Sep 20 12:22:18 roberts-razer gnome-shell[2825]: JS ERROR: TypeError: cancellable.cancel is not a function
                                                 cancelRefreshPropertyOnProxy@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/util.js:81:25
                                                 destroy@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/appIndicator.js:256:14
                                                 _remove@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:182:25
                                                 _itemVanished@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:177:18
                                                 _registerItem/this._nameWatcher[id]<@/usr/share/gnome-shell/extensions/ubuntu-appindicators@ubuntu.com/statusNotifierWatcher.js:102:24
-- Reboot --

(When the system froze, the clock was showing 12:22)

There is output from the program, but redirecting output to a file failed (presumably the freeze prevents it from flushing to the file?)

So here's a photo:

The config.ini file (which I haven't touched) contains this:

gc = std  # Type of garbage collection. std = Normal garbage collection allowing system to decide (this has been known to cause a stop the world in the middle of a CNC job which can cause lost responses from the CNC controller and thus stalled jobs. use max instead to solve.), off = let memory grow unbounded (you have to send in the gc command manually to garbage collect or you will run out of RAM eventually), max = Force garbage collection on each recv or send on a serial port (this minimizes stop the world events and thus lost serial responses, but increases CPU usage)
hostname = unknown-hostname  # Override the hostname we get from the OS
regex = usb|acm|com  # Regular expression to filter serial port list
v = true  # show debug logging
appName = CreateBridge
updateUrl = https://downloads.arduino.cc/
origins = http://local.arduino.cc:8000
#httpProxy = http://your.proxy:port # Proxy server for HTTP requests

[matthijskooijman]

Ah, so your system actually completely freezes and never recovers (i.e. you have to manually reset your system)? I originally thought you said it freezes for a while and then responds again, but reading again you said it freezes after (not for) some time after starting the agent.

Is there any pattern to when the freeze happens? Are you actually using (or even have it open in a browser somewhere) Arduino Create when it happens?

TBH, I'm not too sure how the create agent works nowadays, so I'm not sure if I can provide any further debugging hints from here, but maybe the arduino devs can provide some additional insight..

[robertfromont]

Yes, I need to manually reset the system.

There's no pattern I can discern. Sometimes I start up the agent and within a couple of seconds the system freezes. Other times, I've got enough time to open the debug console from the menu. Sometimes it's a couple of minutes before the system freezes.

I noticed the line in the output that says:

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

So I followed its suggestions and did

export GIN_MODE=release

But if I do that, it won't even start up:

robert@roberts-razer:~$ export GIN_MODE=release
robert@roberts-razer:~$ ArduinoCreateAgent-1.1/Arduino_Create_Bridge
INFO[0000] Version:1.1.251                              
INFO[0000] Hostname: roberts-razer                      
INFO[0000] Garbage collection is on using Standard mode, meaning we just let Golang determine when to garbage collect. 
INFO[0000] You specified a serial port regular expression filter: usb|acm|com 
INFO[0000] Your serial ports:                           
INFO[0000] 	There are no serial ports to list.          
panic: open templates/nofirefox.html: no such file or directory

goroutine 20 [running]:
html/template.Must(...)
	/home/jenkins/go-latest/src/html/template/template.go:372
github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin.(*Engine).LoadHTMLFiles(0xc0002ca000, 0xc0001b1020, 0x1, 0x1)
	/home/jenkins/workspace/Create_agent_master/src/github.com/arduino/arduino-create-agent/vendor/github.com/gin-gonic/gin/gin.go:193 +0x374
main.loop()
	/home/jenkins/workspace/Create_agent_master/src/github.com/arduino/arduino-create-agent/main.go:276 +0xeb3
created by main.main
	/home/jenkins/workspace/Create_agent_master/src/github.com/arduino/arduino-create-agent/main.go:107 +0x98
robert@roberts-razer:~$ 

Is there a super-verbose logging option or something?

[umbynos]

Yes, we have a crash report functionality but I don't think it can help (because the agent does not crash)
Could you please try with the latest version 1.2.0

[github-actions]

This issue has been marked as stale because it has been open 14 days with no activity. Remove stale label or comment, otherwise it will be closed in 7 days

[robertfromont]

Sorry for the delay.
I've just tried version 1.2.0, unfortunately with the same result (system freezes and have to do a hard reset)

[umbynos]

could you please try dmesg -Tw and then start the agent?

[robertfromont]

dmesg -Tw doesn't log anything after the agent starts.

This is the result of running

dmesg -Tw &
/home/robert/ArduinoCreateAgent/Arduino_Create_Agent

Everything was fine for maybe two minutes, then I clicked the agent's icon in the top right of the screen, and the system hung. (Then I took the photo above)