Background updates eventually hang

[lwis]

iOS device model, version and app version

Model Name: iPhone 15 Pro
Software Version: 17.4
App version: 2024.2

Home Assistant Core Version
2024.3.0

Describe the bug
Eventually background sensor updates stop happening, running the "Update sensors" shortcut times out. Force closing the app brings everything back to life and the shortcut starts working again.

To Reproduce
Have iOS app pushing sensor updates in background for a ~period of time.

Expected behavior
App continues to function without repeatedly force closing.

[bgoncal]

When you open the app, does the sensors get updated correctly? I mean, when it is in foreground all works?
Do you have "local push" enabled?

[lwis]

It does not update when the app is in the foreground.

I do not have local push enabled, or an internal URL configured.

[bgoncal]

Did you observe a pattern for when does it stop reporting? In background I know for sure the sensor updates wont be live without local push enabled but foreground should be working properly.

[lwis]

Unfortunately there doesn't seem to be a pattern to it, I have no way to reproduce. Is there anything useful I can capture the next time it happens?

[bgoncal]

Your logs. You can export going in Debugging > export

[bgoncal]

Can you check the latest testflight beta? It may have fixed this

[lwis]

I'll update to the latest beta and continue to monitor! 🤞🏻

[lwis]

Still happening unfortunately, this is what I see when the shortcut fails;

[dmulcahey]

Home Assistant_2024-03-21_181706.txt
Home Assistant_2024-03-21_192239.txt
Home Assistant_2024-03-21_200921.txt
Home Assistant_2024-03-21_191428.txt

[dmulcahey]

POSIXErrorCode(rawValue: 28): No space left on device) Is thrown constantly by the app when the WS connection breaks

was something changed in the test flight version related to URLsessions or any kind of networking introspection libraries?

[dmulcahey]

@bgoncal let me know if there is anything you need to assist in debugging this.

[bgoncal]

https://stackoverflow.com/questions/67318867/error-domain-nsposixerrordomain-code-28-no-space-left-on-device-userinfo-kcf

This looks related, I'll investigate next week

[dmulcahey]

RC still has the issue. I’ll also note that when this happens and you open the app after it has been in the background it doesn’t connect to HA at all. It fails and HA logs the following:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:461
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: March 24, 2024 at 6:32:56 PM (6 occurrences)
Last logged: 7:21:09 AM

[281473247231552] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472302263104] from fe80::1460:ed53:6ed7:34ed (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472222859200] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281471861314368] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds
[281471753716160] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds

hard closing the app and reopening it resolves the issue. I have been running test flight versions since they started getting published again and this just started with this cycle of work: after CarPlay release version so maybe 2 or 3 test flight pushes back.

[dmulcahey]

I just thought of something else. The timing could possibly coincide with the release of iOS 17.4

[bgoncal]

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

[lwis]

Having a Shortcut automation which fires on power connect and disconnect to "Update Sensors" in the background is the way this manifests most often for me.

[dmulcahey]

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

Just leaving the app alone in the background for like 30 mins is all it takes for me

[bgoncal]

@dmulcahey Im checking your logs, when they happened were you somewhere http://192.168.1.78:8123 was accessible? (your home probably) or were you using external URL?

[dmulcahey]

I work from home and am generally here. I just opened the app from the background again. Last open was around the time of my last reply and it happened again. I’ll get new logs for you in a few. Can you give me somewhere to send an iCloud link so I can get you the full bundle from the app?

[bgoncal]

Are you in discord? You can ping me there @bgoncal2

[bgoncal]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

[dmulcahey]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Just saw this update and installed it. Will report back

[dmulcahey]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Initial testing looks good. Let’s see how it does overnight

[dmulcahey]

Everything is still working. What was the issue?

[bgoncal]

I'm still investigating but initially I believe the issue is that StarScream lib we use for websocket interaction creates one URLSession per request, which in a retry loop goes over iOS limits. I believe this wasn't happening in some updates ago because we were still supporting iOS 12 at that time and this limit was not applied.
I created a PR to starscream daltoniam/Starscream#1011

[bgoncal]

Heads up that this fix is not on our main branch yet, so if you see new builds coming in testflight, they may revert this fix

[bgoncal]

@lwis can you also check if this works for you?

[lwis]

Updated this morning, so maybe too early to tell, but I've not run into any issues yet.

[bgoncal]

Feel free to reopen this issue if needed, I'm closing this issue for now and planning the next release to include the fix

[lwis]

Bad news; it happened again this morning. I'm unable to run the shortcut directly at all. The app is fully functional. Closing the app allows the shortcut to run again.

I grabbed logs before closing the app.

2024.624

[lwis]

This is still happening, please let me know if there's any way I can help!

[bgoncal]

Im doing some clean up around shortcuts, widgets and intents in general, I'll let you know when you can test it again