Forward compatibility for firmware versions

[KyleMaas]

I'm looking at deploying some nodes as routers/repeaters (depending on location) in hard-to-access remote areas with good coverage, but I'm wondering:

Are there any planned changes to the protocol planned for the next, say, 18 months which would require these nodes to have their firmware updated?

These are going to be a pain in the neck to update once deployed, and may not even be updateable during certain parts of the year due to seasonal weather. When I see things like these:

#2856

#1509

It makes me wonder if the protocol is going to drastically change for the "3.0" milestone that's being planned, and whether it's prudent to deploy nodes which might be useless if the protocol changes too much.

[KyleMaas]

Another one that looks like it might break compatibility, although I'm not sure if that applies to routers and repeaters or if it's just for the client modes:

#2819

[D0han]

Another, but relevant for this feature we are lacking is OTA updates. Updating software over the air, without the hike for each of our nodes is essential for long term network health and overall usability.

Its not easy feature, but even if it would require weeks of low-prio file transfer over meshtastic during low congestion hours and piecing it handful of bytes at a time, and maybe even some additional hardware (storage) on the receiving node, it would be very much worth it.

[roha-github]

In my opinion, an update over the air should be acceptable for routers and repeaters that are difficult to reach. On the other hand, I don't think blanket statements are good.

Let's say the update is approx. 2000 kilobytes in size and is sent to the device in messages of 200 bytes each on Long / Fast with 1% air utilization or 457 messages per day. It would take three weeks for about ten thousand messages.

A significantly reduced firmware with a smaller binary would be much better for routers and repeaters. The amount of data could also be significantly reduced with a delta.

[garthvh]

There is not going to be LoRa OTA updates, those numbers wreck any mesh does not matter how many napkins you use.

[D0han]

Not sure what you interpreted here as complaining, its not about that at all.
I want the best possible for the project and its users, and to do that we need discussions. Some may be even frustrating for various reasons, but still needed.

To get a feeling if this would be usable at all i calculated some estimates for presets we have.
Assumptions made:

1. Meshtastic Firmware 2.3.6.7a3570a Beta (latest beta as of now)
2. firmware-tlora-v2-1-1_6-tcxo-2.3.6.7a3570a.bin (devices i have right now) -> 2.1MB
3. standard radio presets and their Data-Rates
4. direct connection/no hops (its vastly easier to hike to direct link than placing hand on the node itself)
5. various link quality, retransmissions, metadata abstracted as overhead (+0%, +10%, +20%)
6. duty cycle (100%, 10%)

Channel setting	Alt Channel Name	Data-Rate (kbps)	Transfer time (hours)	Transfer time +10% overhead (hours)	Transfer time +20% overhead (hours)	Transfer time duty cycle 10% (hours)	Transfer time +10% overhead duty cycle 10% (hours)	Transfer time +20% overhead duty cycle 10% (hours)
Short Range / Fast	Short Fast	10,94	0,44	0,48	0,52	4,37	4,80	5,24
Short Range / Slow	Short Slow	6,25	0,76	0,84	0,92	7,65	8,41	9,18
Medium Range / Fast	Medium Fast	3,52	1,36	1,49	1,63	13,58	14,93	16,29
Medium Range / Slow	Medium Slow	1,95	2,45	2,70	2,94	24,51	26,96	29,41
Long Range / Fast	Long Fast	1,07	4,47	4,91	5,36	44,66	49,13	53,59
Long Range / Moderate	Long Moderate	0,34	14,05	15,46	16,87	140,55	154,60	168,66
Long Range / Slow	Long Slow	0,18	26,55	29,20	31,86	265,48	292,03	318,58
Very Long Range / Slow	Very Long Slow	0,09	53,10	58,41	63,72	530,96	584,06	637,16

So, assuming that for my case its duty cycle of 10% (thanks EU) and 20% data overhead, it would take me about 5h 15min to transfer new firmware using Short Fast. That saves me from accessing various masts, rooftops etc where its not convenient or would bother people that allowed me to place the node in the first place.

With that calculation (correct me if i missed something) it looks totally worth to setup separate update channel (next to admin channel) for my nodes that would not route traffic and be on Short Fast, or even Medium Fast if needed. And as originally said, it may even require additional storage hardware for such mechanism to work making the node pricier in the end if someone wants to use OTA updates.

We do not want to wake up to abandoned nodes (often in very advantageous locations) just because its hard to go to them with laptop and USB cable.

[roha-github]

The inner circle does not want certain features and therefore requests and discussions about them are closed. From my point of view, this is how it works in open source projects, where some decision-makers set the direction.

[KyleMaas]

I'd be incredibly happy with a version of the firmware stripped down to its smallest possible size for use with routers/repeaters, with OTA updates, and limited to only even being able to be installed on nRF52840 devices. It wouldn't be for everyone, but for the special case of being installed in significantly inaccessible locations.

[caveman99]

Some changes we are preparing for a new major version can be phased in, some require a breaking change. That's how embedded software works. There's no way around it. When this new 3.x version will happen is not decided yet, and the decision is not made by some inner circle behind closed doors either.

[KyleMaas]

As long as the breaking changes are far enough in the future, then it makes sense to place these nodes now. If you were to wildly prognosticate, how soon do you think breaking changes would be likely to happen?

[roha-github]

"different default modem preset (people have advocated for MediumFast) or another sync word to deliberately make nodes incompatible" ... #2856

Why 3.x can't implement backward compartible and use 2.x repeater / routers in coexistence?

[thebentern]

Let's say the update is approx. 2000 kilobytes in size and is sent to the device in messages of 200 bytes each on Long / Fast with 1% air utilization or 457 messages per day. It would take three weeks for about ten thousand messages.

You are massively over-simplifying this problem. You will have to compete with the existing LoRA traffic, add additional control messages to send / acknowledge chunked delivery of the binary. A special OTA firmware partition (which we currently have for BLE) would have be written martial all of this, and it would at best be extremely unreliable.

[caveman99]

@roha-github have you looked at our partition scheme? We don't have room for OTA partitions the way expressiv wants them to be. Even with a stripped down firmware there's no room for 2 symmetrical partitions. So your assumptions are wrong.

This is really tiring as this has been discussed in the past. Several times. With the same arguments. So unless someone can show me proof of concept that you can indeed do incremental upgrades on flash memory either for nrf52 or ESP32 i am not taking part in this discussion any more. It is simply not feasible to distribute firmware updates over Lora. End of discussion for me.

[roha-github]

@caveman99 I respect your opinion and experience in this project, but I still disagree with you.

An OTA update is a very useful feature, especially for routers and repeaters if they are stationed in an exposed and difficult to access location.

The firmware contains a lot of unnecessary features for routers and repeaters that are not absolutely necessary for forwarding and routing. A reduced firmware with OTA update would be the approach from my point of view, but that may be a separate project.

https://github.com/meshtastic/Meshtastic-repeater

If such discussions keep coming up, where are fundamental decisions made and documented?

[garthvh]

OTA over Lora is not happening, no matter how many times you repeat yourself. This is not point to point tastic, 5 years later this is the same dead end it has always been.

[KyleMaas]

I'm still wondering, though: is there a way to set the Bluetooth TX gain so I could use a Yagi/Uda antenna?

[roha-github]

@garthvh I accept this, that there will never a OTA update over Lora in the main firmware.

[KyleMaas]

Although it's a good discussion to have, we're getting a little off-topic with the OTA stuff. My original question is more about (1) the timeline for breaking changes, and (2) whether they would make repeater nodes completely useless. The general idea I'm getting from this discussion is that the answer to (2) is a definite yes, and the answer to (1) is a "no idea, but probably sometime". What are thoughts on how soon 3.0-style breaking changes would be likely to happen?

[caveman99]

1. There is no timeline set at the time of writing.
2. Breaking changes means you can keep your network on 2.x if you don't update the routers. Expect to be excluded from the public mqtt and don't expect maintenance from the official side on 2.x - We simply don't have the capacity to maintain 2 major firmware branches.

This from someone who has been through this process before, going from 1.2 to 2.0

[KyleMaas]

Excellent advice. Thank you.

Is it likely that routers updated to 3.x would still work with clients on 2.x?

[caveman99]

Very unlikely down to zero. New major number means we take measure this interoperability does NOT happen because it would do way more harm than good. The last change saw new radio zone definitions (this time will see a change to the EU_433 zone), a new default PSK and changes to all radio presets including a new default. Even if you set all values on a 2.x client so it can talk to 3.x routers the message structure will be completely different. Even the unencrypted routing header is not compatible.

[KyleMaas]

Good to know. Thank you!

[goatzillax]

If the devs say it isn't happening, it isn't happening.

"Guys who plan stuff" should perhaps plan for the ability to program the low power bits with something that is more versatile, i.e. an ESP32 with an SD card sitting on a regulator w/ enable which is connected to a GPIO and whatever programming lines are needed.

So when update day comes, you twiddle the GPIO to turn the ESP32 on and either hit it with a directional antenna or a drone to perform the update.

If the cost of, say, an ESP32 + SD card + regulator is less than the cost of physical access... that decision seems pretty obvious.

[roha-github]

If you are close to your node, the OTA update via Bluetooth is the best option.

https://github.com/meshtastic/firmware/blob/d19607ba98019df07ca334168fb57dd3e1afedf1/src/platform/esp32/BleOta.cpp

Everyone else can customize this and send the chunks to their node via Lora instead of BLE. Technically it's almost the same. However, I understand that the decision-makers at Meshtastic don't want to give the general public this option.

[thebentern]

Nobody has prevented this from being implemented... The development team itself isn't interested in chasing down a solution that has been deemed extremely problematic from a technical standpoint at best by multiple parties including the developer who designed the current partitioning strategy for ESP32 @caveman99. I don't know why you keep perceiving this as some sort of council of gatekeepers. We are all volunteers who work on the things we find interesting and things that align with the core needs of the project.

Everyone else can customize this and send the chunks to their node via Lora instead of BLE. Technically it's almost the same. However, I understand that the decision-makers at Meshtastic don't want to give the general public this option.

If you think think it's just that easy, submit a PR and we'll take a look at it.