Change topojson serialization format

[maurizi]

Overview

Upgrading to Node.js 16 in #602 caused a regression in data loading - the maximum size a of string was was reduced from ~1Gb to ~512Mb, which meant we could no longer load our two largest regions by using JSON.parse(buffer.toString())

This PR takes a multi-step approach to tackling this problem:

• I've switched to serialzing the TopoJSON using the v8 binary serialization format: c2a0dcb, d17738e & dc52da8
• Added a command to download the existing TopoJSON files (I used streaming JSON so we can load the now too large files w/o needing to downgrade Node temporarily) f8262c2
  • I've already applied this command on all the files used on staging as well as those referenced by load-dev-data, but I still need to update production, and any one-off regions will no longer work unless updated

After testing this branch on staging, I encountered a lot of resource contention and a large number of layers failed to load the first time, so I added 807dc9d & e8ea4f1 to make the TopologyService more robust.

Checklist

• Description of PR is in an appropriate section of CHANGELOG.md and grouped with similar changes, if possible

Demo

Updated healthcheck, showing which layers are loading / pending.

Notes

I was hopeful this would decrease our load times, but it hasn't done so to any meaningful degree. We did buy a lot of headroom in terms of TopoJSON file size - the max file size for v8.deserialize is ~4GB.

Testing Instructions

Try running ./scripts/manage serialize-topojson ... locally, and make sure you can run the server

I tested this against staging, and was able to get staging to load successfully.

Closes #1095

[KlaasH]

Looks good 👍
I deleted the topo.buf for the NJ config I have in my database, then recreated it and confirmed that it loaded and I could make a map.

Re performance: I found this comparison of Javascript encoding tools, and this PR that adds v8.serialize to the mix. It seems like the tools that are notably faster than v8.serialize require a predefined schema, which is probably more trouble than it would be worth for our purposes.

[KlaasH]

I wish asyncLoop could work as a throttled map rather than a throttled forEach, so we wouldn't have to collect the results as a side effect inside this mapped function. But my attempts to mentally revise it to work that way have not gone well, so this will do just fine.

[maurizi]

Yeah I tried / failed at this as well - good to know it wasn't just me 😅