Develop separate map style & data server

[gmaclennan]

Describe the idea
Serve map tiles & styles with an encapsulated solution that can exist as an independent module. Should rasterize vector tiles for iD editor and serve static map images.

Why?
Currently our offline map server is a mix of different pieces: we server static raster tiles for ID editor, and separately serve style.json for mapbox-gl with accompanying vector tiles, which we store as .asar compressed archives. We separately serve static map images with a custom solution that renders them in an electron renderer process (using mapbox-gl-js with access to the DOM).

Goals
An integrated solution for managing map data:

1. Proxy online map resources and cache for offline.
2. Download map resources for an area for offline use.
3. Rasterize vector tiles for use in iD Editor - fix Compatibility of map tiles between Territory view (iD Editor) and Observation view (Mapbox GL JS) #314
4. Import map tiles from mbtiles & other tile formats
5. Import "map style packages" including resources for mapbox-gl maps.
6. Serve raster tiles with style.json for mapbox-gl
7. Serve static map images from vector tiles

Requirements

1. User can import map tiles from:
   a. .asar files
   b. z/x/y folders of tiles
   c. .zip files containing style.json, sprites, font SDFs and z/x/y tiles
   d. .mbtiles files - fix Reading offline maps from MBTiles #103
2. User can import an offline mapbox map from a zipped folder containing style.json, font SDFs, sprites and .asar tiles (the format currently used by Mapeo)
3. User can add maps via a style URL (including support for mapbox:// urls)
4. User can see online version of map beyond downloaded boundaries if internet is available (link offline tiles to online equivalents)
5. User can continue to see offline maps installed in previous version of Mapeo (Migrate legacy offline maps / tiles, or directly serve legacy offline maps if on mobile without enough memory to migrate, which requires copying map resources)
6. User can request a download of map resources and tiles by bounding box and zoom range, and see progress of download job (for REST API implementation see http://restalk-patterns.org/long-running-operation-polling.html)
7. User does not use valuable bandwidth re-downloading tiles which are already downloaded and unchanged (use stale-while-revalidate)
8. "Light" version that does not include native modules (no rasterizing vector tiles) for use on mobile.
9. User sees text on map in fallback font if original font is not available offline
10. User can manage available maps via UX (maybe this is better in Mapeo, not here?)
11. Cache requests from mapbox-gl-native using something like https://github.com/sindresorhus/got#cache-adapters so that static maps API does not re-request lots of data.
12. Pool requests for rendered raster tiles and static maps to manage performance
13. User has single interface for defining what map styles are available in Mapeo, both online and offline.

Stretch goals

1. Use hyperdrive for map tile storage (quad-keys mapped to folders would allow sparse downloads)
2. Use hyperdrive for all map resource storage
3. Generate font SDFs from font files (e.g. use .ttf files rather than rendered SDFs)
4. Show fallback offline map when outside bounds or zoom of downloaded map
5. Sync offline map data between devices
6. Support streaming downloads of large static maps using https://github.com/digidem/mosaic-image-stream (could also be done client-side) or https://github.com/mapbox/abaculus
7. Serve Mapeo territory data as vector tile source (see mapeo-print-preview)
8. Serve Mapeo observations as a vector tile source
9. Allow GeoJSON overlays (for simple non-global datasets like land title boundaries or protected areas, it is more efficient to store GeoJSON and generate tiles at runtime)
10. Integrate with the MapTiler QGIS Plugin

Prior Art

• maptiler/tileserver-gl does a lot of this, but the way the code is written would require a lot of modifications to implement all requirements. Would be worth borrowing some of the code but maintaining a separate module.
• OSM Scout Server
• naturalatlas/tilestrata

Gotchas
mapbox-gl-native does not support Node >=12 but Natural Atlas is maintaining a forked version

Questions
Should we support directly reading from existing files e.g. mbtiles vs. importing a copy? Importing a copy gives us more control (e.g. what would happen if the user moved the file we were reading tiles from?) but requires disk space.

Implementation suggestions
Use fastify for efficient http serving and built-in validation and error handling.