Fix Metro source maps #1

shaunlebron · 2022-10-07T20:05:35Z

CLJS source maps are not being used by Expo, which makes runtime errors difficult to diagnose.

Currently, this PR is for discovering where the source maps are being lost— Expo, Metro, or Babel.

I currently believe this is a Metro issue, not a Babel issue:

How Metro / Expo / Babel fit together?	A deliberate runtime error showing lack of CLJS Source Maps in Expo Go:

My current understanding is that Expo uses Metro uses Babel, and that Expo’s babel.config.js file can be used to override the Babel options used ultimately by Metro. This is hard to verify.

According to Babel’s (config page](https://babeljs.io/docs/en/configuration#babelconfigjson), all Babel options are allowed in the config here, so I’m trying to add inputSourceMap: true.

I’m throwing in an error inside our main component to test the source map, but still getting back a JS stack trace (see screenshot above).

Seeking Help

The Expo discord channel and Metro contributors on GitHub have been silent so far. But a Babel maintainer has said this is not their issue, since babel-core works with inputSourceMap.

[babel-generator] Generated code doesn’t match with source with input source-maps babel/babel#5408 (comment)

shaunlebron · 2022-10-10T19:16:05Z

Check if babel.config.js is being ignored. I tried throwing an error inside the exported function to see if it disrupted the build, but it never did.

edit: yarn expo start --clear forces babel.config.js to be re-evaluated.

shaunlebron · 2022-10-10T22:51:50Z

Verified that when metro-react-native-babel-transformer calls @babel/core/transformFromAstSync, the internally resolved options object contains inputSourceMap: true.

Despite this, the source maps in Expo Go following the runtime error are still showing up in index.js.

I think I will see next if Metro produces a js and sourcemaps file somewhere so I can check whether or not the inputSourceMap option affects the produced files.

shaunlebron · 2022-10-11T17:32:36Z

Created a build.js script to produce a Metro bundle file to examine.
Verified that inputSourceMap: true has no effect on the final sourcemap file at build/expo-ios.map.

I will next dig deeper to see if Babel is producing correct source maps or not, and if it is I’ll check where Metro might be throwing them away.

shaunlebron · 2022-10-11T18:44:04Z

Verified that Metro is disregarding source maps whenever calling transformFromAstSync in babel-core:
metro-babel-transformer
https://github.com/facebook/metro/blob/v0.70.3/packages/metro-babel-transformer/src/index.js#L89
```
const {ast} = transformFromAstSync(sourceAst, src, babelConfig);
    // ^ `map` key disregarded
```
metro-react-native-babel-transformer
https://github.com/facebook/metro/blob/v0.70.3/packages/metro-react-native-babel-transformer/src/index.js#L223
```
return {ast: nullthrows(result.ast), functionMap}; // <-- `result.map` not returned
```
metro-transform-worker
https://github.com/facebook/metro/blob/v0.70.3/packages/metro-transform-worker/src/index.js#L332-L370

shaunlebron · 2022-10-11T19:43:53Z

I inspected the call of babel-core’s transformFromAstSync when running node build.js, and force-inserted some options {sourceMaps: true, inputSourceMap: true}, but the result did not return source maps. Here is the result:

// BABEL’S RESULT:
{
  metadata: {},
  options: {
    comments: false,
    compact: true,
    assumptions: {},
    inputSourceMap: true,   // <-- RELEVANT OPTION
    caller: { name: 'metro', bundler: 'metro', platform: 'ios' },
    ast: true,
    babelrc: false,
    code: false,
    highlightCode: true,
    filename: '/Users/shaun/code/matrixops/cljs-expo/expo/target/index.js',
    sourceType: 'module',
    sourceMaps: true,       // <-- RELEVANT OPTION
    targets: {},
    cloneInputAst: true,
    configFile: false,
    browserslistConfigFile: false,
    passPerPreset: false,
    envName: 'production',
    cwd: '/Users/shaun/code/matrixops/cljs-expo/expo',
    root: '/Users/shaun/code/matrixops/cljs-expo/expo',
    rootMode: 'root',
    plugins: [
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin], [Plugin],
      [Plugin], [Plugin], [Plugin]
    ],
    presets: [],
    parserOpts: {
      sourceType: 'module',
      sourceFileName: '/Users/shaun/code/matrixops/cljs-expo/expo/target/index.js',
      plugins: [Array]
    },
    generatorOpts: {
      filename: '/Users/shaun/code/matrixops/cljs-expo/expo/target/index.js',
      auxiliaryCommentBefore: undefined,
      auxiliaryCommentAfter: undefined,
      retainLines: undefined,
      comments: false,
      shouldPrintComment: undefined,
      compact: true,
      minified: undefined,
      sourceMaps: true,
      sourceRoot: undefined,
      sourceFileName: 'index.js'
    }
  },
  ast: {
    type: 'File',
    start: 0,
    end: 113946,
    loc: {
      start: [Object],
      end: [Object],
      filename: undefined,
      identifierName: undefined
    },
    errors: [],
    program: {
      type: 'Program',
      start: 0,
      end: 113946,
      loc: [Object],
      sourceType: 'script',
      interpreter: null,
      body: [Array],
      directives: [],
      leadingComments: undefined,
      innerComments: undefined,
      trailingComments: undefined
    },
    comments: [ [Object], [Object] ],
    leadingComments: undefined,
    innerComments: undefined,
    trailingComments: undefined
  },
  code: null,
  map: null,                // <-- SOURCE MAPS NOT BEING PRODUCED
  sourceType: 'script',
  externalDependencies: Set(0) {}
}

I will try to reduce this to a simple example to start testing babel-core directly to diagnose what might be going wrong.
One thing suspicious is that the babel.config.js options are not present in these options.
Maybe source paths are not able to be resolved? I need to look at how source maps actually work.

…ntly errors)

shaunlebron · 2022-10-12T18:59:08Z

The code: false babel option was eliding the options inputSourceMap: true, sourceMaps: true. Debugging with node inspect build-babel.js.

shaunlebron · 2022-10-13T17:12:18Z

The Trace-Mapping library is breaking on our source map because it is expecting keys at root that shadow-cljs places at .sections[0].map:

https://github.com/jridgewell/trace-mapping/blob/main/src/trace-mapping.ts#L163

    const { version, file, names, sourceRoot, sources, sourcesContent } = parsed;

Inspecting:

> Object.keys(parsed)
[ 'version', 'file', 'sections', 'sourceRoot' ]   // <-- keys missing

> Object.keys(parsed.sections[0].map)
[ 'version',
  'file',
  'lineCount',
  'mappings',
  'sources',
  'sourcesContent',
  'names' ]

In Summary

We have two problems so far:

it seems that Metro is not calling Babel-core in the right way to allow it to process inputSourceMaps.
Even if Metro were fixed, our source maps are not compatible with Babel’s source-map merging done by trace-mapping.

shaunlebron · 2022-10-13T22:25:39Z

I fixed build-babel.js by patching node_modules with ./patch.sh. In theory, this means Babel-core should work fine with inputSourceMaps, but this is hard to verify without testing this in Expo, which will require more work.

I shared a huge progress report on what I’ve tried on the Metro issue, hoping to inspire some grace from the maintainers.

shaunlebron · 2022-10-17T20:54:48Z

No feedback from the Metro team yet. I think the easiest way forward is:

Start with an external solution— like a custom node script that can merge our original clojurescript source map into Metro’s bundled source map. This seems obviously possible, so I’ll look at the same tool Babel uses for this, remapping.
Figure out where to put it— it most likely belongs in Metro’s bundler process, but might also work as an extra step inside Expo, or as a part of our own build process, whatever that looks like.

Here is what the source map produced by Metro looks like:

{
  "version": 3,
  "sources": [
    "__prelude__",
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/metro-runtime/src/polyfills/require.js",

    // *** OUR CLOJURESCRIPT BUILD ***
    "/Users/shaun/code/matrixops/cljs-expo/expo/target/index.js",

    // *** A million expo and react-native dependencies ***
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/expo/build/Expo.js",
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/@babel/runtime/helpers/interopRequireDefault.js",
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/expo/build/Expo.fx.js",
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/@babel/runtime/helpers/defineProperty.js",
    "/Users/shaun/code/matrixops/cljs-expo/expo/node_modules/expo/build/environment/validate.fx.js",
 
   // …  
}

And here’s my notes on the Source Map file format:

Source Map file format

_	_
site	https://sourcemaps.info/spec.html
doc	https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k
code	https://github.com/mozilla/source-map

A source map is either basic:

// parsed by BasicSourceMapConsumer
{
	"version" : 3,
	"file": "out.js",
	"sourceRoot": "",
	"sources": ["foo.js", "bar.js"],
	"sourcesContent": [null, null],
	"names": ["src", "maps", "are", "fun"],
	"mappings": "A,AAAB;;ABCDE;"
}

or indexed:

// parsed by IndexedSourceMapConsumer
{
  "version" : 3,
  "file": "app.js",
  "sections": [
    {
      "offset": {"line":0, "column":0},
      "url": "url_for_part1.map"
    },
    {
      "offset": {"line":100, "column":10},
      "map": { // an embedded complete source map object
        "version" : 3,
        "file": "section.js",
        "sources": ["foo.js", "bar.js"],
        "names": ["src", "maps", "are", "fun"],
        "mappings": "AAAA,E;;ABCDE;"
      }
    }
  ]
}

Where the mappings field is an encoded string in the format below:

<mappings> ::= <group> ";" <group> ";" …
                       ---
                     out:nextline

<group>    ::= <segment> "," <segment> "," …

<segment>  ::= <A> | <A><B><C><D> | <A><B><C><D><E>  (fields)
               ---      ---------               ---
              out:col   in:file,line,col        in:name

<A - E>    ::= <Base 64 VLQ>

Specifically, the characters in a mapping string mean the following:

; = increment line of the output code and reset output column to zero
, = start new segment (of fields)
[A-Za-z0-9+/] part of a field character representing any of the following values (delta from previous, not absolute):
A. Output Column
B. Input File
C. Input Line
D. Input Column
E. Input Name

aiba · 2022-10-18T03:08:34Z

This is awesome. Sounds like a good plan to me. Once we get it working on the example project, I think we'll want to package it up in a way that any clojurescript project can use it, and then I'll test it out in my big expo project.

shaunlebron · 2022-10-19T00:11:55Z

I wrote a remap.js script to fix metro source maps against a bb shadow-release build, but I’m having trouble testing it manually— locations in the my_app/main.cljs aren’t mapping to spots in the generated code that I would recognize as compiled, but maybe I’m missing something there.

I tried to test source maps with bb shadow-dev but they don’t seem to produce a target/index.js.map file, despite adding the compiler option to do so. I’ll look into it tomorrow, and will try to see if I can get expo to load our fixed sourcemap.

shaunlebron · 2022-10-20T13:33:28Z

bb shadow-dev bakes the source maps into a base64 data url in a //# sourceMappingUrl inside a dynamically eval’ed string for each source file. This is the same technique used by webpack when its eval-source-map option is enabled for fast rebuilds. I guess that means that when one file rebuilds, it executes a single eval for that file and is able to update the source maps. Maybe the react native runtime or expo or metro doesn’t support dynamically eval’d inline source maps. Will investigate here:

https://docs.expo.dev/get-started/errors/#redbox-errors-and-stack-traces

edit: thomas heller said that the :target :react-native shadow option inlines the source files in an eval’d string in order to prevent Metro’s minifier from processing it, because it is very slow, and in order to make the source maps work in chrome (for metro web?). He suggests trying :target :npm-module to see if this is still the case, but I got some errors related to not being able to resolve paths, which I’ll look into.

shaunlebron · 2022-10-20T16:15:33Z

Source map workarounds I found on #cljsrn on clojurians slack:

For mapping Sentry error reports: the https://github.com/riverford/sourcemap-cljs
Using cljs option :pseudo-names true https://stackoverflow.com/questions/60543802/get-sourcemapped-source-place-in-browser
Circumventing transformation in Metro: http://austinbirch.co.uk/clojurescript-react-native-bundling-release.html
- https://github.com/pesterhazy/re-natal-husk

add inputSourceMap:true to babel config

b13f81b

add build file and more investigation notes

e95f90d

shaunlebron added 2 commits October 11, 2022 16:57

move build.js to build-expo.js in prep for babel build test

c70cc91

add build-babel.js to show missing sourcemaps

c0d744e

shaunlebron force-pushed the source-map branch from 2b6ee39 to c0d744e Compare October 11, 2022 22:16

enable code option in build-babel to prevent sourcemap eliding (curre…

9fe897a

…ntly errors)

shaunlebron changed the title ~~Add source maps~~ Fix Metro source maps Oct 13, 2022

shaunlebron added 2 commits October 13, 2022 14:10

write a patch script to make build-babel run without error

bd916e2

output babel build to target/index2.js(.map)

27fa224

fix build-expo -> build-metro

4ac0b1b

shaunlebron added 3 commits October 18, 2022 17:09

add remap script, needs debugging

45b3d47

fix remap and replace patch.sh with builtin AnyMap workaround (aiba#1)

31fbb71

try to test remap.js

659505a

shaunlebron closed this Jun 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Metro source maps #1

Fix Metro source maps #1

shaunlebron commented Oct 7, 2022 •

edited

shaunlebron commented Oct 10, 2022 •

edited

shaunlebron commented Oct 10, 2022 •

edited

shaunlebron commented Oct 11, 2022

shaunlebron commented Oct 11, 2022 •

edited

shaunlebron commented Oct 11, 2022

shaunlebron commented Oct 12, 2022 •

edited

shaunlebron commented Oct 13, 2022 •

edited

shaunlebron commented Oct 13, 2022

shaunlebron commented Oct 17, 2022 •

edited

aiba commented Oct 18, 2022

shaunlebron commented Oct 19, 2022 •

edited

shaunlebron commented Oct 20, 2022 •

edited

shaunlebron commented Oct 20, 2022

Fix Metro source maps #1

Fix Metro source maps #1

Conversation

shaunlebron commented Oct 7, 2022 • edited

Seeking Help

shaunlebron commented Oct 10, 2022 • edited

shaunlebron commented Oct 10, 2022 • edited

shaunlebron commented Oct 11, 2022

shaunlebron commented Oct 11, 2022 • edited

shaunlebron commented Oct 11, 2022

shaunlebron commented Oct 12, 2022 • edited

shaunlebron commented Oct 13, 2022 • edited

In Summary

shaunlebron commented Oct 13, 2022

shaunlebron commented Oct 17, 2022 • edited

Source Map file format

aiba commented Oct 18, 2022

shaunlebron commented Oct 19, 2022 • edited

shaunlebron commented Oct 20, 2022 • edited

shaunlebron commented Oct 20, 2022

shaunlebron commented Oct 7, 2022 •

edited

shaunlebron commented Oct 10, 2022 •

edited

shaunlebron commented Oct 10, 2022 •

edited

shaunlebron commented Oct 11, 2022 •

edited

shaunlebron commented Oct 12, 2022 •

edited

shaunlebron commented Oct 13, 2022 •

edited

shaunlebron commented Oct 17, 2022 •

edited

shaunlebron commented Oct 19, 2022 •

edited

shaunlebron commented Oct 20, 2022 •

edited