Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: skip merging large input sourcemaps #10890

Merged

Conversation

JLHwung
Copy link
Contributor

@JLHwung JLHwung commented Dec 18, 2019

Q                       A
Fixed Issues? Fixes #10875
Patch: Bug Fix? Yes
Tests Added + Pass? Yes
License MIT

This PR addresses #10875 where a large input source map is consumed and causing OOM error. It is not an ideal fix but still preserve basic support for practical size input source map.

Note that #10885 misidentified the memory bottleneck. But I preserve the change here since we need to detect the source map file size.

Ideally the mergeSourceMap in babel generator should operate in a streaming mode. But this does not fit in a patch release.

I don't consider it as a regression since in 7.6.4 the inputSourceMap is not working properly on external sourcemaps.

@artisb45 @midgethoen Though the URL does not change, here is a new fix. You can download it and overwrite node_modules/@babel/core/lib/transformation/normalize-file.js.

@JLHwung JLHwung added PR: Bug Fix 🐛 A type of pull request used for our changelog categories area: sourcemaps labels Dec 18, 2019
Copy link
Member

@nicolo-ribaudo nicolo-ribaudo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we already do something similar in @babel/generator: if the input is >500K (I think) we disable the formatting.

// when `lastComment` is non-null, EXTERNAL_SOURCEMAP_REGEX must have matches
const match: [string, string] = (EXTERNAL_SOURCEMAP_REGEX.exec(
lastComment,
): any);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this guaranteed to never return null?

Copy link
Contributor Author

@JLHwung JLHwung Dec 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because the capturing group

([^\s'"`]+)

has a + quantifier, if the whole regex passed, the capturing group must not be null.

@JLHwung
Copy link
Contributor Author

JLHwung commented Dec 18, 2019

@nicolo-ribaudo Correct. I have tested on pdfjs-dist/build/pdf.js.map (1.21 MB) and it does not bail. To be conservative I shrink the threshold to 1M. We can easily change it to an option if people really need to configure.

@nicolo-ribaudo nicolo-ribaudo merged commit e85c9b9 into babel:master Dec 19, 2019
@nicolo-ribaudo nicolo-ribaudo deleted the skip-merging-large-external-sourcemap branch December 19, 2019 00:16
@github-actions github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Mar 19, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area: sourcemaps outdated A closed issue/PR that is archived due to age. Recommended to make a new issue PR: Bug Fix 🐛 A type of pull request used for our changelog categories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Very high memory usage compiling mozilla/PDF.js
3 participants