Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: bypass DOM storage quota #15596

Merged
merged 4 commits into from Nov 12, 2018
Merged

fix: bypass DOM storage quota #15596

merged 4 commits into from Nov 12, 2018

Conversation

jacobq
Copy link
Contributor

@jacobq jacobq commented Nov 6, 2018

Description of Change

This updates the dom_storage_map patch to include bypassing the 2 additional locations of quota checking. It also adds a test to ensure that at least 50MiB of data can be stored in localStorage and sessionStorage (I'm surprised that I didn't find something like that already in place. Perhaps that's how the regression sneaked in.) Since the usual limit is about 10MiB the test will fail if the patch has not been applied or is not working.
Fixes #8337.

cc @MarshallOfSound you mentioned you might be willing to help land this. (Thanks in advance!)
cc @zcbenz It looks like you wrote the original patch. (Thank you! That helped me understand how electron's build system applies chromium patches)
cc @asinbow You pointed out the additional checks that needed to be disabled. (Thank you!)

Checklist

  • PR description included and stakeholders cc'd
  • npm test passes (CI is green)
  • tests are changed or added
  • relevant documentation is changed or added (see note below)
  • PR title follows semantic commit guidelines (once WIP prefix is removed)

Note about documentation:
Since this is fixing a regression (doing what the previous patch used to do), I don't think there's any need to change documentation. However, it might make sense to record this behavior in the documentation somewhere (I couldn't find it). Perhaps in the FAQ where localStorage is mentioned.

Note about running tests locally:
It seems that I can't run the test suite locally, though I can build a release and manually test in the console. Here's what I get on my (Debian 9.5) machine when I try to build the debug version (needed for running tests, right?) It seems like my system's libc version does not match the required one.

$ ninja -C out/Debug electron
ninja: Entering directory `out/Debug'
[7817/18095] ACTION //components/url_formatter/top_domains:generate_top_domains_trie(//build/toolchain/linux:clang_x64)
FAILED: gen/components/url_formatter/top_domains/alexa_domains-trie-inc.cc 
python ../../build/gn_run_binary.py top_domain_generator /home/jquant/tmp/electron-gn/src/components/url_formatter/top_domains/alexa_domains.skeletons /home/jquant/tmp/electron-gn/src/components/url_formatter/top_domains/top_domains_trie.template gen/components/url_formatter/top_domains/alexa_domains-trie-inc.cc
./top_domain_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /home/jquant/tmp/electron-gn/src/out/Debug/./libbase.so)
top_domain_generator failed with exit code 1
[7818/18095] ACTION //net/http:generate_transport_security_state(//build/toolchain/linux:clang_x64)
FAILED: gen/net/http/transport_security_state_static.h 
python ../../build/gn_run_binary.py transport_security_state_generator ../../net/http/transport_security_state_static.json ../../net/http/transport_security_state_static.pins ../../net/http/transport_security_state_static.template gen/net/http/transport_security_state_static.h
./transport_security_state_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /home/jquant/tmp/electron-gn/src/out/Debug/./libbase.so)
transport_security_state_generator failed with exit code 1
[7826/18095] CXX obj/third_party/ots/ots/ots.o
ninja: build stopped: subcommand failed.

Release Notes

notes: Fixes regression regarding localStorage quota not being bypassed.

@jacobq jacobq requested a review from a team November 6, 2018 18:06
@welcome
Copy link

welcome bot commented Nov 6, 2018

💖 Thanks for opening this pull request! 💖

We use semantic commit messages to streamline the release process. Before your pull request can be merged, you should update your pull request title to start with a semantic prefix.

Examples of commit messages with semantic prefixes:

  • fix: don't overwrite prevent_default if default wasn't prevented
  • feat: add app.isPackaged() method
  • docs: app.isDefaultProtocolClient is now available on Linux

Things that will help get your PR across the finish line:

  • Follow the JavaScript, C++, and Python coding style.
  • Run npm run lint locally to catch formatting errors earlier.
  • Document any user-facing changes you've made following the documentation styleguide.
  • Include tests when adding/changing behavior.
  • Include screenshots and animated GIFs whenever possible.

We get a lot of pull requests on this repo, so please be patient and we will get back to you as soon as we can.

@deepak1556
Copy link
Member

Although the patch looks good, I would like to get a better understanding of the side effects from changing the rate limiter for writes, storage-dev would be the best place to get more thoughts on this approach.

@jacobq
Copy link
Contributor Author

jacobq commented Nov 6, 2018

I would like to get a better understanding of the side effects from changing the rate limiter for writes, storage-dev

I just threw that in there "just in case" as, once again, it seems like something that shouldn't be restricted in Electron. If it will help land this sooner I will take that out.

@jacobq jacobq force-pushed the master branch 2 times, most recently from 1912cb2 to a9938de Compare November 6, 2018 22:39
@jacobq jacobq changed the title WIP: fix: Bypass localStorage quota (ref #8337) fix: bypass DOM storage quota (fix #8337) Nov 6, 2018
@jacobq
Copy link
Contributor Author

jacobq commented Nov 6, 2018

@deepak1556 I removed the change about which you had concerns. Would you mind reviewing again?

@MarshallOfSound
Copy link
Member

MarshallOfSound commented Nov 7, 2018

needed for running tests, right?

@jacobq Actually on some platforms you can't run the tests on the debug build 😄, we have a testing build config that we use to run the tests

Copy link
Member

@deepak1556 deepak1556 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, can you add a description in the patch file for why it is needed, #15612 is a good example. Thanks!

@jacobq
Copy link
Contributor Author

jacobq commented Nov 7, 2018

@deepak1556 Sure, description starts here. More discussion may also be found in the related issues.

@jacobq
Copy link
Contributor Author

jacobq commented Nov 7, 2018

Could this also be backported to 3-0-x?

@jacobq
Copy link
Contributor Author

jacobq commented Nov 7, 2018

Note: failed win-ia32 test appears unrelated. Would one of the releasers please restart the failed build(s)?

@jacobq
Copy link
Contributor Author

jacobq commented Nov 7, 2018

Similar to the win-ia32 false positive, would one of the reviewers please restart the build for mas-testing-tests and osx-testing-tests. I don't think those failures were caused by this PR.

Copy link
Member

@deepak1556 deepak1556 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM , Thanks!

@alexeykuzmin alexeykuzmin changed the title fix: bypass DOM storage quota (fix #8337) fix: bypass DOM storage quota Nov 10, 2018
Copy link
Contributor

@alexeykuzmin alexeykuzmin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

spec/chromium-spec.js Outdated Show resolved Hide resolved
spec/chromium-spec.js Outdated Show resolved Hide resolved
@@ -937,6 +937,23 @@ describe('chromium feature', () => {
})

describe('storage', () => {
describe('DOM storage quota override', () => {
['localStorage', 'sessionStorage'].forEach((storageName) => {
it(`can store at least 50MiB in ${storageName}`, (done) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A done callback is not needed for synchronous tests, please remove it.

@alexeykuzmin
Copy link
Contributor

Could this also be backported to 3-0-x?

@jacobq
Yes, but not automatically. We have significantly changed a layout of the Electron repositories between versions 3 and 4.
If you want to see this change in the Electron 3.x release line, please open a PR against the electron-3-0-x branch of the https://github.com/electron/libchromiumcontent repo, changing the same patch file. Thank you.

Co-Authored-By: jacobq <jacobq@gmail.com>
@jacobq
Copy link
Contributor Author

jacobq commented Nov 11, 2018

@alexeykuzmin OK, I just added some more commits and think I got all the changes you requested. Once this lands I'll open a separate PR for 3.0.x backport.

Edit: Looks like it won't be quite as simple as committing the same patch file because the hash of the files being patched has changed. So I think I'll have to regenerate it against chrome 66, not a biggy, but not a 5 minute ordeal for me either.

@jacobq
Copy link
Contributor Author

jacobq commented Nov 12, 2018

Hello again...would someone please re-run the 2 failed CI jobs? (mas-testing-tests & osx-testing-tests)

Copy link
Contributor

@alexeykuzmin alexeykuzmin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jacobq Thank you for the fix.
I just restarted those CI jobs.

@alexeykuzmin alexeykuzmin merged commit b21dbdb into electron:master Nov 12, 2018
@release-clerk
Copy link

release-clerk bot commented Nov 12, 2018

Release Notes Persisted

Fixes regression regarding localStorage quota not being bypassed.

@welcome
Copy link

welcome bot commented Nov 12, 2018

Congrats on merging your first pull request! 🎉🎉🎉

@trop
Copy link
Contributor

trop bot commented Nov 12, 2018

I have automatically backported this PR to "4-0-x", please check out #15686

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

localStorage values are limited to 10Mb
5 participants