Discussion: Offline Support

[ibash]

Is anyone else interested in offline support?

I'm using nativefier for Safari Books Online, and I'd like to cache the site and content so I can read anywhere. I've built offline web apps before, so I have some thoughts on how to implement this. Would anyone else be interested in this?

[ghost]

I would be interested.

[ibash]

Started to work on a proof of concept.

I think this should work by:

1. Inject a service worker into the site
2. Use something like sw-toolbox to do a network first cache
3. Make the routes and pre-caching customizeable

I'm running into trouble injecting the service worker (chrome doesn't want you to do that for security reasons). Here are the approaches we can try:

1. use protocol.interceptHttpProtocol

   The caveat is that you need to use a different session to actualy send the requests. I experimented a small bit with this. I think this is still viable, but may lead to annoying edge cases.

2. use webRequest.onBeforeRequest and redirect the service worker url

   This fails with a mysterious error. Chrome doesn't want you to do this, but it might still be viable using a custom protocol. I haven't looke into the error yet.

3. disable security on the web contentso

   Fairly confident this would work, but I haven't tried it yet.

4. use a proxy

   This is probably the best way to go.

5. serve the website from a custom protocol

   If you do that then you can make that custom protocol serve the service worker.

[niutech]

It's a great idea to enable offline mode. You could bundle a caching proxy server in node.js, e.g. node-caching-proxy, node-http-cache, node-fishback or node-http-proxy with caching middleware, then pass all the requests through it.

[ibash]

I have a proof of concept working on my machine using interceptStreamProtocol. Cleaning up the code and then will send a PR :)

[ibash]

I got this mostly working but hit this bug in electron. Once that's fixed, this should work. If anyone is interested the code is here: https://github.com/ibash/nativefier/tree/offline_2

[ibash]

My fix for electron should land soon: electron/electron#14887 after that I can go back to hacking on it in https://github.com/ibash/nativefier/tree/offline_2.

[ibash]

@puneetsl proof of concept works. Electron merged my initial fix but not the backported fix: electron/electron#16209

[louguitar]

Did you ever get this working? I am looking for a way to create an offline nativfier app.

[ibash]

@louguitar not all the way.

The problem I ran into was electron/electron#16421. The summary: my approach was to intercept network requests using electron's apis. If you choose to intercept there's a lot of built-in chrome functionality that gets bypassed (like cookie handling, checking certificates, etc). This leads to a whack-a-mole of trying to recreate chrome's network handling.

I haven't looked at what's changed in electron, but if there was an api that allowed injecting a service worker or the ability to both intercept and then send network requests back to chrome, that would make this trivial to implement.

[ibash]

I found a workaround.

You can inject a service worker without disabling web security by:

1. Enabling service workers on the filesystem scheme
2. Creating a service worker script using the html5 filesystem api
3. Registering the file created

The works because "filesystem:https://example.com" and "https://example.com" are considered to have the same origin. See this comment in chrome.

One gotcha was that enabling service workers for the filesystem scheme needs to be done on the command line and not with protocol.registerSchemesAsPrivileged.

I've made an example here: https://github.com/ibash/service-worker-experiment
If you clone the service-worker-experiment repo and do yarn start you will see:

1. https://example.com is loaded
2. A service worker is registered that's defined in service-worker-loader.js

So: With this technique we could enable injecting service workers into arbitrary sites. Once you're able to inject a service worker, you can use them to cache network requests to make a site work offline.

The only gotcha is that to make this robust we need to use the electron webRequest api to allow service workers and get rid of any CSP restrictions.

[ErickOhm]

is there an update to this? I'm trying to add offline functionality to a website (jstris) if you save the page you get the html/css/js files needed to play offline single player, so I'd like to have this on my app too, it's basically just 1 html file and that links to the css/js and some images, can I add these files to the project or is there a better way to do this?

[ibash]

@erickmack I didn't continue working on it, you could moving the two commits in this proof of concept: https://github.com/ibash/service-worker-experiment and then adding a service worker to see if that lets you make it work offline.

I haven't tried running it in a while, so really not sure if it will work.

[tarruda]

Hi @ibash, it seems https://github.com/ibash/service-worker-experiment is not accessible. Any chance you could publish it again?

[ibash]

Hi @ibash, it seems https://github.com/ibash/service-worker-experiment is not accessible. Any chance you could publish it again?

Done!

[tarruda]

Hi @ibash, it seems https://github.com/ibash/service-worker-experiment is not accessible. Any chance you could publish it again?

Done!

Thank you very much!

[tarruda]

Since the service-worker solution didn't work on electron 8, I ended up patching intercept*Protocol APIs to allow selecting which URLs are intercepted: https://gist.github.com/tarruda/554962f987f7f3a1a94d2c1c76c880a8#file-allow-url-patterns-intercept-protocol-diff-L186

[ibash]

nice!

1. curious what you're working on / what you need offline for?
2. one gotcha with the intercept route (unless I'm misremembering) is that the chrome network stack had some differences of how it handles requests than with the node.js network stack, I can't remember the actual implications.

[tarruda]

curious what you're working on / what you need offline for?

I use it to replace the response of an external resource (say www.example.com) with my own cached version while keeping chromium "thinking" it was loaded from the right origin, so no CORS headers are necessary.

one gotcha with the intercept route (unless I'm misremembering) is that the chrome network stack had some differences of how it handles requests than with the node.js network stack, I can't remember the actual implications

Yes, there are differences, which is why I wanted to continue using chromium network stack for all other URLs except the ones I need to intercept. For simple resources such as static pages, returning a simple response from the intercept handler is often enough to make everything work.

[alexjyong]

Anything with this?

I'm trying to make this website work in an app for easy use: https://eloquentjavascript.net/

Since saving the files and running it via file://path/to/html/file doesn't work.

I can run python3 -m http.server 8000 to get it to work. But trying to set this up so folks can use it offline without having to mess with the terminal