feat(service-worker): add support for configuring navigations URLs

The ServiceWorker will redirect navigation requests that don't match any
`asset` or `data` group to the specified index file. The rules for a
request to be classified as a navigation request are as follows:
1. Its `mode` must be `navigation`.
2. It must accept a `text/html` response.
3. Its URL must match certain criteria (see below).

By default, a navigation request can have any URL except for:
1. URLs containing `__`.
2. URLs to files (i.e. containing a file extension in the last path
   segment).

While these rules are fine in many cases, sometimes it is desirable to
configure different rules for the URLs of navigation requests (e.g.
ignore specific URLs and pass them through to the server).

This commit adds support for specifying an optional `navigationUrls`
list in `ngsw-config.json`, which contains URLs or simple globs
(currently only recognizing `!`, `*` and `**`).
Only requests whose URLs match any of the positive URLs/patterns and
none of the negative ones (i.e. URLs/patterns starting with `!`) will be
considered navigation requests (and handled accordingly by the SW).

(This is an alternative implementation to angular#23025.)

Fixes angular#20404

aio/content/guide/service-worker-config.md

@@ -18,10 +18,11 @@ ngsw-config dist src/ngsw-config.json /base/href
 
 The configuration file uses the JSON format. All file paths must begin with `/`, which is the deployment directory—usually `dist` in CLI projects.
 
-Patterns use a limited glob format:
+{@a glob-patterns}
+Unless otherwise noted, patterns use a limited glob format:
 
 * `**` matches 0 or more path segments.
-* `*` matches exactly one path segment or filename segment.
+* `*` matches 0 or more characters excluding `/`.
 * The `!` prefix marks the pattern as being negative, meaning that only files that don't match the pattern will be included.
 
 Example patterns:
@@ -37,6 +38,7 @@ Each section of the configuration file is described below.
 This section enables you to pass any data you want that describes this particular version of the app.
 The `SwUpdate` service includes that data in the update notifications. Many apps use this section to provide additional information for the display of UI popups, notifying users of the available update.
 
+{@a index-file}
 ## `index`
 
 Specifies the file that serves as the index page to satisfy navigation requests. Usually this is `/index.html`.
@@ -102,7 +104,8 @@ This section describes the resources to cache, broken up into three groups.
 
 * `versionedFiles` is like `files` but should be used for build artifacts that already include a hash in the filename, which is used for cache busting. The Angular service worker can optimize some aspects of its operation if it can assume file contents are immutable.
 
-* `urls` includes both URLs and URL patterns that will be matched at runtime. These resources are not fetched directly and do not have content hashes, but they will be cached according to their HTTP headers. This is most useful for CDNs such as the Google Fonts service.
+* `urls` includes both URLs and URL patterns that will be matched at runtime. These resources are not fetched directly and do not have content hashes, but they will be cached according to their HTTP headers. This is most useful for CDNs such as the Google Fonts service.<br>
+  _(Negative glob patterns are not supported.)_
 
 ## `dataGroups`
 
@@ -128,7 +131,8 @@ export interface DataGroup {
 Similar to `assetGroups`, every data group has a `name` which uniquely identifies it.
 
 ### `urls`
-A list of URL patterns. URLs that match these patterns will be cached according to this data group's policy.
+A list of URL patterns. URLs that match these patterns will be cached according to this data group's policy.<br>
+  _(Negative glob patterns are not supported.)_
 
 ### `version`
 Occasionally APIs change formats in a way that is not backward-compatible. A new version of the app may not be compatible with the old API format and thus may not be compatible with existing cached resources from that API.
@@ -164,3 +168,39 @@ The Angular service worker can use either of two caching strategies for data res
 * `performance`, the default, optimizes for responses that are as fast as possible. If a resource exists in the cache, the cached version is used. This allows for some staleness, depending on the `maxAge`, in exchange for better performance. This is suitable for resources that don't change often; for example, user avatar images.
 
 * `freshness` optimizes for currency of data, preferentially fetching requested data from the network. Only if the network times out, according to `timeout`, does the request fall back to the cache. This is useful for resources that change frequently; for example, account balances.
+
+## `navigationUrls`
+
+This optional section enables you to specify a custom list of URLs that will be redirected to the index file.
+
+### Handling navigation requests
+
+The ServiceWorker will redirect navigation requests that don't match any `asset` or `data` group to the specified [index file](#index-file). A request is considered to be a navigation request if:
+
+1. Its [mode](https://developer.mozilla.org/en-US/docs/Web/API/Request/mode) is `navigation`.
+2. It accepts a `text/html` response (as determined by the value of the `Accept` header).
+3. Its URL matches certain criteria (see below).
+
+By default, these criteria are:
+
+1. The URL must not contain a file extension (i.e. a `.`) in the last path segment.
+2. The URL must not contain `__`.
+
+### Matching navigation request URLs
+
+While these default criteria are fine in most cases, it is sometimes desirable to configure different rules. For example, you may want to ignore specific routes (that are not part of the Angular app) and pass them through to the server.
+
+This field contains an array of URLs and [glob-like](#glob-patterns) URL patterns that will be matched at runtime. It can contain both negative patterns (i.e. patterns starting with `!`) and non-negative patterns and URLs.
+
+Only requests whose URLs match _any_ of the non-negative URLs/patterns and _none_ of the negative ones will be considered navigation requests. The URL query will be ignored when matching.
+
+If the field is omitted, it defaults to:
+
+```ts
+[
+  '/**',           // Include all URLs.
+  '!/**/*.*',      // Exclude URLs to files.
+  '!/**/*__*',     // Exclude URLs containing `__` in the last segment.
+  '!/**/*__*/**',  // Exclude URLs containing `__` in any other segment.
+]
+```

packages/service-worker/config/src/generator.ts

@@ -11,6 +11,13 @@ import {Filesystem} from './filesystem';
 import {globToRegex} from './glob';
 import {Config} from './in';
 
+const DEFAULT_NAVIGATION_URLS = [
+  '/**',           // Include all URLs.
+  '!/**/*.*',      // Exclude URLs to files (containing a file extension in the last segment).
+  '!/**/*__*',     // Exclude URLs containing `__` in the last segment.
+  '!/**/*__*/**',  // Exclude URLs containing `__` in any other segment.
+];
+
 /**
  * Consumes service worker configuration files and processes them into control files.
  *
@@ -23,10 +30,11 @@ export class Generator {
     const hashTable = {};
     return {
       configVersion: 1,
-      index: joinUrls(this.baseHref, config.index),
       appData: config.appData,
+      index: joinUrls(this.baseHref, config.index),
       assetGroups: await this.processAssetGroups(config, hashTable),
       dataGroups: this.processDataGroups(config), hashTable,
+      navigationUrls: processNavigationUrls(this.baseHref, config.navigationUrls),
     };
   }
 
@@ -80,6 +88,14 @@ export class Generator {
   }
 }
 
+export function processNavigationUrls(baseHref: string, urls = DEFAULT_NAVIGATION_URLS): {positive: boolean, regex: string}[] {
+  return urls.map(url => {
+    const positive = !url.startsWith('!');
+    url = positive ? url : url.substr(1);
+    return {positive, regex: `^${urlToRegex(url, baseHref)}$`};
+  });
+}
+
 function globListToMatcher(globs: string[]): (file: string) => boolean {
   const patterns = globs.map(pattern => {
     if (pattern.startsWith('!')) {

packages/service-worker/config/src/in.ts

@@ -26,6 +26,7 @@ export interface Config {
   index: string;
   assetGroups?: AssetGroup[];
   dataGroups?: DataGroup[];
+  navigationUrls?: string[];
 }
 
 /**
@@ -52,4 +53,4 @@ export interface DataGroup {
   cacheConfig: {
     maxSize: number; maxAge: Duration; timeout?: Duration; strategy?: 'freshness' | 'performance';
   };
-}
+}

packages/service-worker/config/test/generator_spec.ts

@@ -20,10 +20,10 @@ import {MockFilesystem} from '../testing/mock';
       });
       const gen = new Generator(fs, '/test');
       const res = gen.process({
-        index: '/index.html',
         appData: {
           test: true,
         },
+        index: '/index.html',
         assetGroups: [{
           name: 'test',
           resources: {
@@ -52,40 +52,56 @@ import {MockFilesystem} from '../testing/mock';
             maxAge: '3d',
             timeout: '1m',
           }
-        }]
+        }],
+        navigationUrls: [
+          '/included/absolute/**',
+          '!/excluded/absolute/**',
+          '/included/some/url?with+escaped+chars',
+          '!excluded/relative/*.txt',
+          'http://example.com/included',
+          '!http://example.com/excluded',
+        ],
       });
       res.then(config => {
            expect(config).toEqual({
-             'configVersion': 1,
-             'index': '/test/index.html',
-             'appData': {
-               'test': true,
+             configVersion: 1,
+             appData: {
+               test: true,
              },
-             'assetGroups': [{
-               'name': 'test',
-               'installMode': 'prefetch',
-               'updateMode': 'prefetch',
-               'urls': [
+             index: '/test/index.html',
+             assetGroups: [{
+               name: 'test',
+               installMode: 'prefetch',
+               updateMode: 'prefetch',
+               urls: [
                  '/test/index.html',
                  '/test/foo/test.html',
                  '/test/test.txt',
                ],
-               'patterns': [
+               patterns: [
                  '\\/absolute\\/.*',
                  '\\/some\\/url\\?with\\+escaped\\+chars',
                  '\\/test\\/relative\\/[^\\/]+\\.txt',
                ]
              }],
-             'dataGroups': [{
-               'name': 'other',
-               'patterns': ['\\/api\\/.*', '\\/test\\/relapi\\/.*'],
-               'strategy': 'performance',
-               'maxSize': 100,
-               'maxAge': 259200000,
-               'timeoutMs': 60000,
-               'version': 1,
+             dataGroups: [{
+               name: 'other',
+               patterns: ['\\/api\\/.*', '\\/test\\/relapi\\/.*'],
+               strategy: 'performance',
+               maxSize: 100,
+               maxAge: 259200000,
+               timeoutMs: 60000,
+               version: 1,
              }],
-             'hashTable': {
+             navigationUrls: [
+               {positive: true, regex: '^\\/included\\/absolute\\/.*$'},
+               {positive: false, regex: '^\\/excluded\\/absolute\\/.*$'},
+               {positive: true, regex: '^\\/included\\/some\\/url\\?with\\+escaped\\+chars$'},
+               {positive: false, regex: '^\\/test\\/excluded\\/relative\\/[^\\/]+\\.txt$'},
+               {positive: true, regex: '^http:\\/\\/example\\.com\\/included$'},
+               {positive: false, regex: '^http:\\/\\/example\\.com\\/excluded$'},
+             ],
+             hashTable: {
                '/test/test.txt': '18f6f8eb7b1c23d2bb61bff028b83d867a9e4643',
                '/test/index.html': 'a54d88e06612d820bc3be72877c74f257b561b19',
                '/test/foo/test.html': '18f6f8eb7b1c23d2bb61bff028b83d867a9e4643'
@@ -95,5 +111,32 @@ import {MockFilesystem} from '../testing/mock';
          })
           .catch(err => done.fail(err));
     });
+
+    it('uses default `navigationUrls` if not provided', (done: DoneFn) => {
+      const fs = new MockFilesystem({
+        '/index.html': 'This is a test',
+      });
+      const gen = new Generator(fs, '/test');
+      const res = gen.process({
+        index: '/index.html',
+      });
+      res.then(config => {
+           expect(config).toEqual({
+             configVersion: 1,
+             appData: undefined,
+             index: '/test/index.html',
+             assetGroups: [],
+             dataGroups: [],
+             navigationUrls: [
+               {positive: true, regex: '^\\/.*$'},
+               {positive: false, regex: '^\\/(?:.+\\/)?[^\\/]+\\.[^\\/]+$'},
+               {positive: false, regex: '^\\/(?:.+\\/)?[^\\/]+__[^\\/]+\\/.*$'},
+             ],
+             hashTable: {}
+           });
+           done();
+         })
+          .catch(err => done.fail(err));
+    });
   });
 }

packages/service-worker/test/integration_spec.ts

@@ -41,6 +41,7 @@ const manifest: Manifest = {
     urls: ['/only.txt'],
     patterns: [],
   }],
+  navigationUrls: [],
   hashTable: tmpHashTableForFs(dist),
 };
 
@@ -55,6 +56,7 @@ const manifestUpdate: Manifest = {
     urls: ['/only.txt'],
     patterns: [],
   }],
+  navigationUrls: [],
   hashTable: tmpHashTableForFs(distUpdate),
 };
 

packages/service-worker/worker/src/app-version.ts

@@ -13,7 +13,6 @@ import {DataGroup} from './data';
 import {Database} from './database';
 import {IdleScheduler} from './idle';
 import {Manifest} from './manifest';
-import {isNavigationRequest} from './util';
 
 
 /**
@@ -40,6 +39,12 @@ export class AppVersion implements UpdateSource {
    */
   private dataGroups: DataGroup[];
 
+  /**
+   * Requests to URLs that match any of the `include` RegExps and none of the `exclude` RegExps
+   * are considered navigation requests and handled accordingly.
+   */
+  private navigationUrls: {include: RegExp[], exclude: RegExp[]};
+
   /**
    * Tracks whether the manifest has encountered any inconsistencies.
    */
@@ -79,6 +84,14 @@ export class AppVersion implements UpdateSource {
                               config => new DataGroup(
                                   this.scope, this.adapter, config, this.database,
                                   `ngsw:${config.version}:data`));
+
+    // Create `include`/`exclude` RegExps for the `navigationUrls` declared in the manifest.
+    const includeUrls = manifest.navigationUrls.filter(spec => spec.positive);
+    const excludeUrls = manifest.navigationUrls.filter(spec => !spec.positive);
+    this.navigationUrls = {
+      include: includeUrls.map(spec => new RegExp(spec.regex)),
+      exclude: excludeUrls.map(spec => new RegExp(spec.regex)),
+    };
   }
 
   /**
@@ -151,15 +164,36 @@ export class AppVersion implements UpdateSource {
 
     // Next, check if this is a navigation request for a route. Detect circular
     // navigations by checking if the request URL is the same as the index URL.
-    if (isNavigationRequest(req, this.scope.registration.scope, this.adapter) &&
-        req.url !== this.manifest.index) {
+    if (req.url !== this.manifest.index && this.isNavigationRequest(req)) {
       // This was a navigation request. Re-enter `handleFetch` with a request for
       // the URL.
       return this.handleFetch(this.adapter.newRequest(this.manifest.index), context);
     }
+
     return null;
   }
 
+  /**
+   * Determine whether the request is a navigation request.
+   * Takes into account: Request mode, `Accept` header, `navigationUrls` patterns.
+   */
+  isNavigationRequest(req: Request): boolean {
+    if (req.mode !== 'navigate') {
+      return false;
+    }
+
+    if (!this.acceptsTextHtml(req)) {
+      return false;
+    }
+
+    const urlPrefix = this.scope.registration.scope.replace(/\/$/, '');
+    const url = req.url.startsWith(urlPrefix) ? req.url.substr(urlPrefix.length) : req.url;
+    const urlWithoutQueryOrHash = url.replace(/[?#].*$/, '');
+
+    return this.navigationUrls.include.some(regex => regex.test(urlWithoutQueryOrHash)) &&
+        !this.navigationUrls.exclude.some(regex => regex.test(urlWithoutQueryOrHash));
+  }
+
   /**
    * Check this version for a given resource with a particular hash.
    */
@@ -239,4 +273,16 @@ export class AppVersion implements UpdateSource {
    * Get the opaque application data which was provided with the manifest.
    */
   get appData(): Object|null { return this.manifest.appData || null; }
+
+  /**
+   * Check whether a request accepts `text/html` (based on the `Accept` header).
+   */
+  private acceptsTextHtml(req: Request): boolean {
+    const accept = req.headers.get('Accept');
+    if (accept === null) {
+      return false;
+    }
+    const values = accept.split(',');
+    return values.some(value => value.trim().toLowerCase() === 'text/html');
+  }
 }