AutoUpdater (NSIS) - Feature Request - Custom URLs Don't Work with Restful APIs

[dapperdandev]

• Version: 22.10.5
• Electron Version: 12.0.5
• Target: Windows 10

Our web application manages several downloadable files that are stored in azure storage accounts. We restrict access to our storage accounts to only our api. Each download has an ID that we use for our downloadable files without the file name. For example:

/api/downloads/{id}/download

We can upload our {application}.exe and {application}.exe.blockmap, but they will each have a different ID. For example:

/api/downloads/1/download - application.exe
/api/downloads/2/download - application.exe.blockmap

Unfortunately, this is hard to adapt to with electron-updater because it has no option (that I'm aware of) to omit the filename. So when I set publish.url to http://localhost:3000/api/downloads/update/{platform}/{channel} with a generic provider, electron-updater changes it to http://localhost:3000/api/downloads/update/{platform}/{channel}/{channel}.yml which is not desirable.

It would be nice to be able to specify each url without interference from electron-updater. It would be even nicer to prevent electron-updater from trying to append the filename since the endpoints can provide the correct MIME type to satisfy the updater.

Current Workaround:
We created a route that returns yaml in the same format as expected /api/downloads/update/{platform}/{channel} and added a rewrite rule in iis. This allows us to fake our way through checking for updates (though I have a feeling we'll run into issues on the SHA512 checksum):

        <rule name="Rewrite Desktop Update">
          <match url="api/downloads/update/(darwin|win32)/(latest|beta).yml$" />
          <action type="Rewrite" url="api/downloads/update/(darwin|win32)/(latest|beta)" />
        </rule>

This endpoint checks our storage account for the latest release for the given channel and builds the download URL with the correct ID:

version: 1.0.4
files:
  - url: http://localhost:3000/api/downloads/1/download # generated by finding the actual ID
    sha512: 711C22448E721E5491D8245B49425AA861F1FC4A15287F0735E203799B65CFFEC50B5ABD0FDDD91CD643AEB3B530D48F05E258E7E230A94ED5025C1387BB4E1B # fake sha512
    size: 96296728
releaseDate: '2021-05-26T17:28:05.116Z'

This only works so far. electron-updater appears to be satisfied with the yaml and downloads the .exe, but it then tries to download a blockmap from: http://localhost:3000/api/downloads/1/download.blockmap rather than http://localhost:3000/api/downloads/2/download

Suggested Improvement

• Allow developers to specify the blockmap URL.
• Provide developers an option to prevent electron-updater from adding anything other than what's specified to the URL.

[bryantb2]

I'm in a very similar situation with blob stores that can only be accessed via pre-determined IDs.

Adding the ability to set custom URLs for each respective file request, i.e. /my/update/route/latest.yml, in the updater configuration would go a long way to making the auto-updater more versatile.

[mmaietta]

I'm very interested in your configuration and the implementation behind it, as it seems akin to what I've been wanting to do with my project for over a year.
How are you setting up the custom URLs to work electron-updater? Do you use the distribution channels or are you looking to specify custom download URLs for each dist+blockmap for each channel? At compile time or at runtime?
I'm looking to have my own auto-check mechanism, and then just trigger electron-updater manually with an installer url.

[dapperdandev]

@mmaietta It's actually fairly simple in .NET Web API. Probably even simpler in Node + Express.

1. Since we're in .NET we had to create classes to replicate the UpdateInfo and UpdateFileInfo interfaces in updateinfo.ts: https://github.com/electron-userland/electron-builder/blob/master/packages/builder-util-runtime/src/updateInfo.ts with a little bit of modification for simplicity.

    /* Implementation of electron-builder/electron-updater UpdateInfo */
    public class ElectronUpdateInfo
    {
        public string FileName { get; set; }
        // We can use a ternary to determine channel. E.g., Version.Contains("-beta") ? "beta" : "latest"
        public string Version { get; set; }
        public IList<ElectronUpdateFileInfo> Files { get; set; }
        public string ReleaseDate { get; set; }
    }

    /* Implementation of electron-builder/electron-updater UpdateInfo */
    public class ElectronUpdateFileInfo
    {
        public long DownloadId { get; set; }
        public string Sha512 { get; set; }
        public long Size { get; set; }
        // Used for the custom download url
        public string Url => $"https://app.example.com/downloads/{DownloadId}/download"; 
    }

2. We store basic information in the database as well as the encrypted blob storage location:

public class Download : BaseEntity
{
        public string Name { get; set; }
        public string Description { get; set; }
        public string Content { get; set; }
        public byte[] ContentKey { get; set; }
        public string Sha512Checksum { get; set; } // Crucial.
}

3. We configured NSIS publish to point to our generic provider URL

  publish: [
    {
      provider: 'generic',
      // electron-updater will append ${channel}.yml to the end of this URL. Part of the request is to optionally disable this.
      url: `${baseUrl}/api/downloads/update/${platform}`,
      channel: someBoolean ? 'latest' : 'beta'
    }

4. We added a rewrite rule in IIS. Should give you an idea of how to translate to nginx or apache if needed.

<rule name="Rewrite Desktop Update">
  <match url="api/downloads/update/(darwin|win32)/(latest|beta).yml$" />
  <action type="Rewrite" url="api/downloads/update/(darwin|win32)/(latest|beta)" />
</rule>

5. Finally, we have an endpoint at the route in Step 3/4 (url: ${baseUrl}/api/downloads/update/${platform}/${channel}) that we generate yaml in the expected format in place. Note the addition of ${channel} at the end. This is because electron-updater adds ${channel}.yml to the end of the publish URL and our Rewrite rule essentially just removes the file extension (.yml).

var sb = new StringBuilder();
sb.Append($"version: {updateInfo.Version}\n");
sb.Append("files:\n");

foreach (var updateFileInfo in updateInfo.Files)
{
    sb.Append($"  - url: {updateFileInfo.Url}\n");
    sb.Append($"    sha512: {updateFileInfo.Sha512}\n");
    sb.Append($"    size: {updateFileInfo.Size}\n");
}

sb.Append($"releaseDate: '{updateInfo.ReleaseDate}'");

return sb.ToString(); // The endpoint is configured to return Content-Type `text/x-yaml`. Very important!

An example response tested with Postman returns:

version: 1.234.5
files:
  - url: https://app.example.com/api/downloads/1/download
    sha512: gmib0kbkMXtuwF6QVINTSrOSis2Gc3Ux591n+P4y6IKmseH1QdU6VHPi4u7HNldPbGRI1us8PQLKHAeLr4vDWw==
    size: 8675312
releaseDate: '2021-05-28T04:32:22.478Z'

---

To answer your questions:

How are you setting up the custom URLs to work electron-updater?

• Setting publish.provider = 'generic'and `publish.url = /* our custom url that returns yaml */

Do you use the distribution channels or are you looking to specify custom download URLs for each dist+blockmap for each channel?

We do this a little backwards. Each dist+blockmap gets it's own URL since we build the URL from the DownloadId after uploading it. So if we upload release 1.0.0-beta, it might go to

• application.exe @ /api/downloads/1/download
• application.exe.blockmap @ /api/downloads/2/download

Assuming we upload release 1.0.0 next, it will go to.

• application.exe @ /api/downloads/3/download
• application.exe.blockmap @ /api/downloads/4/download

At compile time or at runtime?

Technically neither. Our CI pipeline builds the binaries then sends a request to our API to upload them to our server via a route @ /api/downloads/upload.

In summary:

• electron-updater calls /api/downloads/update/${platform}/${channel}
• the above endpoint makes a query to the database using the platform and channel as criteria to retrieve the correct data
• yaml is generated in place with the response from the database.
• the electron app receives the yaml and sends a GET request to the .exe download URL from the yaml: /api/downloads/{id}/download
• the electron app makes a request for the blockmap. This part currently fails, but doesn't break downloading the .exe

---

Hope this helps! Like I said, it's probably way easier in Node + Express.

• You don't have to deal with StringBuilder. Just use a template string
• You might not have to deal with rewrite rules. I think you can just do app.get('/api/downloads/update/:platform/:channelFile'), but I could be wrong.
• You can set the response type with res.type(.yml). Again, I could be wrong. It's a little more work in .NET

[mmaietta]

Thank you for the detailed write-up! I appreciate it

I feel like this could be solved in a different manner, one which doesn't require working with the expectations GenericProvider. What if there were just a "manual" Provider? It'd be explicitly provided by the client at runtime instead of polling, effectively allowing one to kick off a manual update.

I found this magic code all the Provider's use. Bintray and Generic Providers simply ignore the pathTransformer param, but we could potentially leverage that.

electron-builder/packages/electron-updater/src/providers/Provider.ts

Lines 135 to 156 in ef704a1

	export function resolveFiles(updateInfo: UpdateInfo, baseUrl: URL, pathTransformer: (p: string) => string = (p: string): string => p): Array<ResolvedUpdateFileInfo> {
	const files = getFileList(updateInfo)
	const result: Array<ResolvedUpdateFileInfo> = files.map(fileInfo => {
	if ((fileInfo as any).sha2 == null && fileInfo.sha512 == null) {
	throw newError(`Update info doesn't contain nor sha256 neither sha512 checksum: ${safeStringifyJson(fileInfo)}`, "ERR_UPDATER_NO_CHECKSUM")
	}
	return {
	url: newUrlFromBase(pathTransformer(fileInfo.url), baseUrl),
	info: fileInfo,
	}
	})
	const packages = (updateInfo as WindowsUpdateInfo).packages
	const packageInfo = packages == null ? null : packages[process.arch] || packages.ia32
	if (packageInfo != null) {
	;(result[0] as any).packageInfo = {
	...packageInfo,
	path: newUrlFromBase(pathTransformer(packageInfo.path), baseUrl).href,
	}
	}
	return result
	}

We'd then also add an additional optional blockMapUrl field. This would mirror how UpdateInfo is already set up with files-> url + size. Now files -> blockMapSize would allow a corresponding blockMapUrl

electron-builder/packages/builder-util-runtime/src/updateInfo.ts

Lines 20 to 25 in ef704a1

	/**
	* The block map file size. Used when block map data is embedded into the file (appimage, windows web installer package).
	* This information can be obtained from the file itself, but it requires additional HTTP request,
	* so, to reduce request count, block map size is specified in the update metadata too.
	*/
	blockMapSize?: number

It's possible this could all be solved by opening pathTransformer as part of GenericProvider, but that feels more runtime-defined than achievable via a build configuration, which is why allowing custom Providers with a default httpExecutor could be viable. What are your thoughts?

[dapperdandev]

Hey @mmaietta ! I like both of your ideas:

1. Adding a manual provider
2. Adding the optional blockMapUrl

That said, I just noticed that PublishProvider has custom as an option:

electron-builder/packages/builder-util-runtime/src/publishOptions.ts

Line 3 in ef704a1

export type PublishProvider = "github" | "bintray" | "s3" | "spaces" | "generic" | "custom" | "snapStore"

Would adding manual add enough benefit to accomplish my goal of "Provide developers an option to prevent electron-updater from adding anything other than what's specified to the URL.", or should developers ideally use custom?

[mmaietta]

I never realized there was a custom 😅

Looking into that further, it seems you can actually completely custom-ize the Publisher as we were describing as "manual".

electron-builder/packages/app-builder-lib/src/publish/PublishManager.ts

Lines 333 to 346 in 8bf39ab

	default: {
	const name = `electron-publisher-${provider}`
	let module: any = null
	try {
	module = require(path.join(packager.buildResourcesDir, name + ".js"))
	} catch (ignored) {
	console.log(ignored)
	}
	if (module == null) {
	module = require(name)
	}
	return module.default || module
	}

class Publisher {
    async upload(task) { }
  }
  module.exports = Publisher

Example:

electron-builder/packages/electron-publish/src/publisher.ts

Lines 71 to 76 in 8bf39ab

	export abstract class HttpPublisher extends Publisher {
	protected constructor(protected readonly context: PublishContext, private readonly useSafeArtifactName = false) {
	super(context)
	}
	async upload(task: UploadTask): Promise<any> {

I imagine that should allow you to upload each file separately to different URLs as you require?

The downside is that a custom Provider is not (yet) supported:
https://github.com/electron-userland/electron-builder/blob/master/packages/electron-updater/src/providerFactory.ts#L22-L70

I suppose we can mirror the approach the custom publisher was using via a dynamic require to accept a CustomProvider. Then you'd just extend the GenericProvider class and override the
resolveFiles(updateInfo: UpdateInfo):Array<ResolvedUpdateFileInfo>

UpdateInfo has all the info already specified, so I guess the only missing functionality is having the blockmap URL established via the CustomPublisher and then that stored in the UpdateInfo for your CustomProvider?

What are your thoughts?

[wsw0108]

Agree with your idea, and that's the feature I just looking for.

Maybe we can configure two providers 'generic' and 'custom', then use 'generic' for updater, and use 'custom' for publish?

[mmaietta]

@djbreen7, take a look at this new custom provider that can use to define your own updateProvider? function. Might be able to partially solve what you're looking to achieve. It'll allow you to define your own resolveFiles function too AFAICT
#5984

[dapperdandev]

Thanks @mmaietta!

Might be a while before I can dive back and test, but looks promising.