Add support for an include option (in addition to ignore)

[andreineculau]

Preflight Checklist

• I have read the contribution documentation for this project.
• I agree to follow the code of conduct that this project follows, as appropriate.
• I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Problem Description

AFAICS (apologies if I missed smth) electron-packager has a way to ignore paths from getting copied to the output artifact, but blacklisting is a risky way to decide what goes in. Add a file with sensitive info to the repo, forget to add it to ignore, and BAM!

A whitelist is stable and does answer directly the question: what goes into the artifact?

Proposed Solution

In the best of worlds, you should just follow npm pack, meaning respect the files section in package.json, and if none - respect the .npmignore (or just stick to the current --ignore flag).

[welcome]

👋 Thanks for opening your first issue here! If you have a question about using Electron Packager, read the support docs. If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. Development and issue triage is community-driven, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[malept]

I think we need more detail around this feature before we decide whether to add it to the code:

• Does this replace or augment the current ignore support?
• If it replaces ignore, what is the migration strategy for users?
• If it is merely added alongside ignore, which feature takes precedence when Packager runs?
• Are there any potential problems with interacting with the prune feature?
• How is this feature going to be specified by the user? Via glob, regexp, or both?
• What does the API look like? What does the CLI option look like?

Using existing NPM config is probably a bad idea, given the myriad different ways that users structure their code repositories.

[andreineculau]

Sorry, I might have been too terse.

I can answer these from my POV:

Does this replace or augment the current ignore support?

Augment. Makes no sense to break current use-cases.

If it is merely added alongside ignore, which feature takes precedence when Packager runs?

Whitelist is checked first, a subset of files is returned, then ignore settings are applied on top of that subset.

Are there any potential problems with interacting with the prune feature?

Not that I see. Whitelisting doesn't apply to the "node_modules" folder.

How is this feature going to be specified by the user? Via glob, regexp, or both?

My suggestion is simply that electron-packager respects the package.json "standard" of the files section, so glob. I should mention that I wouldn't have brought the files section in the discussion, if it wasn't already for electron-packager using other fields from package.json.

What does the API look like? What does the CLI option look like?

The API is the files section.

Additionally, if really really needed, electron-packager could have some --files CLI optino, but the accepted values should probably be regexp since that's what --ignore uses today AFAIK.

---

Let me know if there's more.

[andreineculau]

Using existing NPM config is probably a bad idea, given the myriad different ways that users structure their code repositories.

I forgot to say that this has little to do with npm, and everything to do with package.json . The fact that package.json is an npm in/convention is besides the point. AFAIK all packagers (npm, yarn, pnpm, etc) adhere to the same rules of the files section.

[LordMidi]

Any news here? Please give me an example to exclude everything except one folder i.e. "dist".
I'm getting mad finding a working RegEx to provide it to the "ignore" parameter. This RegEx should work ^(?!.*(dist|node_modules)).* but it doesn't. Using this as "ignore" param I get no app folder at all - nothing gets copied. I couldn't figure out what is used internally for the ignore match. Greetings (edit: maybe I have to build a script via the API to fit our project needs)

[HaimBendanan]

@LordMidi did you find a solution for the regex?

[LordMidi]

@HaimBendanan No - but I use a package script instead. This removes everything that should not be included. See all options: https://github.com/electron/electron-packager/blob/master/docs/api.md

const fs = require('fs');
const path = require('path');
const rimraf = require('rimraf');
const packager = require('electron-packager');

const packagerOptions = {
  dir: '.',
  out: 'dist_electron',
  platform: 'win32',
  overwrite: true,
  asar: true,
  afterCopy: [
    cleanSources
  ]
};

packager(packagerOptions).then(outPath => {
  console.log(`build path: ${outPath}`);
});

// remove folders & files not to be included in the app
function cleanSources(buildPath, electronVersion, platform, arch, callback) {

  // folders & files to be included in the app
  const appItems = [
    'dist',
    'electron',
    'main.js',
    'node_modules'
  ];

  fs.readdirSync(buildPath).filter((item) => {
    return appItems.indexOf(item) === -1
  }).forEach((item) => {
    rimraf.sync(path.join(buildPath, item));
  });

  console.log('removed folders & files not to be included in the app');
  callback();
}

[HaimBendanan]

@HaimBendanan No - but I use a package script instead. This removes everything that should not be included. See all options: https://github.com/electron/electron-packager/blob/master/docs/api.md

const fs = require('fs');
const path = require('path');
const rimraf = require('rimraf');
const packager = require('electron-packager');

const packagerOptions = {
  dir: '.',
  out: 'dist_electron',
  platform: 'win32',
  overwrite: true,
  asar: true,
  afterCopy: [
    cleanSources
  ]
};

packager(packagerOptions).then(outPath => {
  console.log(`build path: ${outPath}`);
});

// remove folders & files not to be included in the app
function cleanSources(buildPath, electronVersion, platform, arch, callback) {

  // folders & files to be included in the app
  const appItems = [
    'dist',
    'electron',
    'main.js',
    'node_modules'
  ];

  fs.readdirSync(buildPath).filter((item) => {
    return appItems.indexOf(item) === -1
  }).forEach((item) => {
    rimraf.sync(path.join(buildPath, item));
  });

  console.log('removed folders & files not to be included in the app');
  callback();
}

I ended up using electron-builder, which support a 'files' attribute for added dependencies- but thanks anyway :)

[malept]

I made an attempt to implement this over the US holiday weekend. The simple case works fine (top-level files/folders), but once you start trying to handle complex globs, the amount of code that would need to be written on the Electron Packager side in order for it to work with fs-extra's copy function would be too much to maintain. The reason for this is that fs-extra.copy recurses through the folder tree, so globs would match leaves, not branches. Adding branch support is the complication, and I fear that there would be a lot of edge cases in that code.

The alternative is to use the cpy module instead of fs-extra.copy. You can pass it multiple globs as the source argument. The reason why I haven't used it is because it currently lacks support for dereferencing symlinks, which is also an option in Electron Packager. If that feature is implemented in cpy, then we can proceed with this feature (at the cost of adding a new dependency + any extra transitive dependencies).

[glen-84]

@malept,

Why not just use something like fast-glob (which has a followSymbolicLinks option), and then copy the resulting file paths? (after filtering out ignored files)

[malept]

That would require effectively reimplementing recursive copy, which I don't want to write or maintain.

[glen-84]

@malept,

Why would it be recursive? fast-glob would give you a flat array of paths like:

['.editorconfig', 'services/index.js']

... I might be missing something, as I'm not familiar with this project's code.

[malept]

It's recursive because you have to also ensure that the directories are created with the same metadata as the original, macOS/Linux permissions for example. In your example, the services directory.

Turns out cpy uses fast-glob as a transitive dependency.

[malept]

I looked at fast-glob's option and it's a different option. The docs say:

Indicates whether to traverse descendants of symbolic link directories.

This is different from the fs-extra.copy option, which dereferences symlinks on copy for both files and folders.

[glen-84]

So using fs-extra.copy on each path from fast-glob wouldn't work?

[malept]

So using fs-extra.copy on each path from fast-glob wouldn't work?

That has the same problem as #972 (comment):

you have to also ensure that the directories are created with the same metadata as the original

It would be much cleaner to use cpy.

[glen-84]

Oh, I thought that copying a/b/c.d to e would result in all the directories having their attributes preserved, and that cpy copied in this way as well (except using cp-file instead of fs-extra.copy). I didn't see any special recursive processing after globbing the files, but perhaps I missed it.

Anyway, hopefully a solution will be found soon, because blacklisting is far from ideal.

[jameshfisher]

I believe the perceived problem stems from the horrible Electron convention to mix distributable source files (e.g. main.js) with non-distributable files (e.g. my_sensitive_cert.pfx) in the same root directory, and then package it with packager({ dir: '.', ... }). If you do this, you need a whitelist/blacklist to sort them out.

A better convention is to put your distributable source files in a separate directory, e.g. app/, and distribute everything in that directory with packager({ dir: 'app', ... }). The app/ directory is then your whitelist.

(Also, if you really want a whitelist, note that ignore can also take a predicate function instead of a regex, letting you convert it to a whitelist.)

[zacharymarshal]

👋 I was able to get the regex to work using '^\/(?!(dist|node_modules|package\.json))' (the initial slash ensures it is in the root) ... if you include this as a script in your package.json make sure you escape the backslashes, e.g., '^\\/(?!(dist|node_modules|package\\.json))'

[flevi29]

👋 I was able to get the regex to work using '^\/(?!(dist|node_modules|package\.json))' (the initial slash ensures it is in the root) ... if you include this as a script in your package.json make sure you escape the backslashes, e.g., '^\\/(?!(dist|node_modules|package\\.json))'

That works great! However I'm a little confused as to what's happening because in the docs it says it's matched against the absolute path. Maybe I'm not understanding something or the docs are simply wrong.