How to exclude specific file types with addPassthroughCopy?

[jussivirtanen]

I have a website with a blog. Blog posts are arranged so that blog post folder (posts) contains all the necessary files (Markdown, images, pdfs etc) related to the post (this structure comes as given from a client). I'm trying to output all the blog files except the Markdown file of the post, but so far I haven't been able to do that.

The folder structure of the blog posts is as follows:

posts
- post-1
-- post-name.md
-- some-image.jpg
- post-2
-- another-post.md
-- some-file.pdf
-- maybe-even-something-else.svg

When I try to use addPassthroughCopy I'm not able to exclude the md files. The syntax I've tried is as follows:

eleventyConfig.addPassthroughCopy({ 'posts/**/!(*.md)': 'blog' })

The output will be:

blog
- post-1
-- post-name.html
- post-2
-- another-post.html

No other files (images, pdfs etc) will be copied to the output. The only way I'm able to copy all the content is to include, well, all the content, even the md file:

eleventyConfig.addPassthroughCopy({ 'posts': 'blog' })

This way the output is:

blog
- post-1
-- post-name.html
-- post-name.md <-- This I don't want
-- some-image.jpg
- post-2
-- another-post.html
-- another-post.md <-- This I don't want
-- some-file.pdf
-- maybe-even-something-else.svg

I'm not sure why, but I can get the exclusion working when working with a folder without subfolders. Here's an example with a press folder I also have on the website:

press
- press.liquid
- logo.jpg
- logo.svg

With the following pattern I can exclude the .liquid file:

eleventyConfig.addPassthroughCopy({ 'press/!(*.liquid)': 'press' })

The output will in this case be:

press
- press.html
- logo.jpg
- logo.svg

Any thoughts or tips how to get the md file exclusion working on blog posts?

[binyamin]

@jussivirtanen Passthrough File Copy allows using globs.

// Copy any markdown file
module.exports = function(eleventyConfig) {
    eleventyConfig.addPassthroughCopy("**/*.md);`
}

Edit: I just read this more closely. Try explicitly setting the templateFormats, so markdown is not processed. Out of interest, how are you generating the html if you don't want to output the markdown?

[jussivirtanen]

Thanks @binyamin for your quick reply! I probably explained my problem poorly, but I want my Markdown files to be processed (to html), but not passed through themselves. What I want to avoid is having the md file on the built site.

Let’s say I have a folder called posts, which includes blog posts each in their own subfolder. These post folders include all kinds of files in addition to the post itself (which is a markdown file). What I’d like to achieve is this:

• From posts folder (and its subfolders) copy all files except Markdown files to a folder called blog and keep folder structure
• From the same postsfolder, process all markdown files to html

So something like this:

// Input
posts
- post-1
-- post-1.md
-- some-pic.jpg
-- document.pdf

// Output
blog
- post-1
-- post-1.html
-- some-pic.jpg
-- document.pdf

Is that possible to achieve?

[binyamin]

That should be what you get without using Passthrough File Copy

[jussivirtanen]

Interesting. So you mean that all contents of the posts folder (like pdf, jpg, txt, rtf, anything) should automatically be output? I thought that Eleventy was supposed to process only certain files and everything else you need to pass manually.

I think what I'm attempting here is to use addPassthroughCopy to move everything except some files, but for some reason I can't figure out the right syntax. One option would be to manually add all possible file formats to addPassthroughCopy, but that seems tedious as the client might add some files that I haven't taken into account.

I could spin up a sample repo to demonstrate this better, or record a short screencast as soon as I get back to my computer.

[binyamin]

Hold on, I think I get this. You have assets (images, etc.) in the same folder as your md file. You want to passthroughFileCopy on everything except the md, because the md is already being processed. Is that it?

[jussivirtanen]

That's exactly it! Do you happen to know how to achieve that?

[binyamin]

I see two problems here, both of which are potential bugs.

First, when you use glob negation (eg. **/*.(!md)) there is no output at all. Second, eleventyConfig.addPassthroughCopy() doesn't duplicate the directory structure when you change the output. Eg. eleventyConfig.addPassthroughCopy({"posts/**/*.jpg": "blog"}); would output everything to the root of the blog folder.

In your case, globs won't work because of the first bug. You can't specify the individual file extensions, because of the second bug. I can think of two workarounds.

1. Name the input and output directories the same, and use passthrough file copy to target individual file types (eg. blog/**/*.{png,jpg,pdf}).
2. Keep all your images and assets in a single, higher-level folder. This might be the easiest to implement, although you would need to ensure against duplicate filenames.

[jussivirtanen]

Thanks for chiming in again @binyamin. I think I'll go with my current setup which passes through also the md file and wait for potential updates on this issue.

[milahu]

solved in patch #1686 where we allow to pass options to recursive-copy

eleventyConfig.addPassthroughCopy("node_modules/@fontsource/noto-sans", { expand: true });

eleventyConfig.addPassthroughCopy({
  'posts/**': 'blog',
}, {
  filter: path => (path.endsWith('md') == false)
});

edit: (path.slice(-3) != '.md') to (path.endsWith('md') == false)

[javanigus]

I'm trying to pass through copy all js files except 11tydata.js files. The folder structure should be maintained. I tried to add a filter but it didn't work. Any suggestions?

eleventyConfig.addPassthroughCopy('**/*.js', "_site", {
filter: path => (path.slice(-12) != '.11tydata.js')
});

[milahu]

eleventyConfig.addPassthroughCopy('**/*.js', "_site", {
  //filter: path => (path.endsWith('.11tydata.js') == false)
  // debug
  filter: path => {
    const doCopy = (path.endsWith('.11tydata.js') == false)
    console.log("copy", doCopy, path)
    return doCopy
  }
});

does this log your 11tydata.js files?
do you have other addPassthroughCopy?

[javanigus]

It doesn't log my 11tydata.js files. Also, when I rebuild the pages, I get multiple, nested _site folders. I do have other addPassthroughCopy statements. Here's my .eleventy.js file.
https://github.com/javanigus/eleventy-basic-site/blob/299e838db3211003836248c9a79e9a6adefbf32b/.eleventy.js

[javanigus]

I found a workaround. When the build is complete (eleventy.after event), I find and remove certain js (.11tydata.js) and json files from the build folder. This way, I can keep page-specific js and json files next to the pages themselves rather than put them all in the global /js folder, e.g.

/about/index.html
/about/index.js (local page-specific js)
/about/index.json (local page-specific json)
/js/main.js (global js)

I added the following to .eleventy.js.

const findRemoveSync = require('find-remove');

module.exports = function(eleventyConfig) {
    ...
    // Copy `css/` to `_site/css`, etc
    //eleventyConfig.addPassthroughCopy("css");
    eleventyConfig.addPassthroughCopy("img");
    //eleventyConfig.addPassthroughCopy("js");

    // tell 11ty which files to process and which files to copy while maintaining directory structure
    eleventyConfig.setTemplateFormats(["md","html","njk","css","json", "js"]);

    // Run me after the build ends
    eleventyConfig.on('eleventy.after', async () => {
        // Find and remove certain files from the build folder
        const pathToBuildFolder = __dirname + "\\_site";
        console.log(pathToBuildFolder);
        var result = findRemoveSync(pathToBuildFolder, { files: '\.11tydata\.js$', regex: true })
        console.log(result);
        result = findRemoveSync(pathToBuildFolder, { files: ['.eleventy.js', 'package.json', 'package-lock.json'] })
        console.log(result);
    });
    ...
};

[milahu]

Also, when I rebuild the pages, I get multiple, nested _site folders.

thats because you have the default config input: '.'
generally its better to use input: 'src'
then eleventyConfig.addPassthroughCopy('src/**/*.js', "_site") will not copy node_modules/

probably easier to start with a better starter

see also repos with filename:*.11tydata.js

[javanigus]

Good point about setting input to 'src' to avoid copying node_modules. I'll make that change and look at those eleventy starters. Thanks.

[pdehaan]

If you're using eleventy v2 (canary builds; npm i @11ty/eleventy@canary -D), you can now pass custom config options to the internal recursive-copy package (see https://www.11ty.dev/docs/copy/#advanced-options).

.eleventy.js

/**
 * @param {import("@11ty/eleventy/src/UserConfig")} eleventyConfig
 * @returns {ReturnType<import("@11ty/eleventy/src/defaultConfig")>}
 */
module.exports = function (eleventyConfig) {
  eleventyConfig.addPassthroughCopy("src", {
    // debug: true,
    filter: [
      "**/*.js",
      "!**/*.11ty.js",
      "!**/*.11tydata.js",
    ]
  });

  return {
    dir: {
      input: "src",
      output: "www",
    }
  };
};

Where we passthrough copy the "src" input directory, and then filter the files using the filter array to copy all *.js files, except *.11ty.js or *.11tydata.js files [by using the ! negation character].

OUTPUT

> DEBUG=Eleventy:TemplatePassthrough eleventy

  Eleventy:TemplatePassthrough Copying './src' +0ms
  Eleventy:TemplatePassthrough Copying individual file 'src/index.js' +13ms
[11ty] Copied 1 file / Wrote 2 files in 0.03 seconds (v2.0.0-canary.23)

tree -aI node_modules
.
├── .eleventy.js
├── package-lock.json
├── package.json
├── src/
│   ├── foo.11ty.js
│   ├── index.js
│   └── pages/
│       ├── index.11ty.js
│       └── pages.11tydata.js
└── www/
    ├── foo/index.html
    ├── index.js
    └── pages/index.html

5 directories, 10 files

[javanigus]

Thanks, @pdehaan. I changed the input and output to src and www, installed 11ty canary, copied your example code, and now this is working perfectly. It wasn't at first because I was running 11ty using --serve. Then I read the docs and added

eleventyConfig.setServerPassthroughCopyBehavior("copy");

and now pass-through copying works both with and without the --serve flag. I normally use --serve so I can preview my changes locally as I develop.

Thanks again :)