[RFC] cleaning distDir on next build

[timneutkens]

Feature request

Is your feature request related to a problem? Please describe.

Currently Next.js will keep the distDir (by default .next) when running next build because some deployment strategies involve running next build while the production application is running. However in most cases you actually want to start with a clean directory. For example when running next export after the build and Next.js having been in development mode before the build, this would cause HMR files to be written to the export out directory as outlined in #5554.

Describe the solution you'd like

One solution is cleaning distDir by default and providing an option to opt-out of this behavior. I'd like to get some feedback on this though.

[Janpot]

Maybe only clean the HMR files?

[timneutkens]

It's not just HMR files. It's static files in .next/static/development/pages. It's the development runtime in .next/static/webpack. If you run next build multiple times you'd have multiple .next/static/<buildid>/pages folders etc.

[Janpot]

Do you know what's the rationale behind running next build on a project that is already started in production? Are they using nodemon or something to automatically restart their server when it updates? And if they are, then doesn't that mean they also suffer from old build files leaking on their file system?

If that's the case, I guess they (maybe unknowningly) suffer from the same problem, and need some sort of cleaning solution as well. (although it rather sounds like an antipattern to me)

[timneutkens]

Probably a good idea to cc @DullReferenceException @alexindigo @ztanner

[alexindigo]

@timneutkens Sounds like you're proposing to auto clean up distDir prior running export step? Or you just talking about build step?

[timneutkens]

next build would start with rm -rf .next basically. export needs the result of next build

[Janpot]

could it be an idea to clean at the end of the build? i.e.

1. store timestamp startTime
2. run build
3. remove all files that were last modified before startTime

Edit:
or keep a list of files in the .next folder before building and compare with build output

[timneutkens]

Sounds like too much complexity imo.
The point is that in certain builds you want to keep the old files. The main question I'm asking is if it would break the deployment process of users like Trulia, I doubt it will, just making sure.

[Janpot]

Sounds like too much complexity imo.

Completely agree. Just mentioning it in case you'd favor a bit of complexity over adding another option, it's always a trade off 🙂.

[alexindigo]

@timneutkens if it's just build then we're good :)

[iamSaikatPaul]

I use this technique. It works fine for me

[timneutkens]

Right exactly what I thought 💯

[ztanner]

+1, this makes total sense and would not break things for us.

[sergiodxa]

Why not release with opt-in flag (let's say --clean-dist) and an extra flag to opt-out (--no-clean-dist) not doing anything and then in the next release remove the opt-in and make clean distDir default making opt-out required?

That way if you need to keep it you have a major version to add the --no-clean and be prepared to update to next version. The build command could warn about the usage without --no-clean-dist in next version in order to let people know they need to either add it or add --clean-dist flag.

[timneutkens]

We're already going to have a major version with current changes, hence why I created this rfc.

[iamstarkov]

next build would start with rm -rf .next basically. export needs the result of next build

it sounds good to me and that's exactly what I ended up doing for my blog

[ha-ku]

Are there any further progress about this RFC? I still find next keeping the old version of static files after next build is performed.

[AndrewRayCode]

I'm not sure if this is from the same issue, but when developing locally, then running

next build
next export

I get a never-ending list of:

6557f073caf41dee7851.hot-update.json                                                                 100%   43     0.8KB/s   00:00
1.7c098274f66f6c6c3991.hot-update.js                                                                 100% 9111    16.5KB/s   00:00
1.b3347dcff2efeb29cec9.hot-update.js                                                                 100%   28KB 109.6KB/s   00:00
1.ab3250b2c5eb9527f594.hot-update.js.map                                                             100% 8396    94.6KB/s   00:00
1.5263a20a0a6cd904f145.hot-update.js                                                                 100% 9002    18.1KB/s   00:00
1.bdae25d1623ba611a08d.hot-update.js                                                                 100% 6427    13.3KB/s   00:00
1.2aedeef3a474eed07ad8.hot-update.js                                                                 100%   10KB  38.5KB/s   00:00
45a164be94cbd52deeec.hot-update.json                                                                 100%   43     0.1KB/s   00:00
1.a7bb361a0404e13b5189.hot-update.js                                                                 100%   11KB  75.5KB/s   00:00
1.58069868f21c29300c39.hot-update.js.map                                                             100% 8571    70.4KB/s   00:00
deb3ac7eebca8574bf7d.hot-update.json                                                                 100%   43     0.4KB/s   00:00

Adding rm -r out .next in scripts in package.json for build fixes it, although I'm worried about what, if anything, I'm losing by nuking the .next folder.

Also for context, i'm developing locally, and running npm export multiple times, then scp'ing the out directory to my server.

[MuhammedAO]

Hi i seem to be experiencing this same issue. my question is would there be a problem if just add the .next folder to my git ignore because it really messing up my files when i want to add files and make commits.. i get a whole bunch of webpack stuff screaming in my face.

[stovmascript]

@MuhammedAO Definitely do that. Build output files should not be in your repo. They get recreated anytime you build. Unless there's a specific reason to have such files checked into source control, this principle should apply to any project.

[dpyzo0o]

What's the status of this RFC?

[cmsbased]

I cannot upload static site to CDN due to huge files that are added to out folder if I do any local npm run dev before export. rm -r out .next fixes the problem but does not look like an optimal solution.

[ArbyLeon]

@timneutkens any updates on this?

[timneutkens]

This is planned for Next.js 10.

[kvedantmahajan]

Is this shipped in next10? @timneutkens Where can I find it as it not mentioned in https://nextjs.org/blog/next-10

[timneutkens]

It has not, we couldn't get it in before the conference!

[freynolds]

I'm having a similar issue, Next.JS hosted in AWS EC2, deployed with PM2, npm ci && npm run build && pm2 reload ecosystem.config.js --env production, it creates a lot of files, that isn't really an issue instead that sentry uploads them each time (all of them) on build. It makes things slow as hell and I wonder if sentry is working ok because of this

[stovmascript]

The issue with Sentry uploading previous build files is tracked here: #14076

[alex-drocks]

Hello guys, I'm stepping in with the same concern with next build && next export producing all the extra build files in NextJs 10. Is the plan to release an option for cleanup before new export command still up?

Right now on Windows I do this using Gulp:

package.json script:

"export-prod": "gulp preBuildCleanDirs && next build && next export"

gulpfile.js:

const gulp = require("gulp");
const clean = require("gulp-clean");

function cleanNextDir() {
  return gulp.src(".next/*", {read: false})
    .pipe(clean({force: true}));
}

function cleanOutDir() {
  return gulp.src("out/*", {read: false})
    .pipe(clean({force: true}));
}


exports.preBuildCleanDirs = gulp.series(
  cleanNextDir,
  cleanOutDir,
);

[ArnaudValensi]

I wonder too.

[alex-drocks]

see his better solution: #6009 (comment)

[JuanmaMenendez]

Run npx rimraf .next will delete the .next folder.

This will work:

  "scripts": {
    "dev": "npm run clean && next dev",
    "build": "npm run clean && next build",
    "start": "next start",
    "test": "npm run clean && jest",
    "clean": "npx rimraf .next"
  },

[alex-drocks]

Nice one you completely beat my solution. I was not aware of npx rimraf.

I used this on windows 10 and it works nicely to delete the .next and out folders:

  "scripts": {
    "dev": "next dev",
    "build": "npm run clean && next build && next export",
    "start": "next start",
    "clean": "npx rimraf .next out"
    },

[SeinopSys]

For my purposes it would have been nice to have the previous build available until the new one is finished, then I could restart the app server and then purge the old build from the server. I ultimately decided against maintaining my own build process and just moved to Vercel but an ever-growing build output directory is not ideal for those looking to self-host in general.

[francoismassart]

If your Next.js app running on production generates a lot of static assets you might have a long build time, if so cleaning dir prior to build would create downtimes on your site (404)...

[francoismassart]

Keeping old versions of builds might be useful when visitors are browsing your site between 2 builds ?
Their (outdated) build might refer to old chunks and erasing them too early might cause errors for some visitors during their current browsing session.

Maybe the best of both world would be to keep the last x versions of the build or apply the same logic based on the creation date...

I am pretty new to Next.js so I might be mistaken.

[polarathene]

As a new user to next.js, I got confused with the output of /out with next export.

It turned out it's because I did a next build, and also ran a next start prior to the next export as an error advised for using next/image. That copied over a bunch of unnecessary files from next start output which was just intended to provide an image API for local images so I could export a static build.

I'd need to duplicate the project I think to run a separate instance, or use something like Docker as a workaround. Alternatively, if next export was a bit smarter and knew what could be stripped perhaps that'd work?

My initial confusion was due to files like main.js being 4MB with a content hash version less than 20KB in the /out dir tree, along with unnecessary files like amp.js which had no relevance to my basic site.

At the same time, the HTML output for the page has plenty of content referencing JS scripts, but in this case the page built is basic enough that no JS is needed for it. That's presumably out of scope for the exporter to know and could be handled by the developer through post-process tooling.

[danoc]

Looks like this has shipped in Next.js 11: https://nextjs.org/docs/upgrading#cleaning-the-distdir-is-now-a-default 🎉

[magnusriga]

Sorry for bumping this old thread, but I am seeing the issue discussed here when running npm run build. Has the default behavior of deleting the old .next folder changed?

We use Contentful, which triggers a rebuild when content changes (via a web-hook). Unfortunately, when the old .next folder is not deleted automatically at npm run build, we see stale Contentful content after rebuilding.

Anything I am missing here?

---

EDIT:

It seems my problem was the .next/cache folder being stale (the cache subfolder), which Next does not delete automatically.

That is an issue though, because data fetches to CDNs like Contentful will then be stale even after re-building the whole site.