npm 5.7.1 pack/publish setting timestamp to epoch

[davidlehn]

I'm opening this issue because:

• npm is producing an incorrect install.

What's going wrong?

• npm pack with 5.7.1 on node 8.9.4 is setting bogus epoch file timestamps.
• This is not an issue in 5.6.0.
• These old timestamps are apparently a problem for some zip programs.
• This was brought to my attention via researching the cause of node-forge: Unhandled exception - ZIP does not support timestamps before 1980 digitalbazaar/forge#568
• Looks similar to a previous issue: npm5 beta 56: npm pack changes timestamps to 01/01/1970 on Windows 7 #16734
• I've been publishing packages with these bogus timestamps since upgrading to 5.7.1. Based on grabbing some random recent tarballs from npm homepage it looks like other people are publishing packages with the same timestamp issue.

How can the CLI team reproduce the problem?

• Pick a project, any project:
• npm pack
• tar ztvf xxx-x.y.z.tgz

-rw-rw-r-- 0/0            3487 1969-12-31 19:00 package/package.json
-rw-rw-r-- 0/0            4801 1969-12-31 19:00 package/CHANGELOG.md
...

Timestamps for all files are like new Date(0) with display adjusted for my timezone.

supporting information:

• npm -v prints: 5.7.1
• node -v prints: v8.9.4
• npm config get registry prints: https://registry.npmjs.org/
• Windows, OS X/macOS, or Linux?: Linux (Debian latest)

[marijnh]

This is pretty bad. See also acornjs/acorn#680

[charsleysa]

Just to add to the list tj/commander.js#777, according to npmjs.com package was downloaded 2,138,687 times in the last day (not specifically the broken version but has already started causing problems).

This is pretty much breaking all development and deployment systems using npm without package-lock.json (or npm-shrinkwrap.json).

[ofrobots]

Also googleapis/google-p12-pem#27.

[davidlehn]

Great Scott! Not the solution I had expected.

[zkat]

For the curious: We started using node-tar's noMtime option, which set all the timestamps to 0. The reason for this was so two separate npm pack calls done from the same commit -- even on separate computers, would be able to generate hash-identical tarballs. We didn't really expect that random software out there would have... pathological issues with old timestamps. We still want to preserve the feature, though, so picking an arbitrary date more recent than 1980 seemed like the way to go. Hopefully there's no more incompatibilities!

[pleunv]

@zkat We're now experiencing packaged files having their lastModified date overwritten with Oct 26th, 1985, which is messing with cache headers. However, it's a bit of an odd one:

We have 2 package.json files that each publish a package.

• 1 in the project root directory that publishes all source files and their deps (project/package.json)
• 1 in the dist/ directory that publishes only bundled dist output and strips all deps (project/dist/package.json), this is for direct consumption on a static file host.

The files in the first package have a correct lastModified date, the files originating from the second package.json files are having their lastModified dates set to Oct 26th, 1985.

Any idea what could be causing this behavior?

This is on npm v5.8.0.

[pleunv]

Some further investigation:

• The problem occurs on both Windows & OSX
• The problem does not occur on npm 5.6.0
• It's caused by the packaging/publishing, not the installing. For example, publishing with 5.6.0 and installing with 5.8.0 shows the correct lastModified dates, not the 1985. Publishing with 5.8.0 breaks it.

So for now I guess we'll stick to 5.6.0.

Would you like me to open a separate issue for this?

[zkat]

@pleunv: this is working as intended. We have every intention of keeping the consistent dates going forward because they enable reproducible builds for npm packages.

[pleunv]

I don't understand, how is this consistent? Depending on some variable I can't figure out (package.json location?) I'm getting all the lastModified timestamps of the installed files reset to epoch (or new equivalent) while the creation date is current. Something has to be affecting this somehow otherwise I would be seeing the same behavior when packaging from the project root, which I'm not.

Is there a specific scenario in which this is the expected outcome? Because that would already help me forward. Or, maybe easier: what is supposed to be the default behavior now? Should files be installed in node_modules with lastModified set to Oct 26th 1985?

[pleunv]

Or maybe I'm not explaining it properly, so another attempt 😄

left side: build output, creation & lastModified date = time of build
right side: after publish & install, creation date = time of build, lastModified = Oct 26th, 1985. Is this the new default behavior? Keep in mind that when using a different package.json (located one level up instead of in the dist/) the lastModified is once again equal to the time of build, which has me puzzled.

[ofrobots]

@pleunv what you're observing is indeed intended. Can you clarify how this default timestamp is causing you problems?

[pleunv]

The packages are pushed to a static file host, currently used in conjunction with a If-Modified-Since caching strategy. This however is based on the last-modified date, so this does not function at all with the current behavior since the server just always returns Oct 26th, 1985 10:15.

[zkat]

@pleunv the behavior now is that new packages are published with that timestamp, which means that's what you get when you extract, regardless of what version of npm you're extracting from -- what matters is what was used to pack it.

My only guess as to your inconsistency is that you're not using the npm version you think you're using to package the toplevel package (keep in mind that if you're doing this in a run-script, you might be getting an npm binary from a dependency -- try $ npx npm -v in both locations).

[pleunv]

Alright thanks, will give that a try. Guess I'll be looking for a workaround for my caching issue then :)

[zkat]

@pleunv might I recommend that you use a hash-based caching scheme, or, in lieu of that, have a run-script that touches files after an install. You can also use hook scripts to configure this in one place and only do it for individual packages as they get extracted.

[pleunv]

Hey @zkat, thanks for the tips, will take a look! The backend is not really under my control and the server is running Windows which both complicate things a little bit :)

I'm guessing a postinstall script that looks up all files and updates the mdate is probably my best bet for now.

I appreciate your time, I understand this repo is very busy :)

[wasigh]

Is there a CommandLine flag to disable his behaviour? We use NPM for internal used packages. And we also use this timestamp for caching purposes.
Also note that the node.js website has problems with the new behaviour:

For example: https://docs.npmjs.com/cli/build now states: "Last modified October 26, 1985"

[legodude17]

@wasigh as @zkat said:

might I recommend that you use a hash-based caching scheme, or, in lieu of that, have a run-script that touches files after an install. You can also use hook scripts to configure this in one place and only do it for individual packages as they get extracted.