[RRFC] Respect platform specific directories for config, data, cache, etc.

[natestemen]

This is a follow on since npm/cli#654 was closed.

Motivation ("The Why")

When using npm, it should not leave files/directories in the users' home directory if known environment variables are set, e.g. XDG_CONFIG_HOME.

How

Current Behaviour

npm creates a ~/.npm directory.

Desired Behaviour

npm should create the needed config/cache files/directories with respect to https://wiki.archlinux.org/title/XDG_Base_Directory.

[lousyd]

To clarify, the XDG Base Directory Specification does not require that environment variables be set. For example, it says, "$XDG_CONFIG_HOME defines the base directory relative to which user-specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used."

While npm could certainly choose to use existence of the environment variable as an indication of whether any particular person wants npm to observe XDG or not, it would not be a reliable signal. There are going to be lots of cases of people who don't set the variables and yet still expect software to "play well with others" by putting files in the XDG-expected locations. The converse situation -- people who really don't want you to observe XDG -- is surely.... rare.

Just follow XDG. Don't require the environment variables be set.

[ljharb]

In my experience maintaining nvm, users who do not set the environment variables but have any concept or expectation of XDG in their minds don't exist. I think only caring about XDG when the env vars are set is a perfectly reasonable approach.

[samhh]

@ljharb I disagree with you.

I care about XDG but only explicitly set them when I either want to use irregular locations or when I need access to them for some other purpose (e.g. ironically handling programs that support specifying the path but are ignorant of XDG). I assume the defaults will be respected per the specification.

A better way of framing it may be:

• Users who care about XDG might or might not have set the variables. They'll expect XDG-compliance by default.
• Users who don't care about XDG won't have set the variables and probably won't care where their data lives.

[ljharb]

The vast majority of users don't care about XDG at all, though. I explicitly and actively want the way things are, and I personally don't like the XDG spec at all. Currently I do that by simply not setting the XDG variables, and thus "almost every tool in existence" does nothing, since virtually every tool also does not respect XDG even when set.

[samhh]

You're absolutely right that the vast majority of users don't care about XDG. Equally I don't think they'd care if it became the default. The most impassioned voices on this issue will I think naturally be those like myself who want support. Do you not think you're going to be in the minority here @ljharb? I may be mistaken.

A lot - not all - of major Linux applications support the spec. Notably the usual reason for lack of support is backwards-compatibility concerns. I think it's a much different story on macOS.

To be clear I'd ideally like support without the need to explicitly opt-in; see here how noisy my shell config has become trying to opt everything in.

[WhyNotHugo]

Prior discussion on the topic has happened here: npm/cli#1451

A few others have voiced support for the issue, and there's also a PR, although in an abandoned state, for anyone willing to pick that up.

FWIW, any user who "does not care" about XDG won't be negatively affected by this. I very much doubt there are many users (if any) that actively want random applications to dump cache data right into their home directories.

[ljharb]

A large part of the js ecosystem describes the locations of npm’s cache folder, as well as a user-level $HOME/.npmrc, and changing these defaults would never be worth the churn.

[samhh]

How many significant tools are there that make these assumptions? In theory if it's just a few then it wouldn't be too hard to update them to support checking two locations, first XDG and then legacy.

[ljharb]

@samhh ive never met a tool that publicized XDG support. I’ve only ever heard of XDG (before these npm issues) because like 3 people in 10 years have requested that nvm support it, which it does only when the env vars are set.

[samhh]

fish references it in its documentation for example.

Which tools are significant and directly access npm files?

[natestemen]

In my view supporting XDG isn't about being able to publicize/advertise the fact that you support XDG, but rather because it makes using a terminal ever so slightly better, and more understandable.

@ljharb I don't understand your considerable pushback here. Would you mind elucidating what it actually is about supporting XDG that you don't like? As of now it seems like you don't want to support it because:

1. you don't believe it's useful to those who don't have the env vars set (which I think we'll agree will be the majority of users)
2. it's not widely supported (as of now, but my understanding is that this is moving towards XDG adoption)
3. you believe very few people care about this

I think the last point is probably true, but that doesn't indicate whether it will be helpful for them or not. I think most people have never heard of XDG to begin with.

[WhyNotHugo]

I'm looking at my dotfiles and see 21 apps that support the basedir spec (including git, neovim and systemd).

Chromium also has support, albeit imperfect and it mixes runtime files with config files (this is more of a bug than refusal to support it).

The only exceptions I see are npm, go, yarn, Firefox (just historical reasons and slow moving) and tools that cannot support it, like ssh and zsh. This last group cannot support the spec, since they're the ones who define environment variables and that'd be a chicken and egg.

That's still 21 supporting apps vs 4 that don't -- none of which refuse to support the standard, just haven't gotten around to it yet.

Honestly, I see no reason to NOT support this standard and keep saving cache files into home directories. I'm still curious why someone would want cache files in their $HOME rather than in the directory specially designated for that. Do you want all cache files in $HOME, or just npm's?

[ljharb]

@natestemen i have no objection to supporting it when the env vars are defined. My pushback is against the following points:

1. Not supporting xdg is never a bug, it’s merely a choice, because xdg isn’t a universal or mandated thing
2. When the env vars are not defined, the current/default/original behavior of a tool should be retained.

I personally prefer, and find it much cleaner, to have my $HOME filled with things from every too., and actively don’t want them hidden away in an obscure directory. Those who disagree can certainly set their XDG vars and ask for tools to support that.

[samhh]

@ljharb Isn't it more reasonable - on the basis of active support for either approach - to suggest that those who want everything in their home directory to just set their XDG vars accordingly?

[WhyNotHugo]

I'm happy to debate this further, but I see no actual arguments in favour of not supporting the standard so far.

• Behaving as other apps, and following a general direction makes usability easier for end users.
• It's actually possible to easily backup all config files since they're all in one place.
• It's simple to free up disk space since cache data is all in one place (I can't image how you'd do it without following this spec to keep things separate).
• Littering users' home adds noise. For example, I've 8 directories in my $HOME. Having random apps create their own directories, my own stuff would be drowned in a sea of noise.

I understand that some people may want cache data in their $HOME, but there may also be people who want temporary files in /usr. Just because there's a random desire for it doesn't seem like a valid argument for that.

[ljharb]

@samhh when creating a new tool, sure. this isn't a new tool, and "the way it's always been done in this tool" is the default.

[samhh]

@ljharb That's the default until there's a compelling reason to change. The reason here is that there's a specification a lot of major software follows that a not insignificant minority of users want, and which doesn't preclude any other behaviour including stuffing everything in ~ as we do now.

[ljharb]

@samhh "a lot" and "not insignificant minority" both remain completely unestablished.

I completely agree that it's totally fine, when XDG env vars are set, for npm to start respecting them - that's what I made nvm do, when the quite insignificant minority of its users who even knew about XDG requested the feature. The only part I'm really pushing back on is changing the default behavior in the absence of those env vars.

[WhyNotHugo]

Those env vars are seldom defined. They're mostly there for users who want some unusual behaviour, or for unique environments (like Flatpak, for example).

The spec is all about sane default, implementing the spec except the defaults is pointless.

I've listed four good reasons for the improvement. I still have to hear one against it, or any criticism of those reasons.

[ljharb]

1. Most apps in the js ecosystem do not in any way follow XDG, and that - not the shell ecosystem - is the more important idiom to look to
2. it’s already possible to easily back up config files when you need them, but for npm, you generally shouldn’t have user-level config anyways except your credentials - which you shouldn’t be backing up
3. npm has its own command to clear its cache, you never should nor have to muck with paths
4. “noise” is subjective. For me, it makes them readily discoverable, and i want them there, by default.

[toastal]

Most apps in the js ecosystem do not in any way follow XDG

Data? My own counter anecdote, I've seen a growing number of projects unaware of the spec happily merge changes to the config location if asked (more so if someone opens a merge request), including Yarn and Berry or how about Netlify's CLI, etc.. I would then actually assume a vast majority of Node applications are deployed and running on Linux (because most servers run Linux). And while not scientific, the StackOverflow surveys show Linux support is quite high among developers. With Linux, you would need to start counting the number of distros that don't follow the spec because of its ubiquity. Their users are used to it and expect it...

and i want them there, by default

If it's just subjective, then personally don't use or set any XDG environment variables and don't punish environments that want applications to follow their specifications, especially when the fallback is what you want. This is the whole point people are trying to make.

[ljharb]

@toastal as i've stated many times, those who want different behavior and set the XDG vars should be supported. I'm only expressing a strong opinion that the defaults do not change.

[Flare576]

Ad-hoc data:

• From Github Search: "$HOME/.npmrc"
  • 33 Wikis reference $HOME/.npmrc
    • Most are small/personal projects, but of note is Microsoft/typescript
  • 9,344 code results
• From Github Search: "$XDG_CONFIG_HOME"
  • 491 Wikis
  • 291K Code results
• From Homebrew (sorry @ljharb, I know it's unsupported but it's the data I know how to get)

install: 49,534 (30 days), 160,736 (90 days), 533,418 (365 days)

If NPM were to adopt a new set of default directories (which seems to be the primary point of contention), it would absolutely be a breaking change in my opinion. Even nvm users, despite managing their npm installs and cache via a separate folder structure, would be impacted - adopting XDG standards would be a "all or nothing" venture, so the $XDG_CONFIG_HOME/.n[pv]mrc change would be felt.

[obedm503]

All it has to do is use the XDG environment variables if defined.

I'm with @Flare576. As much as I would like XDG defaults, it would be a breaking change. I would guess most users are not even aware XDG exists much less care for it.

It could be done if npm included some sort of migration code that moved files around when upgrading the version (like a database migration). Unlike a dabase migration, npm devs do not control where npm runs. It may be impossible to know if the migration went well (other than the ton of question on SO if it fails).

[samhh]

It depends how you define a breaking change. It potentially wouldn't be breaking for npm itself if we continued to check for and use the legacy location if it exists, and defaulted to XDG (present or not) otherwise.

[WhyNotHugo]

The XDG spec refers to several directories, and I think the breakage cause by moving each one varies.

I was pointed to this issue because I want npm's cache to be in the proper directory. It bugs me to no end that each time I use npm, it'll dump it's cache into my home, along with all my other important stuff.

I don't think using the XDG location for cached files would be an "unexpected breakage" for anyone. Quite the contrary, I image most people prefer cache directories out of the way, and not as first-level directories inside their $HOME, pretty much the same way you don't want temporary files in $HOME, but rather in $TMPDIR.

Config files are a bit different, since people (and tools) are expected to want to interact with them more and care about its location. git handles this in an very good backwards-compatible way: If $XDG_CONFIG_HOME/git/config exists, it will use that, but if it doesn't, it falls back to the non-XDG location: $HOME/.gitconfig. (note on this: for newer files, they only support XDG, like $XDG_CONFIG_HOME/git/ignore).

[ljharb]

Assertions about what most people prefer are impossible to back up in either direction. I personally actively expect everything in my home dir, and I find XDG to be a really weird way to solve what isn't a real problem.

The way git handles things (in the git config case) is exactly what I think npm should do here.

[toastal]

The way git handles things (in the git config case) is exactly what I think npm should do here.

Just to be clear, git-config looks for configuration files in this order:

1. $(prefix)/etc/gitconfig
2. $XDG_CONFIG_HOME/git/config
3. ~/.gitconfig
4. $GIT_DIR/config
5. $GIT_DIR/config.worktree

[samhh]

^ Notably I'm 99% sure you don't need to explicitly set $XDG_CONFIG_HOME for it to search for configs there, it falls back to defaults.

[isaacs]

The unfortunate answer is that this proposal is complicated to do in a backwards-compatible way, and not valuable enough, for enough users, to justify either that complexity or breakage.

The good news is, you can make npm behave in an XDG-correct way relatively easily, by putting these lines in your ~/.bashrc or equivalent:

export npm_config_userconfig=$XDG_CONFIG_HOME/npm/config
export npm_config_cache=$XDG_CACHE_HOME/npm
# if you do this one, make sure to add $XDG_DATA_HOME/npm/bin to the $PATH,
# otherwise global installed executables won't be accessible on the cli.
export npm_config_prefix=$XDG_DATA_HOME/npm
export PATH=$PATH:$XDG_DATA_HOME/npm/bin

Customize those exports based on $XDG_BASE_DIR if you aren't setting the specific ..._HOME values, of course.

If someone wanted to distribute a build of npm that behaves this way, they could do so by including an ini-formatted file named npmrc in the root of the npm build that they distribute (ie, alongside npm's own package.json and other files), containing the following:

userconfig=${XDG_CONFIG_HOME}/npm/config
cache=${XDG_CACHE_HOME}/npm
prefix=${XDG_DATA_HOME}/npm

You can learn more about npm config and the many ways it can be customized by looking at https://docs.npmjs.com/cli/v7/using-npm/config or npm help 7 config.

If I had a time machine to go back and rewrite npm, this would of course be on my list of things to do. But if I'm being completely honest, if I had a time machine, I probably would have more interesting things to do than rewrite npm.

[WhyNotHugo]

The unfortunate answer is that this proposal is complicated to do in a backwards-compatible way, and not valuable enough, for enough users, to justify either that complexity or breakage.

So there's a few different issues that were closed and discussion was pointed here, and discussing them all together has made some of the items get lost.

I understand there's complexity due to backwards compatibility in moving config files and other files to a new location, however, my initial request was aimed only at cache files, not config files: npm creates its cache directory in my home every time it runs.

And I don't really even use npm myself: I use tools that themselves run npm, and as a side effect, litter my home.

I can't image that moving a cache directory in the right place can have compatibility issues. And, honestly, creating temporary files or cache files in a user's $HOME is almost a bug. I'm sure if it were temporary files (and not just cache) it would be considered a bug.

Can at least the cache be created out of the way, even if the rest of the spec is not implemented? I can't imagine there's any compelling reason to create a cache directory in $HOME.

[sebdanielsson]

Cluttering our home directories with cache files by default is just insane..

[probablykasper]

The good news is, you can make npm behave in an XDG-correct way relatively easily, by putting these lines in your ~/.bashrc or equivalent:

export npm_config_userconfig=$XDG_CONFIG_HOME/npm/config
export npm_config_cache=$XDG_CACHE_HOME/npm
# if you do this one, make sure to add $XDG_DATA_HOME/npm/bin to the $PATH,
# otherwise global installed executables won't be accessible on the cli.
export npm_config_prefix=$XDG_DATA_HOME/npm
export PATH=$PATH:$XDG_DATA_HOME/npm/bin

Was hoping to do this to remove some of the 18 items cluttering my home folder. Unfortunately it looks like setting npm_config_cache doesn't fully work due to npm/cli#3350

Can at least the cache be created out of the way, even if the rest of the spec is not implemented? I can't imagine there's any compelling reason to create a cache directory in $HOME.

I strongly agree with this, there should be no big migration issues from changing the cache location.

As for the config file, it would be backwards compatible if npm by default looks for .npmrc in the following order (similar approach to git, which @toastal mentioned). When creating the config file, the first one should be used:

• ~/Library/Application Support/npm/.npmrc
• ~/.npmrc

@isaacs Do you have any thoughts on this? Is it worthy of re-opening?

[thernstig]

@isaacs please re-visit the cache directory proposal to use XDG_CACHE_HOME (or falling back to the default ~/.cache/npm/ if XDG_CACHE_HOME is not set).

This is only breaking the cache and should not be considered backwards incompatible in any meaningful way.

There are literally thousands of Linux packages out there as an example, and if they all put their cache files in ~/.something we can all imagine how that would look.

I do not even consider this a matter of personal preference for what the npm project prefer, but it is more akin to good manners.

[isaacs]

@thernstig while I might be the one to blame, I'm no longer the one to convince about npm decisions, since I haven't worked on the project for a few years now.

[thernstig]

I wonder if we can open a new RFC regarding this? Or possibly ping some maintainer. Even bigger projects like git have started to follow the XDG spec in recent years.

[wesleytodd]

I believe the folks currently on the npm team are following most of these discussions, even if they are closed. There are no RFC calls anymore so I don't think opening a new issue will result in much action.

[thernstig]

I see, I would not have thought closed issues are given much thought, but if we are sure of this then I understand.

[danielpza]

Should this issue be reopened, then?

[ljharb]

@danielpza that would imply that the cli team is reconsidering it. I'm not sure why a few projects "started to follow the XDG spec in recent years" would change the previous reasoning.

[thernstig]

@ljharb there was no "previous reasoning" that was agreed on afaics. #389 (comment) is a good comment in that regard.

[ljharb]

@thernstig #389 (comment) is that, and matches previous discussions on RFC calls whenever it came up.

[thernstig]

@ljharb it has one upvote. git with arguably a factorial larger user base managed to do this change. I do not claim to know this is impossible, but even that referenced comments does not say it cannot be done.

[ljharb]

Whether it can be done isn’t being argued; of course it can. The question is whether it’s worth the churn and additional complexity in npm and a ton of userland tooling, which it’s been previously agreed by the CLI team multiple times it is not.

[codyrobbins]

I feel like I am in bizarro land reading the pushback against this change.