chezmoi v3 general discussion

[twpayne]

chezmoi v3 may or may not happen. If chezmoi v3 happens, then the goals of the v2->v3 transition will be similar to the v1->v2 transition (#654), namely:

• Fix user-facing inconsistencies, breaking backwards compatibility if necessary.
• Refactor chezmoi's internals to enable functionality that would be hard or impossible to implement with the v2 architecture.

v2 has held up well, and a lot of functionality has been added since its release two years ago.

Note that the v1->v2 internal rewrite (#654) took almost a year of calendar time, with the help of COVID-enforced isolation. Expect a similar calendar timescale for v2->v3, even though the changes should be smaller in scale.

So, that said, here are a few ideas that I have for chezmoi v3. Where there is a linked issue, please comment on the linked issue rather than this general discussion.

• Unified source state config file. Currently chezmoi's source state config is spread over multiple files (e.g. .chezmoiexternal, .chezmoiignore, .chezmoiremove, etc.). These can be unified into a single config file. Importantly, per-entry configuration (e.g. file and directory attributes and chezmoi:template directives) should still remain local to the file. Ref Allow .chezmoiscripts to live out of the home directory #2585 (comment) cc @Larandar).
• Internal tree-based representation of the target state. Bugs like chezmoi add mishandles cases where entry or parent is external #1574 and Apply fails if using externals with non-existent/ignored subfolder in chezmoiignore #2597 exist because chezmoi internally uses a flat map[path]Entry to represent the source state. This flat representation means that there are few/no checks to see if a parent directory exists before creating a child entry. This will also make features like Rewrite target paths #2273, Add an entire directory so re-add would include new files there in the future #2298, and Download externals concurrently #2670 easier to implement.
• Tidy up template functions. Notably moving away from minorminds/sprig (Replace github.com/masterminds/sprig template functions #2668) and removing old 1Password template functions (Tidy up 1Password template functions #2669 cc @halostatue).
• Enable install-over-SSH (Remote install dotfiles over ssh without installing chezmoi on the machine #1410). See the issue for details.

Link to all issues tagged v3.

If you have an new idea for chezmoi v3 then please do share. If there's a relevant existing issue, then please use the existing issue. If your idea can probably be implemented in chezmoi v2 then please open a feature request instead. Otherwise, please discuss here :)

[kenliu]

My suggestion for an overhaul in v3 is to move away from using file/directory name prefixes to manage source state attributes. I can see the benefits of encoding this information in the directory name as it makes these attributes more visible to the user, but it feels a bit awkward as you lose the 1:1 mapping of files from source to target and is a bit complex to understand when you start combining prefixes. I also found the dot_ convention to be confusing as a new chezmoi user and this makes migration to chezmoi a bit more difficult.

Perhaps this information could be stored as a .chezmoisourcestate file alongside the source files, since these attributes are often just metadata such file permissions.

[halostatue]

On one hand, I’ve sort of gotten used to the name / source state, although there are…some drawbacks. I’ve been playing with neovim some (I’m still on vim because there’s still no great neovim gui like MacVim) and in ~/.config/nvim, I have .stylua.toml, which makes it really easy to keep files formatted as long as I edit them in ~/.config/nvim, because the file needs to be private_dot_config/nvim/dot_stylua. I’ve been able to do ln -s dot_stylua .stylua in my chezmoi source directory and it works, but it is frustrating.

On the other hand, I don’t think that side-by-side files or a central set of files would be appropriate. There’s too much chance of drift.

On the gripping hand, the presences of the filename attributes could cause some files to exceed the 255-character filename limit on most filesystems, so an alternative might be appropriate. Possibly as an optional enhancement, rather than the default.

[bradenhilton]

@halostatue #2273 could help with the character limit. You could use a shortened version in source state and a longer version in the target state.

That said, there are still cases where you might run over the limit with deeply nested directories, so it's not a perfect solution.

[halostatue]

Should those rewrite files also potentially contain external state instead of using the source state attributes, as @kenliu suggested?

[kenliu]

I wanted to also highlight that the original post on this discussion thread mentions a potential "unified source state config file" which could open an opportunity to additionally overhaul source state attributes (unclear if what @twpayne had in mind includes redoing how file/directory name prefixes work).

[halostatue]

Overlays / layers / workspaces. The discussion in #2771 has presented a possibly novel way of using chezmoi that I think might be worth considering for the v3 upgrade in a more formal way than simply specifying -C altconfigfile.toml.

I haven’t completely thought this through, but it feels like we could come up with a way that chezmoi apply could maintain one source state file, but do so from multiple source paths. One thought is that we could add a layers key to chezmoi.$FORMAT which points to alternative configuration files that would be applied in succession:

[layers]
pre = [ "~/.config/chezmoi/pre.toml" ]
post = [ "~/.config/chezmoi/post.toml" ]

We could also say that there are no pre layers:

layers = [
  "~/.config/chezmoi/post1.toml",
  "~/.config/chezmoi/post2.toml",
]

Rules:

1. layers keys would not apply recursively. They are ignored in layer configurations. That is, any layers in pre.toml or post.toml are ignored.
2. Layers are data independent—the data from ~/.config/chezmoi/chezmoi.toml is not visible in ~/.config/chezmoi/post.toml or vice-versa. This is a chezmoi-managed way of running chezmoi apply && chezmoi -C ~/.config/chezmoi/post.toml that stores the state data in a single boltdb.

[Lockszmith-GH]

I like the idea as well. See my reply in #2771

[halostatue]

Thanks. I had been looking for #2574 and at least one other discussion / ticket on something similar to increase the depth of discussion for this sort of feature, but I wasn’t able to find them.

[Lockszmith-GH]

Probably because we don't have a community established term. This is either not highly requested, or people have all gone their own way to figure out a way to establish this solution.

I can share that I'm using this solution with a team of ~10 people - with me maintaining the base repo, and probably 1 or 2 (other than myself) who are actually using their personal chezmoi environments.

This allows me to push updates to shared local scripts fast with full control.
There are probably other methods (like shared application repositories, or stuff like NixOS and brew), but with chezmoi I have the most control while it being totally out of the way for my peers.

[Lockszmith-GH]

I've updated #2574 with a diagram in the hopes that it makes it easier to grok.

[halostatue]

Archive externals should be hardened against a possible repeat of the GitHub archive apocalypse.

On January 30, the GitHub blog carried a brief notice that the checksums of archives (such as tarballs) generated by the site had just changed. GitHub's engineers were seemingly unaware of the consequences of such a change — consequences that were immediately evident to anybody familiar with either packaging systems or Hyrum's law. Those checksums were widely depended on by build systems, which immediately broke when the change went live; the resulting impact of jawbones hitting the floor was heard worldwide. The change has been reverted for now, but it is worth looking at how GitHub managed to casually break vast numbers of build systems — and why this sort of change will almost certainly happen again.

This is would be helped if we had a way to resolve #2741 in code, but the correct solution (computing SHASUM over the uncompressed contents of the archive) is computationally expensive. Chezmoi would also require some protection against maliciously crafted archives (I believe that this responsibility could be pushed into the upstream dependencies used for archives).

[halostatue]

Dynamic chezmoi data extensions. I have a number of things which depend on the presence of executables via lookPath, and these happen across multiple files (in the case of the lists below, .gitconfig and .gitattributes).

{{- $delta := lookPath "delta" -}}
{{- $difftastic := lookPath "difft" -}}
{{- $gvim := lookPath "gvim" -}}
{{- $icdiff := lookPath "icdiff" -}}
{{- $ksdiff := lookPath "ksdiff" -}}
{{- $mvim := lookPath "mvim" -}}
{{- $vim := lookPath "vim" -}}
{{- $exiftool := lookPath "exiftool" -}}
{{- $pdfinfo := lookPath "pdfinfo" -}}
{{- $pdftotext := lookPath "pdftotext" -}}
{{- $djvutxt := lookPath "djvutxt" -}}
{{- $odt2txt := lookPath "odt2txt" -}}

It would be useful to be able to have something like .chezmoidata.extended.$FORMAT.tmpl which is merged into .chezmoi.data after chezmoi.$FORMAT and .chemzoidata.$FORMAT, so that I could execute these lookPath calls once during a chezmoi apply.

Transformed, this would look something like:

[data.in_path]
delta = {{ lookPath "delta" }}
difftastic = {{ lookPath "difft" }}
gvim = {{ lookPath "gvim" }}
icdiff = {{ lookPath "icdiff" }}
ksdiff = {{ lookPath "ksdiff" }}
mvim = {{ lookPath "mvim" }}
vim = {{ lookPath "vim" }}
exiftool = {{ lookPath "exiftool" }}
pdfinfo = {{ lookPath "pdfinfo" }}
pdftotext = {{ lookPath "pdftotext" }}
djvutxt = {{ lookPath "djvutxt" }}
odt2txt = {{ lookPath "odt2txt" }}

This could also be restricted to be under a specific configuration key (e.g., data.extended). There was a similar request for cached templates a few months ago, but it was deferred.

[2-4601]

Would love to see something like this because I'm also a heavy user of lookPath (e.g. I have to use lookPath to create Bash shebangs in scripts because Termux has issues with traditional paths).

[twpayne]

chezmoi already caches the output of lookPath (although I admit that this is not documented, so you don't actually need to do this. You can sprinkle as many lookpath "command" calls around your templates as you want and chezmoi will only actually look for command once.

[twpayne]

I added documentation in 841b947.

[halostatue]

The purpose here isn’t for caching, but to avoid putting all of these in templates where the configuration (e.g., ~/.gitconfig or an include) uses these. See https://github.com/halostatue/dotfiles/blob/main/home/private_dot_config/git/diff-merge.tmpl#L1-L47 for how it looks just for the setup. I don’t particularly care about the tiny milliseconds it takes even if lookPath weren’t cached; I care about cleaning up my templates.

Yes, it means that if I brew uninstall delta without remembering to do chezmoi apply, my git configuration will be incorrect, but that’s a risk I would take if I didn’t guard these tools with lookPath.

[halostatue]

This has been suggested again with #3526. As much as I would like to see this, it's hard because chezmoidata extends the data in chezmoi, but templates need the data that's in chezmoi to execute them.

[twpayne]

I will not add localization. The tech world uses English, the docs use simple grammar which Google Translate handles well.

[triallax]

From what I understand this isn't exactly trivial, but #2852 (or anything that implements opt-in as opposed to opt-out applying of dotfiles) would be a significant quality-of-life improvement for me.

[halostatue]

Template function deprecations and removals. We would need to add ways of reporting on the use of deprecated functions (for the first few versions that a template function is deprecated, the message should only be output every Xth run or Y days, whichever is less; this would mean additional state to be tracked somehow) as well as whether there are replacements or alternatives (and document those).

I was looking at the documentation for mkdocs material and saw a way to add "version" chips to template function pages, but it requires changes and other stuff that is still being researched slowly.

[halostatue]

The thought here for a page would be to indicate the version in which it was introduced. Not important, but nice.

The thought on reporting is that there should be a log message (maybe a summary thing at the end of chezmoi apply that gets shown every two weeks) that indicates which template functions are in use (and preferably where, but that may be asking a bit much) that are deprecated.

[bradenhilton]

The thought here for a page would be to indicate the version in which it was introduced. Not important, but nice.

Yeah, that's what I meant. I vaguely remember the reason given was as I stated above, but my memory of this is not perfect, and I can't seem to find where it was discussed.

[twpayne]

Not sure if it was discussed, but chezmoi's security policy states that only the most recent version is supported here.

The tl;dr on this reasoning is:

• chezmoi goes to moderate to high effort to maintain backwards compatibility for users at chezmoi's API level.
• chezmoi's API level for users is the contents of the source state and its effect on the target state.
• Maintaining multiple stable releases, e.g. backporting security fixes to previous stable versions, is a huge effort that benefits at most a handful of users.
• Therefore, the effort/value ratio of maintaining old versions simply isn't worth it, especially as chezmoi is developed by and maintained by volunteers in their spare time.

[halostatue]

I am not at all suggesting maintaining old versions, merely that it would be nice to have in the documentation something that says "this function has existed since version 2.3.0". It's not important, but it would let someone know whether they should upgrade (the answer is yes, but some people don't upgrade working software until they see a reason for it).

[twpayne]

Ah, having "this has existed since version ..." indeed makes total sense. Thank you!

[halostatue]

I am unconvinced of the value in this, but #3528 suggests aligning more with XDG for the chezmoi state file, and I believe things like the externals httpcache directory should also be moved if we are reviewing these.

Similarly, Windows and macOS native locations for these should be considered (~/Library/Caches and/or ~/Library/Application Support, possibly other locations).

[62832]

As I said in the linked PR, I'm still willing to help make it happen both for the httpcache directory and the native locations for Windows/macOS.

[twpayne]

As I said in the linked PR, I'm still willing to help make it happen both for the httpcache directory and the native locations for Windows/macOS.

@62832 please do open a PR for this.

[62832]

Hmm, is httpcache already saved under the cache dir?

[62832]

Also, regarding the paths for Windows/macOS, it seems like the adrg/xdg library provides some reasonable defaults for each of them. It might be worth migrating to this library as opposed to the current in-house solution.

EDIT: #3535

[VorpalBlade]

About your suggestions of changes to .chezmoiignore: I have multiple such files local to each directory. Would you propose that I would have to switch to a single one in the root? I find that approach awkward.

Also on the topic of ignoring files: I find the logic is backwards to the way I think. E.g. "this file should only be applied on servers that aren't work related" means that I actually have to write "if not (server and not work)". Yes it can be "simplified" using DeMorgan's laws to "if not server or work", but now it is even further from how I think about it. I don't have a good suggestion for how it should work instead though.

Another issue I have with chezmoi is lack of failure handling in templates. E.g., if looking up a password fails (maybe I'm applying the config over ssh), it is an error. I can't provide a default fallback. Nor can I say "if you fail, keep the current value in the system config file" (that might be hard to do in general, though chezmoi_modify_manager can do it for ini files).

[VorpalBlade]

I would like a way for modify scripts to interact with the diff process, specifically the "overwrite/keep" question. This is the biggest pain point for chezmoi_modify_manager currently.

This would require chezmoi to also provide info to the modify script on what the system state looked like at last apply so I can tell where the changes come from. (Or something equivalent I guess? Not sure how chezmoi itself handles it currently.) And a way for me to signal to chezmoi that we need to prompt the user.

Maybe there should be a difference between modify script and "advanced plugin" that has a more complex interaction protocol, but allows things like this? As long as I can write my code in Rust and not Go I'm game for something like that.

[halostatue]

This would require a parallel file as described for path rewriting, but being able to declare multiple files as related to one another in a group might be useful. Suggested as "package name" management in #3698.