Skip to content

Commit 61f7ed1

Browse files
halostatuetwpayne
halostatue
authored and
twpayne
committedSep 26, 2023
feat: Ignore AppleDouble files in externals
This is somewhere between a fix and a feature because it is a potential bug if an archive contains AppleDouble data, as such data would automatically be written out as parallel `._FILE` files and not reintegrated. With this change, we are still not reintegrating the data, but we are explicitly *ignoring* the data unless `EXTERNAL.archive.extractAppleDoubleFiles = true`. There are tools such as `https://github.com/SiviourBla/appledouble` that can theoretically reintegrate this using `copyfile(3)`. This checks that the extracted file name will be `._*` before trying to determine whether the file is an AppleDouble file. - This also adds `/.venv` directories to the `lint-whitespace` ignore list. I am exploring the use of Python venvs for using mkdocs. Resolves: #3220 Closes: #3221
·
v2.62.2v2.40.1
1 parent 66070dc commit 61f7ed1

File tree

4 files changed

+83
-24
lines changed

4 files changed

+83
-24
lines changed
 

‎assets/chezmoi.io/docs/reference/special-files-and-directories/chezmoiexternal-format.md

+28-21
Original file line numberDiff line numberDiff line change
@@ -19,27 +19,28 @@ will create them as regular directories.
1919

2020
Entries may have the following fields:
2121

22-
| Variable | Type | Default value | Description |
23-
| ----------------- | -------- | ------------- | ---------------------------------------------------------------- |
24-
| `type` | string | *none* | External type (`file`, `archive`, `archive-file`, or `git-repo`) |
25-
| `encrypted` | bool | `false` | Whether the external is encrypted |
26-
| `exact` | bool | `false` | Add `exact_` attribute to directories in archive |
27-
| `exclude` | []string | *none* | Patterns to exclude from archive |
28-
| `executable` | bool | `false` | Add `executable_` attribute to file |
29-
| `format` | string | *autodetect* | Format of archive |
30-
| `path` | string | *none* | Path to file in archive |
31-
| `include` | []string | *none* | Patterns to include from archive |
32-
| `refreshPeriod` | duration | `0` | Refresh period |
33-
| `stripComponents` | int | `0` | Number of leading directory components to strip from archives |
34-
| `url` | string | *none* | URL |
35-
| `checksum.sha256` | string | *none* | Expected SHA256 checksum of data |
36-
| `checksum.sha384` | string | *none* | Expected SHA384 checksum of data |
37-
| `checksum.sha512` | string | *none* | Expected SHA512 checksum of data |
38-
| `checksum.size` | int | *none* | Expected size of data |
39-
| `clone.args` | []string | *none* | Extra args to `git clone` |
40-
| `filter.command` | string | *none* | Command to filter contents |
41-
| `filter.args` | []string | *none* | Extra args to command to filter contents |
42-
| `pull.args` | []string | *none* | Extra args to `git pull` |
22+
| Variable | Type | Default value | Description |
23+
| ---------------------------- | -------- | ------------- | ---------------------------------------------------------------- |
24+
| `type` | string | *none* | External type (`file`, `archive`, `archive-file`, or `git-repo`) |
25+
| `encrypted` | bool | `false` | Whether the external is encrypted |
26+
| `exact` | bool | `false` | Add `exact_` attribute to directories in archive |
27+
| `exclude` | []string | *none* | Patterns to exclude from archive |
28+
| `executable` | bool | `false` | Add `executable_` attribute to file |
29+
| `format` | string | *autodetect* | Format of archive |
30+
| `path` | string | *none* | Path to file in archive |
31+
| `include` | []string | *none* | Patterns to include from archive |
32+
| `refreshPeriod` | duration | `0` | Refresh period |
33+
| `stripComponents` | int | `0` | Number of leading directory components to strip from archives |
34+
| `url` | string | *none* | URL |
35+
| `checksum.sha256` | string | *none* | Expected SHA256 checksum of data |
36+
| `checksum.sha384` | string | *none* | Expected SHA384 checksum of data |
37+
| `checksum.sha512` | string | *none* | Expected SHA512 checksum of data |
38+
| `checksum.size` | int | *none* | Expected size of data |
39+
| `clone.args` | []string | *none* | Extra args to `git clone` |
40+
| `filter.command` | string | *none* | Command to filter contents |
41+
| `filter.args` | []string | *none* | Extra args to command to filter contents |
42+
| `pull.args` | []string | *none* | Extra args to `git pull` |
43+
| `archive.extractAppleDouble` | bool | `false` | If `true`, AppleDouble files are extracted |
4344

4445
If any of the optional `checksum.sha256`, `checksum.sha384`, or
4546
`checksum.sha512` fields are set, chezmoi will verify that the downloaded data
@@ -66,6 +67,12 @@ The supported archive formats are `tar`, `tar.gz`, `tgz`, `tar.bz2`, `tbz2`,
6667
`xz`, `.tar.zst`, and `zip`. If `format` is not specified then chezmoi will
6768
guess the format using firstly the path of the URL and secondly its contents.
6869

70+
When `type` is `archive` or `archive-file`, the optional setting
71+
`archive.extractAppleDouble` controls whether
72+
[AppleDouble](https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats)
73+
files are extracted. It is `false` by default, so AppleDouble files will not
74+
be extracted.
75+
6976
The optional `include` and `exclude` fields are lists of patterns specify which
7077
archive members to include or exclude respectively. Patterns match paths in the
7178
archive, not the target state. chezmoi uses the following algorithm to

‎internal/chezmoi/sourcestate.go

+29-3
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,10 @@ var (
5555
)
5656
templateDirectiveRx = regexp.MustCompile(`(?m)^.*?chezmoi:template:(.*)$(?:\r?\n)?`)
5757
templateDirectiveKeyValuePairRx = regexp.MustCompile(`\s*(\S+)=("(?:[^"]|\\")*"|\S+)`)
58+
59+
// AppleDouble constants.
60+
appleDoubleMagicCode = []byte{0x00, 0x05, 0x16, 0x07}
61+
appleDoubleVersion = []byte{0x00, 0x02, 0x00, 0x00}
5862
)
5963

6064
// An External is an external source.
@@ -80,9 +84,12 @@ type External struct {
8084
Command string `json:"command" toml:"command" yaml:"command"`
8185
Args []string `json:"args" toml:"args" yaml:"args"`
8286
} `json:"filter" toml:"filter" yaml:"filter"`
83-
Format ArchiveFormat `json:"format" toml:"format" yaml:"format"`
84-
Include []string `json:"include" toml:"include" yaml:"include"`
85-
ArchivePath string `json:"path" toml:"path" yaml:"path"`
87+
Format ArchiveFormat `json:"format" toml:"format" yaml:"format"`
88+
Archive struct {
89+
ExtractAppleDoubleFiles bool `json:"extractAppleDoubleFiles" toml:"extractAppleDoubleFiles" yaml:"extractAppleDoubleFiles"`
90+
} `json:"archive" toml:"archive" yaml:"archive"`
91+
Include []string `json:"include" toml:"include" yaml:"include"`
92+
ArchivePath string `json:"path" toml:"path" yaml:"path"`
8693
Pull struct {
8794
Args []string `json:"args" toml:"args" yaml:"args"`
8895
} `json:"pull" toml:"pull" yaml:"pull"`
@@ -2338,6 +2345,11 @@ func (s *SourceState) readExternalArchive(
23382345
if err != nil {
23392346
return fmt.Errorf("%s: %w", name, err)
23402347
}
2348+
2349+
if !external.Archive.ExtractAppleDoubleFiles && isAppleDoubleFile(name, contents) {
2350+
return nil
2351+
}
2352+
23412353
lazyContents := newLazyContents(contents)
23422354
fileAttr := FileAttr{
23432355
TargetName: fileInfo.Name(),
@@ -2470,6 +2482,11 @@ func (s *SourceState) readExternalArchiveFile(
24702482
if err != nil {
24712483
return fmt.Errorf("%s: %w", name, err)
24722484
}
2485+
2486+
if !external.Archive.ExtractAppleDoubleFiles && isAppleDoubleFile(name, contents) {
2487+
return nil
2488+
}
2489+
24732490
lazyContents := newLazyContents(contents)
24742491
fileAttr := FileAttr{
24752492
TargetName: fileInfo.Name(),
@@ -2815,3 +2832,12 @@ func allEquivalentDirs(sourceStateEntries []SourceStateEntry) bool {
28152832
}
28162833
return true
28172834
}
2835+
2836+
// isAppleDoubleFile returns true if the file looks like and has the
2837+
// expected signature of an AppleDouble file.
2838+
func isAppleDoubleFile(name string, contents []byte) bool {
2839+
return strings.HasPrefix(path.Base(name), "._") &&
2840+
len(contents) >= 8 &&
2841+
bytes.Equal(appleDoubleMagicCode, contents[0:4]) &&
2842+
bytes.Equal(appleDoubleVersion, contents[4:8])
2843+
}

‎internal/cmd/testdata/scripts/external.txtar

+25
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
symlink archive/dir/symlink -> file
2+
[darwin] exec xattr -w -s io.chezmoi.test metadata-test archive/dir archive/dir/symlink archive/dir/file
23
exec tar czf www/archive.tar.gz archive
4+
# Force the collection of the Mac metadata for the home15/user test
5+
[darwin] exec tar czf www/archive-mac-metadata.tar.gz --mac-metadata archive
36

47
httpd www
58

@@ -116,6 +119,12 @@ chhome home14/user
116119
exec chezmoi apply
117120
cmp $HOME/.file golden/dir/file
118121

122+
[darwin] chhome home15/user
123+
124+
# test that chezmoi managed --include=externals lists external targets including AppleDouble files
125+
[darwin] exec chezmoi managed --include=externals
126+
[darwin] cmp stdout golden/managed-appledouble
127+
119128
-- archive/dir/file --
120129
# contents of dir/file
121130
-- golden/.file --
@@ -127,6 +136,14 @@ cmp $HOME/.file golden/dir/file
127136
.dir/dir
128137
.dir/dir/file
129138
.dir/dir/symlink
139+
-- golden/managed-appledouble --
140+
.dir
141+
.dir/._dir
142+
.dir/dir
143+
.dir/dir/._file
144+
.dir/dir/._symlink
145+
.dir/dir/file
146+
.dir/dir/symlink
130147
-- home/user/.local/share/chezmoi/.chezmoiexternal.toml --
131148
[".file"]
132149
type = "file"
@@ -168,6 +185,14 @@ cmp $HOME/.file golden/dir/file
168185
url = "{{ env "HTTPD_URL" }}/archive.tar.gz"
169186
path = "dir/file"
170187
stripComponents = 1
188+
-- home15/user/.dir/file --
189+
-- home15/user/.local/share/chezmoi/.chezmoiexternal.yaml --
190+
.dir:
191+
type: archive
192+
url: {{ env "HTTPD_URL" }}/archive-mac-metadata.tar.gz
193+
archive:
194+
extractAppleDoubleFiles: true
195+
stripComponents: 1
171196
-- home2/user/.local/share/chezmoi/.chezmoiexternal.toml --
172197
[".file"]
173198
type = "file"

‎internal/cmds/lint-whitespace/main.go

+1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ var (
2020
regexp.MustCompile(`\A\.vagrant\z`),
2121
regexp.MustCompile(`\A\.vscode\z`),
2222
regexp.MustCompile(`\Aassets/chezmoi\.io/site\z`),
23+
regexp.MustCompile(`/.venv\z`),
2324
regexp.MustCompile(`\Aassets/scripts/install\.ps1\z`),
2425
regexp.MustCompile(`\Acompletions/chezmoi\.ps1\z`),
2526
regexp.MustCompile(`\Adist\z`),

0 commit comments

Comments
 (0)
Please sign in to comment.