git commit fails, ultimately causing lost work

[papiro]

This seems like it might be related to #550, #561, possibly #387, possibly #75... I'm really not sure, but just want to share where my investigation has led me.

This exact issue has happened twice to me, and while I'm not sure what the cause of it is, we reasoned it was related to husky so uninstalled it. Turns out, husky just sets up pre-commit hooks for us which we define, and the only one we're using is: lint-staged I'm on a Windows, using MINGW64 git version 2.19.1.windows.1, Node v8.12.0. Here is the flow of events, including the error:

PPirault@Dev38 MINGW64 ~/Desktop/proj (feature/transfers-description)
$ git status
On branch feature/transfers-description
Your branch is up to date with 'origin/feature/transfers-description'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   app/components/account-selector/account-selector.spec.js
        modified:   app/components/checkbox/checkbox.spec.js
        new file:   app/components/input-field/input-field.spec.js
        modified:   app/components/selector/select.js
        renamed:    app/components/transfers-screen/transfers-screen.spec.js -> app/components/transfers-form/transfers-form.spec.js

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   app/components/checkbox/checkbox.spec.js
        modified:   app/components/input-field/input-field.js
        modified:   app/components/transfers-screen/transfers-screen.js
        modified:   bin/scaffold-component-test.js


PPirault@Dev38 MINGW64 ~/Desktop/proj (feature/transfers-description)
$ git commit
husky > pre-commit (node v8.12.0)
Stashing changes... [started]
Stashing changes... [failed]
→ Error: Command failed: git checkout-index -af
error: unable to create file README.md: Permission denied


Error: Command failed: git checkout-index -af
error: unable to create file README.md: Permission denied


husky > pre-commit hook failed (add --no-verify to bypass)

PPirault@Dev38 MINGW64 ~/Desktop/proj (feature/transfers-description)
$ git status
On branch feature/transfers-description
Your branch is up to date with 'origin/feature/transfers-description'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   app/components/account-selector/account-selector.spec.js
        modified:   app/components/checkbox/checkbox.spec.js
        new file:   app/components/input-field/input-field.spec.js
        modified:   app/components/selector/select.js
        renamed:    app/components/transfers-screen/transfers-screen.spec.js -> app/components/transfers-form/transfers-form.spec.js

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    README.md

As you can see, there is a git checkout-index -af (happening under lint-staged's hood?), which is clobbering the unstaged changes. At that point, the only way (that I know of) to recover the lost work is to comb through git fsck --unreachable and inspect each blob (or grep through them if I can remember a change) to find the changes. This happened intermittently and we have been using lint-staged in our project without issues for the last 2 months, with an average of 5 git commits per weekday. The two occurrences happened only within the last week. Please correct me if this is not lint-staged's issue.

[okonet]

Hmm, this looks like an issue with some permissions on your machine to me.

error: unable to create file README.md: Permission denied

Do you know why this might happen? Why there are this permissions problem occur?

To clarify: yes, lint-staged uses git checkout-index -af under the hood to remove unstated changes after "stashing" the index here

https://github.com/okonet/lint-staged/blob/adfc1d4a024ce731b22c2c26544d6b7ec3be1ecd/src/gitWorkflow.js#L78

Did something change on the permissions since last week?

[papiro]

Hmmm. The reason I'm suspicious about the permissions issue is because Windows users often have permissions issues with Git which aren't really issues. Usually it's because a file is open and Git can't modify it because there's a lock on it or something. The weird thing is that the README.md was not open on my machine, and I hadn't opened it for even months. Both times the linting failed on me, it complained about the README. I'm okay with closing this, as this issue seems to be an enigma, by nature of how many environmental factors are involved. It's difficult to test, and I can't even guarantee that I can reproduce it.

[okonet]

I think we could do a better job in case a command has failed but it will increase the complexity greatly so I’d like to keep things simple until it’s proven to be an issue for many users. Please feel free to close if you can’t repro it for now.

[unekinn]

My team is experiencing the same issue from time to time, also on Windows, using Git for Windows. Haven't found a way to reproduce it.

For us, it would be great if this stash-then-lint-then-checkout mechanism (and thus support for linting staged parts of a file only) could be disabled in configuration. No matter if the fault is in lint-staged or Git for Windows, risking loss of work seems like a high price to pay for the convenience of not having to fix linting errors in the entire file.

[okonet]

Feel free to propose a PR with such an option.

[guidetheorient]

if there are file changes on both working directory and staging area, sometimes, this issue will reproduce, in my case.

husky > pre-commit (node v10.14.2)
Stashing changes... [started]
Stashing changes... [completed]
Running linters... [started]
Running tasks for *.js [started]
eslint --fix [started]
eslint --fix [failed]
→
Running tasks for *.js [failed]
→
Running linters... [failed]
Updating stash... [started]
Updating stash... [skipped]
→ Skipping stash update since some tasks exited with errors
Restoring local changes... [started]
Restoring local changes... [failed]
→ Error: Command failed: git checkout-index -af
error: unable to unlink old 'maven-repository/com/umpay/mascloud/sdkclient/0.0.1-SNAPSHOT/sdkclient-0.0.1-20180802.022702-1.jar': Invalid argument
error: unable to create file www/src/common/audio/alert.mp3: Permission denied
error: unable to create file www/src/common/audio/alert.wav: Permission denied
error: unable to create file www/src/common/const/log-history.js: Permission denied
error: unable to create file www/src/common/echart-theme/v-charts-theme.js: Permission denied

[okonet]

This looks related:

https://serverfault.com/questions/253682/via-ssh-git-checkout-into-directory-the-user-doesnt-own-permission-denied

[FlorianWendelborn]

@guidetheorient can you please share your environment metadata? (e.g. operating system)

[papiro]

I should mention that I do git add -p/git commit -p almost exclusively, so there was usually a file's changes split between staged and unstaged in my case.

[tasos-ch]

I have the exact same issue.

It is difficult for me to reproduce it, as well.

I am only guessing here, but it may have something to do with file watchers running at the same time with lint-staged.

In particular, I am running lint-staged on an Angular project along with ng serve. The latter watches for file changes.

I think that the permission denied error may stem from the fact that the files are locked from another process (ng serve in this case).

[okonet]

Are you also on Windows? This seems to be a OS related problem but I’m not sure yet. I’ve been using this feature on several webpack-based projects with watchers running without any issues for months. Also there are lots of people using this version at the moment so it looks like there is something specific to the setup or os here.

[tasos-ch]

Yes, I am also on Windows (10, to be precise). And it happens randomly in my case, too.

The only reason I mention file watchers and ng serve is because in my case the file that is unable to be created is a SCSS file which is one of the files watched by ng serve. A similar issue is described in this StackOverflow question. Although that issue is about git checkout it seems related to this one.

Thanks a lot for the quick reply! I really appreciate it. Please, tell me if there is anything else that you would like to know.

[okonet]

Could you run an experiment and every time you commit a partially staged files to stop the watcher processes? If we could narrow down to the root cause it would definitely help with a possible solution.

[tasos-ch]

Yes, of course. I will do that and let you know if the problem persists.

[guidetheorient]

@FlorianWendelborn sorry for the late response. I'm on windows 10 home basic(version 1803), git version 2.14.2.windows.1, node v8.11.1

[silbinarywolf]

I'm seeing this issue on Windows as well, related to if a file is still open by another application and you try to run this code.

Is there any reason this code doesn't just use git stash to temporarily hold the information? ie.

git stash save "lint-staged-{uniqueidhere}"

Why does it execute the following?
lint-staged\src\gitWorkflow.js

// eslint-disable-next-line
async function gitStashSave(options) {
  debug('Stashing files...')
  // Save ref to the current index
  indexTree = await writeTree(options)
  // Add working copy changes to index
  await execGit(['add', '.'], options)
  // Save ref to the working copy index
  workingCopyTree = await writeTree(options)
  // Restore the current index
  await execGit(['read-tree', indexTree], options)
  // Remove all modifications
  await execGit(['checkout-index', '-af'], options)
  // await execGit(['clean', '-dfx'], options)
  debug('Done stashing files!')
  return [workingCopyTree, indexTree]
}

async function writeTree(options) {
  return execGit(['write-tree'], options)
}

[okonet]

There is a reason for that. It is required to support partially staged files. Feel free to experiment with stash since there is a good test coverage in place.

[okonet]

Ideally if someone could add a failing test for Windows that would be a great help already.

[silbinarywolf]

We've simply opted to just no longer use lint-staged for the time being as this "Stashing changes" feature has proven to be incredibly unsafe, especially since we're still waiting on PR #573 to be merged and published as well.

If this issue persists, I'll see if I can set aside more time to help with this but that might take a few months.

[Svish]

Experiencing this sporadically myself. Super scary. Thought it was related to having webpack watching or something, but then it happened while that wasn't running either, so...

Shortened log from latest incident below. Just cut out a lot of the "error: unable to create file ... " errors, as there were quite many. Most of the files listed weren't even staged or changed.

git -c diff.mnemonicprefix=false -c core.quotepath=false --no-optional-locks -c credential.helper= -c credential.helper="C:/Users/tbe/AppData/Local/ATLASS~1/SOURCE~1/GIT_EX~1/git-credential-manager.exe" commit -q -F C:\Users\tbe\AppData\Local\Temp\mswsakom.dgz
husky > pre-commit (node v11.2.0)
Stashing changes... [started]
Stashing changes... [completed]
Running linters... [started]
Running tasks for app/**/*.scss [started]
Running tasks for app/**/*.ts [started]
Running tasks for app/**/*.scss [skipped]
→ No staged files match app/**/*.scss
prettier --write [started]
prettier --write [completed]
tslint -c tslint.json --fix [started]
tslint -c tslint.json --fix [completed]
eslint -c .eslintrc --fix [started]
eslint -c .eslintrc --fix [completed]
git add [started]
git add [completed]
Running tasks for app/**/*.ts [completed]
Running linters... [completed]
Updating stash... [started]
Updating stash... [completed]
Restoring local changes... [started]
Restoring local changes... [failed]
→ Error: Command failed: git checkout-index -af
error: unable to create file README.md: Permission denied
error: unable to create file app/components/order/assets/downloadButton/controller.ts: Permission denied
error: unable to create file app/components/order/assets/rows/view.tag: Permission denied
error: unable to create file app/components/order/data/document/index.ts: Permission denied
error: unable to create file app/components/order/dataEditor/view.tag: Permission denied
error: unable to create file app/components/order/messages/create/view.tag: Permission denied
error: unable to create file app/components/orderAddress/editor/view.tag: Permission denied
error: unable to create file app/components/orderCollection/referenceEditor/view.tag: Permission denied
error: unable to create file app/components/orderCollection/sidebar/orders/controller.ts: Permission denied
error: unable to create file app/components/organization/editParameters/controller.ts: Permission denied
error: unable to create file app/components/organization/editParameters/editor/index.ts: Permission denied
error: unable to create file app/components/organization/search/view.tag: Permission denied
error: unable to create file app/components/organization/sidebar/departments/controller.ts: Permission denied
error: unable to create file app/components/organization/sidebar/departments/styles.scss: Permission denied
error: unable to create file app/components/organization/type/controller.ts: Permission denied
...
error: unable to create file app/services/orderData.ts: Permission denied
error: unable to create file app/services/url/authentication/departments.ts: Permission denied
error: unable to create file app/services/url/authentication/organizations.ts: Permission denied
error: unable to create file app/services/url/productCatalog/base.ts: Permission denied
error: unable to create file app/services/url/shopping/base.ts: Permission denied
error: unable to create file postcss.config.js: Permission denied
husky > pre-commit hook failed (add --no-verify to bypass)
Completed with errors, see above.

[papiro]

i should also mention that it would happen in our case without watchers running as well. Just simply on the lint-staged pre-commit hook, and failing on a file which wasn't even open or modified: in our case README.md. Although the nature of the error message indicates to me that it is a file-lock issue with that file. The error: unable to create file README.md: Permission denied is typical of that scenario.

[okonet]

Is anyone here would be able to create a failing test case at least? It sounds like the issue is with git on Windows but I have no idea what it might be and I’m not a windows user. So please everyone don’t rely on me with fixing this and take over this issue. I need your contributions here.

[egeriis]

It sounds like the issue is with git on Windows

We have macOS and Linux users experiencing this issue. We're effectively removing lint-staged from our workflow, because this has now caused us to lose many hours of work twice.

It's easily reproducible, as documented in #550, so the lack of acting is a bit worrying.

[unekinn]

It sounds like the issue is with git on Windows

We have macOS and Linux users experiencing this issue. We're effectively removing lint-staged from our workflow, because this has now caused us to lose many hours of work twice.

It's easily reproducible, as documented in #550, so the lack of acting is a bit worrying.

@egeriis I'm as frustrated as you with this, but to be fair @okonet has reached out for assistance multiple times about this issue, and none of us have stepped up. I also see that in the issue you linked, he specifically asked you for more info, and you haven't answered or yet provided a repo with a minimal example.

@okonet I'm sorry I haven't had the time to follow up on this. I was planning to at least try implementing the workaround mentioned earlier. I'll bump it to the top of my list after the next project deadline...

[egeriis]

@larskinn I wouldn’t go as far as to say I haven’t answered any of their questions, but I haven’t prioritized creating a repo to reproduce the issue, correct. I’ve thoroughly documented how to repro though.

I’m more annoyed at leaving such a glarring and major issue in a project of this scale in what’s supposedly a stable release. That’s my main critique and in my opinion inexcusable.

[okonet]

I’m more annoyed at leaving such a glarring and major issue in a project of this scale in what’s supposedly a stable release. That’s my main critique and in my opinion inexcusable.

I’m also annoyed by the assumption that I’m going to jump on the issue that doesn’t even have a reproducible case and fix it for you. It’s not my full time job and I’m not getting paid for any work I did in this repo.

Instead of telling me you’re removing it from your company’s workflow (I couldn’t care less tbh) you could ask the company to allocate time and developers to fix the issue. But I guess open source reached the point there i have to explain how it works.

I’m blocking this conversation since none of the recent comments moved us closer to the resolution nor the root cause.

If someone has something valuable to add, create a PR with a fix or a failing test case.

[okonet]

🎉 This issue has been resolved in version 9.5.0-beta.1 🎉

The release is available on:

• npm package (@beta dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[okonet]

🎉 This issue has been resolved in version 10.0.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀