Make lint-staged more compatible with git sparse checkout feature

[chengcyber]

Description

Now, lint-staged has compatibility issue when developing in a monorepo with git sparse checkout feature enabled. It throws errors for loading config from package.json files out side of sparse checkout folders.

Steps to reproduce

I created a repo to show the idea at https://github.com/chengcyber/lint-staged-sparse-issue

NOTE: make sure the git version supports "git sparse checkout" and "--sparse" parameter in "git clone". The git version in my end is 2.43.0. I believe >=2.25.0 should be fine.

# clone this repo with git sparse feature
git clone --sparse git@github.com:chengcyber/lint-staged-sparse-issue.git

# go to the repo folder
cd lint-staged-sparse-issue

# sparse checkout for app1
git sparse-checkout add apps/app1

# ... real develop for app1 ...
touch apps/app1/a.txt
git add apps/app1/a.txt

# Run lint-staged
npx lint-staged

However, npx lint-staged complains:

npx lint-staged
[Error: ENOENT: no such file or directory, open '/private/tmp/lint-staged-sparse-issue/apps/app2/package.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/private/tmp/lint-staged-sparse-issue/apps/app2/package.json'
}
✖ No valid configuration found.

Debug Logs

expand to view

lint-staged:bin Running `lint-staged@15.2.1` on Node.js v16.20.2 (darwin) +0ms
  lint-staged:bin Options parsed from command-line: {
  allowEmpty: false,
  concurrent: true,
  configPath: undefined,
  cwd: undefined,
  debug: true,
  diff: undefined,
  diffFilter: undefined,
  maxArgLength: undefined,
  quiet: false,
  relative: false,
  shell: false,
  stash: true,
  hidePartiallyStaged: true,
  verbose: false
} +0ms
  lint-staged:validateOptions Validating options... +0ms
  lint-staged:validateOptions Validated options! +0ms
  lint-staged Unset GIT_LITERAL_PATHSPECS (was `undefined`) +0ms
  lint-staged:runAll Running all linter scripts... +0ms
  lint-staged:runAll Using working directory `/private/tmp/lint-staged-sparse-issue` +0ms
  lint-staged:resolveGitRepo Resolving git repo from `/private/tmp/lint-staged-sparse-issue` +0ms
  lint-staged:resolveGitRepo Unset GIT_DIR (was `undefined`) +0ms
  lint-staged:resolveGitRepo Unset GIT_WORK_TREE (was `undefined`) +0ms
  lint-staged:execGit Running git command [ 'rev-parse', '--show-prefix' ] +0ms
  lint-staged:resolveGitRepo Resolved git directory to be `/private/tmp/lint-staged-sparse-issue` +33ms
  lint-staged:resolveGitRepo Resolved git config directory to be `/private/tmp/lint-staged-sparse-issue/.git` +0ms
  lint-staged:execGit Running git command [ 'log', '-1' ] +33ms
  lint-staged:execGit Running git command [ 'diff', '--name-only', '-z', '--diff-filter=ACMR', '--staged' ] +26ms
  lint-staged:runAll Loaded list of staged files in git:
  lint-staged:runAll [ '/private/tmp/lint-staged-sparse-issue/apps/app1/a.txt' ] +447ms
  lint-staged:searchConfigs Searching for configuration files... +0ms
  lint-staged:execGit Running git command [ 'ls-files', '-z', '--full-name' ] +388ms
  lint-staged:execGit Running git command [ 'ls-files', '-z', '--full-name', '--others', '--exclude-standard' ] +3ms
  lint-staged:searchConfigs Found possible config files: [
  '/private/tmp/lint-staged-sparse-issue/apps/app1/package.json',
  '/private/tmp/lint-staged-sparse-issue/apps/app2/package.json',
  '/private/tmp/lint-staged-sparse-issue/package.json'
] +54ms
  lint-staged:loadConfig Loading configuration from `/private/tmp/lint-staged-sparse-issue/apps/app1/package.json`... +0ms
  lint-staged:loadConfig Loading configuration from `/private/tmp/lint-staged-sparse-issue/apps/app2/package.json`... +0ms
  lint-staged:loadConfig Loading configuration from `/private/tmp/lint-staged-sparse-issue/package.json`... +1ms
  lint-staged:loadConfig Failed to load configuration! +0ms
[Error: ENOENT: no such file or directory, open '/private/tmp/lint-staged-sparse-issue/apps/app2/package.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/private/tmp/lint-staged-sparse-issue/apps/app2/package.json'
}
  lint-staged:loadConfig Successfully loaded config from `/private/tmp/lint-staged-sparse-issue/package.json`:
  lint-staged:loadConfig null +0ms
  lint-staged:loadConfig Successfully loaded config from `/private/tmp/lint-staged-sparse-issue/apps/app1/package.json`:
  lint-staged:loadConfig null +1ms
  lint-staged:searchConfigs Could not find config files inside "/private/tmp/lint-staged-sparse-issue" +2ms
  lint-staged:loadConfig Searching for configuration from `/private/tmp/lint-staged-sparse-issue`... +0ms
  lint-staged:searchConfigs Could not find parent configuration files from "/private/tmp/lint-staged-sparse-issue" +15ms
  lint-staged:searchConfigs Found 0 config files +0ms

Environment

• OS: MacOS Sonoma
• Node.js: 16
• lint-staged: 15.2.1 (latest)

[chengcyber]

A possible solution to this issue could be reading the file content of missing package.json from git as well, since the existence of those package.json are read from git.

It could happen at this line of code:

lint-staged/lib/loadConfig.js

Line 83 in 999dcce

const content = loader.length > 1 ? await readFile(filepath) : undefined

I would like to put my psedocode here to show the idea:

BEFORE:

const loadConfigByExt = async (filepath) => {

  // ...

  const content = await readFile(filepath);

  // ...
}

AFTER:

const loadConfigByExt = async (filepath) => {

  // ...

  const content = fs.exists(filepath) ? await readFile(filepath) : await readFileFromGit(filepath) ;

  // ...
}

function readFileFromGit(filepath) {
   const relativeFilepath = getRelativeFilepathSomehow(filepath):
   const content = exec(`git show HEAD:${relativeFilepath}`);
   return content;
}

[iiroj]

Related issue with a proposed solution that doesn't require file system operations: #1383

Thanks! I will provide a fix now that there's multiple issues.

[iiroj]

@chengcyber would you mind testing the linked PR in your setup?

[chengcyber]

@iiroj Sure. I am glad to test it. Is there a specific prerelease version I can run locally?

[iiroj]

Sorry no, easiest is to copy/paste the changes into your node_modules/lint-staged/lib/ as there is no build step or anything. 👍

[chengcyber]

I tested it locally and yes it has been fixed and no more errors. The debug log is as follows:

DEBUG=* testlint-staged
  lint-staged:bin Running `lint-staged@15.2.1` on Node.js v16.20.2 (darwin) +0ms
  lint-staged:bin Options parsed from command-line: {
  allowEmpty: false,
  concurrent: true,
  configPath: undefined,
  cwd: undefined,
  debug: false,
  diff: undefined,
  diffFilter: undefined,
  maxArgLength: undefined,
  quiet: false,
  relative: false,
  shell: false,
  stash: true,
  hidePartiallyStaged: true,
  verbose: false
} +1ms
  lint-staged:validateOptions Validating options... +0ms
  lint-staged:validateOptions Validated options! +0ms
  lint-staged Unset GIT_LITERAL_PATHSPECS (was `undefined`) +0ms
  lint-staged:runAll Running all linter scripts... +0ms
  lint-staged:runAll Using working directory `/Users/admin/code/lint-staged-sparse-issue` +0ms
  lint-staged:resolveGitRepo Resolving git repo from `/Users/admin/code/lint-staged-sparse-issue` +0ms
  lint-staged:resolveGitRepo Unset GIT_DIR (was `undefined`) +0ms
  lint-staged:resolveGitRepo Unset GIT_WORK_TREE (was `undefined`) +0ms
  lint-staged:execGit Running git command [ 'rev-parse', '--show-prefix' ] +0ms
  lint-staged:resolveGitRepo Resolved git directory to be `/Users/admin/code/lint-staged-sparse-issue` +18ms
  lint-staged:resolveGitRepo Resolved git config directory to be `/Users/admin/code/lint-staged-sparse-issue/.git` +0ms
  lint-staged:execGit Running git command [ 'log', '-1' ] +19ms
  lint-staged:execGit Running git command [ 'diff', '--name-only', '-z', '--diff-filter=ACMR', '--staged' ] +23ms
  lint-staged:runAll Loaded list of staged files in git:
  lint-staged:runAll [ '/Users/admin/code/lint-staged-sparse-issue/apps/app1/a.txt' ] +76ms
  lint-staged:searchConfigs Searching for configuration files... +0ms
  lint-staged:execGit Running git command [ 'ls-files', '-z', '--full-name', '-t' ] +34ms
  lint-staged:execGit Running git command [
  'ls-files',
  '-z',
  '--full-name',
  '-t',
  '--others',
  '--exclude-standard'
] +3ms
  lint-staged:searchConfigs Found possible config files: [
  '/Users/admin/code/lint-staged-sparse-issue/apps/app1/package.json',
  '/Users/admin/code/lint-staged-sparse-issue/apps/app1/.lintstagedrc.cjs',
  '/Users/admin/code/lint-staged-sparse-issue/package.json'
] +37ms
  lint-staged:loadConfig Loading configuration from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/package.json`... +0ms
  lint-staged:loadConfig Loading configuration from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/.lintstagedrc.cjs`... +0ms
  lint-staged:loadConfig Loading configuration from `/Users/admin/code/lint-staged-sparse-issue/package.json`... +0ms
  lint-staged:loadConfig Successfully loaded config from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/.lintstagedrc.cjs`:
  lint-staged:loadConfig { '*': "echo 'good'" } +1ms
  lint-staged:validateConfig Validating config from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/.lintstagedrc.cjs`... +0ms
  lint-staged:validateConfig Validated config from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/.lintstagedrc.cjs`: +0ms
  lint-staged:validateConfig {
  lint-staged:validateConfig   '*': "echo 'good'"
  lint-staged:validateConfig } +0ms
  lint-staged:loadConfig Successfully loaded config from `/Users/admin/code/lint-staged-sparse-issue/package.json`:
  lint-staged:loadConfig null +1ms
  lint-staged:loadConfig Successfully loaded config from `/Users/admin/code/lint-staged-sparse-issue/apps/app1/package.json`:
  lint-staged:loadConfig null +0ms
  lint-staged:searchConfigs Found 1 config files +3ms
  lint-staged:groupFilesByConfig Grouping 1 files by 1 configurations +0ms
  lint-staged:chunkFiles Resolved an argument string length of 58 characters from 1 files +0ms
  lint-staged:chunkFiles Creating 1 chunks for maxArgLength of 131072 +0ms
  lint-staged:generateTasks Generating linter tasks +0ms
  lint-staged:generateTasks Generated task:
  lint-staged:generateTasks {
  lint-staged:generateTasks   pattern: '*',
  lint-staged:generateTasks   commands: "echo 'good'",
  lint-staged:generateTasks   fileList: [ '/Users/admin/code/lint-staged-sparse-issue/apps/app1/a.txt' ]
  lint-staged:generateTasks } +0ms
  lint-staged:makeCmdTasks Creating listr tasks for commands "echo 'good'" +0ms
  lint-staged:resolveTaskFn cmd: echo +0ms
  lint-staged:resolveTaskFn args: [ 'good' ] +0ms
  lint-staged:resolveTaskFn execaOptions: {
  cwd: '/Users/admin/code/lint-staged-sparse-issue',
  preferLocal: true,
  reject: false,
  shell: false,
  stdin: 'ignore'
} +0ms
  lint-staged:chunkFiles Resolved an argument string length of 58 characters from 1 files +1ms
  lint-staged:chunkFiles Creating 1 chunks for maxArgLength of 131072 +0ms
✔ Preparing lint-staged...
✔ Running tasks for staged files...
✔ Applying modifications from tasks...
✔ Cleaning up temporary files...

lint-staged:GitWorkflow Backing up original state... +0ms
lint-staged:GitWorkflow Getting partially staged files... +1ms
lint-staged:execGit Running git command [ 'status', '-z' ] +92ms
lint-staged:GitWorkflow Found partially staged files: [] +29ms
lint-staged:GitWorkflow Backing up merge state... +0ms
lint-staged:file Reading file `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_HEAD` +0ms
lint-staged:file Reading file `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_MODE` +0ms
lint-staged:file Reading file `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_MSG` +0ms
lint-staged:file File `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_HEAD` doesn't exist, ignoring... +0ms
lint-staged:file File `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_MODE` doesn't exist, ignoring... +0ms
lint-staged:file File `/Users/admin/code/lint-staged-sparse-issue/.git/MERGE_MSG` doesn't exist, ignoring... +0ms
lint-staged:GitWorkflow Done backing up merge state! +0ms
lint-staged:GitWorkflow Getting deleted files... +0ms
lint-staged:execGit Running git command [ 'ls-files', '--deleted' ] +29ms
lint-staged:GitWorkflow Found deleted files: [] +26ms
lint-staged:execGit Running git command [ 'stash', 'create' ] +26ms
lint-staged:execGit Running git command [
  'stash',
  'store',
  '--quiet',
  '--message',
  'lint-staged automatic backup',
  '01dfce55f75c37769f25240d49526f4f2f8ab369'
] +78ms
lint-staged:GitWorkflow Done backing up original state! +94ms
lint-staged:GitWorkflow Adding task modifications to index... +37ms
lint-staged:execGit Running git command [
  'add',
  '--',
  '/Users/admin/code/lint-staged-sparse-issue/apps/app1/a.txt'
] +53ms
lint-staged:GitWorkflow Done adding task modifications to index! +31ms
lint-staged:execGit Running git command [ 'diff', '--name-only', '-z', '--diff-filter=ACMR', '--staged' ] +31ms
lint-staged:GitWorkflow Dropping backup stash... +30ms
lint-staged:execGit Running git command [ 'stash', 'list' ] +30ms
lint-staged:execGit Running git command [ 'stash', 'drop', '--quiet', '0' ] +25ms
lint-staged:GitWorkflow Done dropping backup stash! +51ms
  lint-staged Tasks were executed successfully! +470ms