Parallel git calls with simple-git leads to random errors in git-submodules manager

[ericcitaire]

How are you running Renovate?

Self-hosted

Please select which platform you are using if self-hosting.

Bitbucket Server

If you're self-hosting Renovate, tell us what version of Renovate you run.

25.76.2

Describe the bug

We are using Renovate to update up to 80 Git submodules on a single BitBucket repository.
On each execution, Renovate logs show Invalid url errors on a random subset of submodules, causing Renovate to ignore these submodules. I want to emphasize that the list of failing submodules varies on each execution. Occasionally, there is no error at all.

After a lot of troubleshooting, we found that this is most likely related to a bug in simple-git (see steveukx/git-js#320), which doesn't play well with parallel executions of Git.
When updating submodules, Renovate executes a bunch of Git commands for each submodule, all in parallel :

renovate/lib/manager/git-submodules/extract.ts

Lines 101 to 134 in 6eafd2f

	const deps = (
	await Promise.all(
	depNames.map(async ({ name, path }) => {
	try {
	const [currentDigest] = (await git.subModule(['status', path]))
	.trim()
	.replace(/^[-+]/, '')
	.split(/\s/);
	const subModuleUrl = await getUrl(git, gitModulesPath, name);
	// hostRules only understands HTTP URLs
	// Find HTTP URL, then apply token
	let httpSubModuleUrl = getHttpUrl(subModuleUrl);
	httpSubModuleUrl = getRemoteUrlWithToken(httpSubModuleUrl);
	const currentValue = await getBranch(
	gitModulesPath,
	name,
	httpSubModuleUrl
	);
	return {
	depName: path,
	lookupName: getHttpUrl(subModuleUrl),
	currentValue,
	currentDigest,
	};
	} catch (err) /* istanbul ignore next */ {
	logger.warn(
	{ err },
	'Error mapping git submodules during extraction'
	);
	return null;
	}
	})
	)
	).filter(Boolean);

In our case, it causes const subModuleUrl = await getUrl(git, gitModulesPath, name); to occasionally return an empty string, causing an Invalid url error in getHttpUrl().

We tried several workarounds. One of them seems to work pretty well : replacing Promise.all(...) by a sequential loop :

const deps = [];
for (const { name, path } of depNames) {
    try {
      const [currentDigest] = (await git.subModule(['status', path]))
        .trim()
        .replace(/^[-+]/, '')
        .split(/\s/);
      const subModuleUrl = await getUrl(git, gitModulesPath, name);
      // hostRules only understands HTTP URLs
      // Find HTTP URL, then apply token
      let httpSubModuleUrl = getHttpUrl(subModuleUrl);
      httpSubModuleUrl = getRemoteUrlWithToken(httpSubModuleUrl);
      const currentValue = await getBranch(
        gitModulesPath,
        name,
        httpSubModuleUrl
      );
      deps.push({
        depName: path,
        lookupName: getHttpUrl(subModuleUrl),
        currentValue,
        currentDigest,
      });
    } catch (err) /* istanbul ignore next */ {
      logger.warn(
        { err },
        'Error mapping git submodules during extraction'
      );
    }
}

Relevant debug logs

Logs

 WARN: Error mapping git submodules during extraction (repository=TEAM/repo)
       "err": {
         "message": "Invalid url.",
         "stack": "Error: Invalid url.\n    at parseUrl (/usr/src/app/node_modules/parse-url/lib/index.js:41:15)\n    at gitUp (/usr/src/app/node_modules/git-up/lib/index.js:29:18)\n    at Object.gitUrlParse [as default] (/usr/src/app/node_modules/git-url-parse/lib/index.js:42:19)\n    at Object.getHttpUrl (/usr/src/app/node_modules/renovate/lib/util/git/url.ts:6:32)\n    at /usr/src/app/node_modules/renovate/dist/manager/git-submodules/extract.js:162:42\n    at runNextTicks (internal/process/task_queues.js:60:5)\n    at listOnTimeout (internal/timers.js:524:9)\n    at processTimers (internal/timers.js:498:7)\n    at async Promise.all (index 33)\n    at Object.extractPackageFile (/usr/src/app/node_modules/renovate/dist/manager/git-submodules/extract.js:153:19)\n    at Object.getManagerPackageFiles (/usr/src/app/node_modules/renovate/lib/workers/repository/extract/manager-files.ts:51:19)\n    at /usr/src/app/node_modules/renovate/lib/workers/repository/extract/index.ts:45:28\n    at async Promise.all (index 0)\n    at Object.extractAllDependencies (/usr/src/app/node_modules/renovate/lib/workers/repository/extract/index.ts:43:26)\n    at Object.extract (/usr/src/app/node_modules/renovate/lib/workers/repository/process/extract-update.ts:82:20)\n    at Object.extractDependencies (/usr/src/app/node_modules/renovate/lib/workers/repository/process/index.ts:40:33)\n    at Object.renovateRepository (/usr/src/app/node_modules/renovate/lib/workers/repository/index.ts:46:52)\n    at Object.start (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:111:7)\n    at /usr/src/app/node_modules/renovate/lib/renovate.ts:16:22"
       }

Have you created a minimal reproduction repository?

No reproduction repository

[viceice]

Please provide a pr, we will accept that change

[renovate-release]

🎉 This issue has been resolved in version 27.3.2 🎉

The release is available on:

• GitHub release
• 27.3.2

Your semantic-release bot 📦🚀