Support arbitrary repository types for private cocoapods datasource

[mitchins]

Problem Definition

Our organisation uses Mend/Renovate however not being able to query our internal packages for automatic PR can be limiting - particularly because we mirror third party packages internally for security purposes as we are a bank.

From what I can tell, the Cocoapods datasource should in theory work with a private repository - assuming it's:
(1) On Github
(2) Structured in the way that is expected (with the content and specs folders)

It does not however work with any other repository type as it uses the github v3 api to query contents and packages. After looking through the source and building locally - I think it can be handled fairly elegantly, but would hope that the work could be accepted upstream if I goto the effort of supporting it.

Rationale for the suggested enhancement

(A) The organisation I work for uses Azure Devops (We won't be moving to GitHub actions anytime soon either) however manyorganisations still use BitBucket or Gitlab.
(B) Cocoapods is still a popular package manager even with Swift Package Manager as it's much more powerful and proven for enterprise and advanced usages.

Suggested approach

Assumptions:
(1) There are too many different git providers besides GitHub to handle their HTTP APIs
(2) We would have permissions to check out the repository holding the podspecs, given we are supposed to update packages
(3) A shallow checkout of depth=1 should be quick enough, and we can clean it up once done

With the above, if the repository is not GitHub, e.g. dev.azure.com or myorg.bitbucket.com (or bitbucket.myorg.com), rather than what it does now (which is to try to call the GitHub api and fail with 404/401), let's just check out the repository locally using a shallow clone, and query the directory structure.

This approach is more or less how cocoapods itself queries it, and even caches it locally (although we should let it do its thing rather than try to inspect that).

In Summary

For the index.ts of the Cocoapods DataSource, if the repository url is not the public CDN or a github repository, can we use a strategy of checking out the repository to query available packages in the repository structure? This pattern isn't new either as it looks like Cargo (among others) uses a repository clone.

The approach would work seamlessly with any "other" provider besides the CDN or GitHub, and as mentioned there's still alot of life left in CocoaPods as well as enterprise usage of non GitHub repositories.

Again while I am willing to look into the work, I'd prefer to have an approach that could be accepted upstream so that it could:
(A) Work with Mend cloud service
(B) Not require manual merges from upstream

[rarkins]

Do you mean that you mirror https://github.com/CocoaPods/Specs? And that you'd want to configure Renovate to tell it that using packageRules with registryUrls?

Regarding:

(1) There are too many different git providers besides GitHub to handle their HTTP APIs

We have a getJson function for all our supported platforms, including Azure DevOps. It's currently used for fetching presets. So I don't think this assumption holds.

[mitchins]

Thanks for your reply.

We don't mirror cocoa pods, we:

Have our own private packages for internal libraries in the format:
/<Library>/<Version>/<Library>.podspec.json
That is the standard format for a private repository as sharding etc isn't require - so it's the simple way.

Most of these libraries are internal modules but we have some vendor packages hosted too.

At the top of our Podfile we declare the repository:

Source "https://dev.azure.com/myOrg/myProject/_git/pod-specs-repository/

With regards to Azure, it supports fetching the actual projects from azure devops just fine. However cocoapods data source can only fetch from GitHub.
The Datasource assumes it is either official CDN or GitHub, and it uses the GitHub HTTP api only to access it. This isn't the same API as azure.

I looked in the http component folder and I didn't see a class that exposes the same function for azure as GitHub with getJson.

From here:

https://github.com/renovatebot/renovate/blob/04f4edbd518f4fe7409a573357efed3fa8ff82a9/lib/modules/datasource/pod/index.ts

It has a reference to:

githubHttp: GithubHttp; in the code below it makes use of .getJson

However, I cannot see an equivalent class or function for azure based on the above.

Called here:

function releasesGithubUrl(
  packageName: string,
  opts: {
    hostURL: string;
    account: string;
    repo: string;
    useShard: boolean;
    useSpecs: boolean;
  },
): string {
  const { hostURL, account, repo, useShard, useSpecs } = opts;
  const prefix =
    hostURL && hostURL !== 'https://github.com'
      ? `${hostURL}/api/v3/repos`
      : 'https://api.github.com/repos';
  const shard = shardParts(packageName).join('/');
  // `Specs` in the pods repo URL is a new requirement for legacy support also allow pod repo URL without `Specs`
  const packageNamePath = useSpecs ? `Specs/${packageName}` : packageName;
  const shardPath = useSpecs
    ? `Specs/${shard}/${packageName}`
    : `${shard}/${packageName}`;
  const suffix = useShard ? shardPath : packageNamePath;
  return `${prefix}/${account}/${repo}/contents/${suffix}`;
}

private async getReleasesFromGithub(
    packageName: string,
    opts: { hostURL: string; account: string; repo: string },
    useShard = true,
    useSpecs = true,
    urlFormatOptions: URLFormatOptions = 'withShardWithSpec',
  ): Promise<ReleaseResult | null> {
    const url = releasesGithubUrl(packageName, { ...opts, useShard, useSpecs });
    const resp = await this.requestGithub<{ name: string }[]>(url, packageName);
    if (resp) {
      const releases = resp.map(({ name }) => ({ version: name }));
      return { releases };
    }

    // support different url formats
    switch (urlFormatOptions) {
      case 'withShardWithSpec':
        return this.getReleasesFromGithub(
          packageName,
          opts,
          true,
          false,
          'withShardWithoutSpec',
        );
      case 'withShardWithoutSpec':
        return this.getReleasesFromGithub(
          packageName,
          opts,
          false,
          true,
          'withSpecsWithoutShard',
        );
      case 'withSpecsWithoutShard':
        return this.getReleasesFromGithub(
          packageName,
          opts,
          false,
          false,
          'withoutSpecsWithoutShard',
        );
      case 'withoutSpecsWithoutShard':
      default:
        return null;
    }
  }

What would be a great help is
(A) point me towards the equivalent azure function if I've missed it.
(B) indicate the preferred method of refactoring the existing cocoapods datasource to support the repository.

Cocoapods itself can use any GIT repository over HTTPS or SSH.
However it appears that because the repository is being queried with a platform specific api and not cloned, the current implementation for a data source won't work on anything besides GitHub.

Any help or direction would be much appreciated, however after extensive testing I've been unable to have renovate work with an azure repository for the cocoapods spec (even though pod itself works fine).

[rarkins]

The file fetching is in lib/modules/platform, such as

renovate/lib/modules/platform/azure/index.ts

Lines 123 to 180 in 23421c5

	export async function getRawFile(
	fileName: string,
	repoName?: string,
	branchOrTag?: string,
	): Promise<string | null> {
	try {
	const azureApiGit = await azureApi.gitApi();
	let repoId: string | undefined;
	if (repoName) {
	const repos = await azureApiGit.getRepositories();
	const repo = getRepoByName(repoName, repos);
	repoId = repo?.id;
	} else {
	repoId = config.repoId;
	}
	if (!repoId) {
	logger.debug('No repoId so cannot getRawFile');
	return null;
	}
	const versionDescriptor: GitVersionDescriptor = {
	version: branchOrTag,
	} satisfies GitVersionDescriptor;
	const buf = await azureApiGit.getItemContent(
	repoId,
	fileName,
	undefined,
	undefined,
	undefined,
	undefined,
	undefined,
	undefined,
	branchOrTag ? versionDescriptor : undefined,
	);
	const str = await streamToString(buf);
	return str;
	} catch (err) /* istanbul ignore next */ {
	if (
	err.message?.includes('<title>Azure DevOps Services Unavailable</title>')
	) {
	logger.debug(
	'Azure DevOps is currently unavailable when attempting to fetch file - throwing ExternalHostError',
	);
	throw new ExternalHostError(err, id);
	}
	if (err.code === 'ECONNRESET' || err.code === 'ETIMEDOUT') {
	throw new ExternalHostError(err, id);
	}
	if (err.statusCode && err.statusCode >= 500 && err.statusCode < 600) {
	throw new ExternalHostError(err, id);
	}
	throw err;
	}
	}

This can be used if you're fetching from the same AZDO org as the repos Renovate is running against - is that the case?

An example of it being used in the presets logic is in

renovate/lib/config/presets/local/common.ts

Lines 7 to 33 in 23421c5

	export async function fetchJSONFile(
	repo: string,
	fileName: string,
	_endpoint?: string,
	tag?: string | undefined,
	): Promise<Preset> {
	let raw: string | null;
	try {
	raw = await platform.getRawFile(fileName, repo, tag ?? undefined);
	} catch (err) {
	if (err instanceof ExternalHostError) {
	throw err;
	}
	logger.debug(
	`Preset file ${fileName} not found in ${repo}: ${err.message}}`,
	);
	throw new Error(PRESET_DEP_NOT_FOUND);
	}
	if (!raw) {
	throw new Error(PRESET_DEP_NOT_FOUND);
	}
	return parsePreset(raw, fileName);
	}

The logic could (hopefully) be like this:

• If it's not the CDN and not GitHub
• Is it the same server as "endpoint"?
• If so, call platform.getRawFile() with appropriate params

[mitchins]

Thanks, yes it is the same platform/organisation azure/org (although the repository is different).
I will dig into what platform is, and I agree with your logic, if it’s not known as CDN or GitHub then fall back to pulling platform.

Although we would need to use “get file listing” as the versions are in hierarchical directory format, is this part of platform?

[rarkins]

File listing is not an exposed API like getRawFile(), but for which step do you need it?

BTW if you think it's necessary to use git, and the approach taken in Cargo would work, you are welcome to implement that.

[mitchins]

Great thanks.GIT would be the perfect “catch all”.I doubt performance of a couple extra seconds would matter, with shallow commit.Just wanted to get a sense check before I commit significant effort. With regards to GitHub currently it doesn’t actually check the domain, I believe “git” just needs to be in the host url regex. Obviously only GitHub supports the GitHub query API.I’m unsure what forms the domain takes with enterprise accounts, eg, does GitHub always occur even for big enterprises? Long story short:If url == CDN {}else if url == GitHub {}else { do GIT checkout and collect versions}As long as that seems endorse im happy to go ahead - just wanted to sense check.Given that cocoapods requires directory hierarchy to list versions this is not standardised, so apart from GitHub unless you want to support every sundry provider then a shallow clone is guaranteed to work (although not super elegant).Sent from my iPhoneOn 14 May 2024, at 19:28, Rhys Arkins ***@***.***> wrote: File listing is not an exposed API like getRawFile(), but for which step do you need it? BTW if you think it's necessary to use git, and the approach taken in Cargo would work, you are welcome to implement that. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>