No PR created on outdated submodule #8529

jimbojd72 · 2021-02-03T21:26:11Z

jimbojd72
Feb 3, 2021

Hello renovate helpers!

Using renovate/renovate:24.32.5

We are currently creating to use git-submodule feature (https://docs.renovatebot.com/modules/manager/git-submodules/) feature but I am not able to make it create a PR when one submodule is out of date.

Note that <private-repo> below is usually an normal string :)

I tried to put "git-submodules": { "enabled": true} where it is needed (both the bot config or renovate.json). I tried to set a branch to follow inside my .gitmodules (ex: git submodule set-branch --branch python3 -- packages/repo1).

I can also go in the repository and manually do git submodule update --remote and git correctly update the submodule to the head of the branch

$ git submodule update --remote
Submodule path 'packages/repo2': checked out '730c984c171b638ff201bee17794872d0d88afaf'
Submodule path 'packages/repo1': checked out 'e26a4c33c1bf713acad4f0f5bcf4fcc0a14671da'

I see from the logs that my submodule are correctly detected, but still no PR are opened. I can see this error in the log. I'm wondering if it is related to ssh vs https usage.

Note that this setup works correctly for other PR generation. Only submodule support is problematic

15:33:25  DEBUG: Datasource unknown error (repository=<private-repo>/pstn-portability)
15:33:25         "datasource": "git-submodules",
15:33:25         "lookupName": "packages/repo1",
15:33:25         "err": {
15:33:25           "task": {
15:33:25             "commands": [
15:33:25               "ls-remote",
15:33:25               "--refs",
15:33:25               "https://github.com/<private-repo>/repo1.git",
15:33:25               "python3"
15:33:25             ],
15:33:25             "format": "utf-8"
15:33:25           },
15:33:25           "message": "fatal: could not read Username for 'https://github.com': No such device or address\n",
15:33:25           "stack": "Error: fatal: could not read Username for 'https://github.com': No such device or address\n\n    at GitExecutorChain.onFatalException (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:67:77)\n    at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:59:21)\n    at Generator.throw (<anonymous>)\n    at rejected (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
15:33:25         }
15:33:25  DEBUG: Failed to look up dependency packages/repo1 (repository=<private-repo>/pstn-portability, packageFile=.gitmodules, dependency=packages/repo1)

Bot config:

module.exports = {
  logFileLevel: "debug",
  onboarding: true,
  onboardingConfig: {
    extends: ["config:base"],
    groupName: "external",
    semanticCommits: "enabled",
    packageRules: [
      {
        packagePatterns: ["github.com/<private-repo>"],
        groupName: "<private-repo>-internal"
      }
    ]
  },
  "git-submodules": {
    "enabled": true
  },
  autodiscover: true,
  autodiscoverFilter: "<private-repo>/+(xyxPROJECTSxyx)",
  commitMessageSuffix: "[NS]",
  assignees: ['<private-repo>-renovate'],
  labels: ['bot', 'dependencies'],
  python: {
    enabled: true
  }
};

renovate.json:

{
  "extends": [
    "config:base",
    ":semanticCommitTypeAll(feat)"
  ],
  "groupName": "external",
  "packageRules": [
    {
      "packagePatterns": [
        "github.com/<private-repo>"
      ],
      "groupName": "<private-repo>-internal"
    }
  ],
  "git-submodules": {
    "enabled": true
  },
  "reviewers": ["team:dev-pstn"]
}

.gitmodules

[submodule "packages/repo1"]
	path = packages/repo1
	url = git@github.com:<private-repo>/repo1.git
	branch = python3
[submodule "packages/repo2"]
	path = packages/repo2
	url = git@github.com:<private-repo>/repo2.git
	branch = master

Answered by rarkins

Feb 3, 2021

@JamieMagee do we have support for private submodules? Source: https://github.com/renovatebot/renovate/blob/master/lib/datasource/git-submodules/index.ts

View full answer

rarkins · 2021-02-03T21:39:13Z

rarkins
Feb 3, 2021
Maintainer

@JamieMagee do we have support for private submodules? Source: https://github.com/renovatebot/renovate/blob/master/lib/datasource/git-submodules/index.ts

9 replies

jimbojd72 Feb 4, 2021
Author

Thanks for your suggestion, but it seems I may still miss something.

I tried below to set a hostRule for both domainName github.com and baseUrl https://github.com with username/password field and I have still the same issue where it doesn't see the actual Username at all.

08:34:39  DEBUG: Combined config
08:34:39         "config": {
08:34:39           "logFileLevel": "debug",
08:34:39           "onboarding": true,
08:34:39           "onboardingConfig": {
08:34:39             "extends": ["config:base"],
08:34:39             "groupName": "external",
08:34:39             "semanticCommits": "enabled",
08:34:39             "packageRules": [
08:34:39               {
08:34:39                 "groupName": "<private-repo>-internal",
08:34:39                 "matchPackagePatterns": ["github.com/<private-repo>"]
08:34:39               }
08:34:39             ]
08:34:39           },
08:34:39           "git-submodules": {"enabled": true},
08:34:39           "hostRules": [
08:34:39             {
08:34:39               "baseUrl": "https://github.com", #Also tried `github.com` domain
08:34:39               "username": "****",
08:34:39               "password": "***********"
08:34:39             },
08:34:39             {"hostType": "github", "domainName": "github.com", "token": "***********"}
08:34:39           ],
08:34:39           "autodiscover": true,
08:34:39           "autodiscoverFilter": "<private-repo>/+(pstn-portability)",
08:34:39           "commitMessageSuffix": "[NS]",
08:34:39           "assignees": ["****"],
08:34:39           "labels": ["bot", "dependencies"],
08:34:39           "python": {"enabled": true},
08:34:39           "logLevel": "debug",
08:34:39           "logFile": "/var/lib/jenkins/workspace/e_PSTN-3797-git-submodule-branch/renovate-python-pip.log",
08:34:39           "token": "***********"
08:34:39         }

...


08:34:53  DEBUG: Dependency debian has unsupported value stretch (repository=<private-repo>/pstn-portability)
08:34:54  DEBUG: Datasource unknown error (repository=<private-repo>/pstn-portability)
08:34:54         "datasource": "git-submodules",
08:34:54         "lookupName": "packages/repo1",
08:34:54         "err": {
08:34:54           "task": {
08:34:54             "commands": [
08:34:54               "ls-remote",
08:34:54               "--refs",
08:34:54               "https://github.com/<private-repo>/repo1.git",
08:34:54               "python3"
08:34:54             ],
08:34:54             "format": "utf-8"
08:34:54           },
08:34:54           "message": "fatal: could not read Username for 'https://github.com': No such device or address\n",
08:34:54           "stack": "Error: fatal: could not read Username for 'https://github.com': No such device or address\n\n    at GitExecutorChain.onFatalException (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:67:77)\n    at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:59:21)\n    at Generator.throw (<anonymous>)\n    at rejected (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
08:34:54         }
08:34:54  DEBUG: Failed to look up dependency packages/repo1 (repository=<private-repo>/pstn-portability, packageFile=.gitmodules, dependency=packages/repo1)

Thanks again

agaudreault Feb 10, 2021

Setting git config --global url."https://$TOKEN:x-oauth-basic@github.com/".insteadOf "https://github.com/" in the container fix the issue.

However, I would expect renovate to work without that global config since using https with a github access token will always result with that error. It seems caused by a problem with the url generator dependency, so I created IonicaBizau/git-url-parse#126.

https://github.com/IonicaBizau/git-url-parse/blob/71c7298bff814af8d3d5ca60583ccad42236b734/lib/index.js#L272-L279

JamieMagee Feb 10, 2021

@agaudreault-jive You're talking about git-url-parse? I didn't think we needed to set x-oauth-basic in the URL explicitly, and that the token was sufficient. I'll investigate further.

jjathman Mar 3, 2021

@JamieMagee does git-submodules work if the port for SSH access isn't the same as what should be used for HTTPS access? We have a submodule with a url like ssh://git@bitbucket.mycompany.com:7999/dev/myrepo.git but it looks like when the URL is converted to HTTPS it is still maintaining the non-standard port of 7999. I don't see that hostRules supports specifying a port.

I can make this a new discussion is that's better, it just seemed related. Thank you.

JamieMagee Mar 6, 2021

@jjathman I created an issue for this #9003

jimbojd72 · 2021-02-15T12:46:24Z

jimbojd72
Feb 15, 2021
Author

I think we are done. We can close this.

7 replies

augi Apr 13, 2021

I will definitely prepare all the required data later and create a new issue/discussion, I just wanted to check if there is some low-hanging fruit, some obvious thing I forget.

Anyway, we are using self-hosted renovate/renovate:24.113 running in k8s, pointing it to our GitHub Enterprise instance.

In the log, we can see Unable to initialise git submodule at ... warning (Host key verification failed. seems to be the core message). But this is probably OK as submodule cloning is not required for the extraction.

The following warning seems to be the root cause - it says Error mapping git submodules during extraction with fatal: could not read Username for 'https://our_github_instance': No such device or address message.

Please note that renovate is able to access both repositories (it actually creates another PRs in both).

Also please note that url of the submodule is in git@our_github_instance:some-org/some-project.git format so the conversion from ssh to https went OK.

augi Apr 13, 2021

@viceice I tried to enable debug logging and printConfig, and also set RENOVATE_HOST_RULES. According to the logs, the Env config contains the expected hostRules, Combined config as well, but hostRules is [] in the Full resolved config including presets. Do you have idea what could be wrong? 🙏

jimbojd72 Apr 13, 2021
Author

#8529 (reply in thread) That was our solution in the end

augi Apr 13, 2021

Just for the record, our solution is to add RENOVATE_HOST_RULES with following value: [{"domainName":"our_github_domain","token":"same_as_RENOVATE_TOKEN"}]

augi Apr 13, 2021

@viceice I've also prepared #9529 for this - I believe it could improve usability for others ❤️

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No PR created on outdated submodule #8529

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 16 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

No PR created on outdated submodule #8529

Replies: 2 comments · 16 replies

rarkins Feb 3, 2021 Maintainer

jimbojd72 Feb 4, 2021 Author

jimbojd72 Feb 15, 2021 Author

jimbojd72 Apr 13, 2021 Author

Replies: 2 comments 16 replies

rarkins
Feb 3, 2021
Maintainer

jimbojd72 Feb 4, 2021
Author

jimbojd72
Feb 15, 2021
Author

jimbojd72 Apr 13, 2021
Author