copier: Renovate does not support git+https

[rarkins]

Discussed in #31454

^{Originally posted by reitzig September 18, 2024}

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Bitbucket Data Center v8.19.6; Renovate 38.82.1

Please tell us more about your question or problem

Copier supports source paths with several Git-related protocols:

If Copier doesn't detect your remote URL as a Git repository, make sure it starts with one of git+https://, git+ssh://, git@ or git://, or it ends with .git.
https://copier.readthedocs.io/en/stable/generating/

Running Renovate on a repository that contains a .copier-answers.yaml with a git+https-URL, we get:

git: 'remote-git+https' is not a git command. See 'git --help'.\n

I can reproduce a similar error message with my local Git:

🐧❯ git --version
git version 2.25.1

🐧❯ git clone git+https://bitbucket.some-org/scm/some-project/some-template.git
Cloning into 'some-template'...
fatal: unable to find remote helper for 'git+https'

It seems as if Renovate forwarded the URL it found in .copier-answers.yaml to git; in this case, it would need to remove a git+ prefix for that to work.

Notes

• In our case, plain https://(...).git works.
• We would like to use ssh:// but our Renovate setup uses HTTPS to access our Bitbucket; an automagic mapping would be great (but finicky since Bitbucket HTTPS vs SSH URLs have different structure).

References

• reitzig/reproduction-renovate-31454
• renovatebot/renovate#29215 -- introduces copier support; tests only extraction of https.
• copier documentation

Logs (if relevant)

Logs

DEBUG: Datasource unknown error (repository=some-project/some-repo, baseBranch=doc/bounded-contexts)
       "datasource": "git-tags",
       "packageName": "git+https://bitbucket.some-org/scm/some-project/some-template.git",
       "err": {
         "task": {
           "commands": [
             "ls-remote",
             "git+https://bitbucket.some-org/scm/some-project/some-template.git"
           ],
           "format": "utf-8",
           "parser": "[function]"
         },
         "message": "git: 'remote-git+https' is not a git command. See 'git --help'.\n",
         "stack": "Error: git: 'remote-git+https' is not a git command. See 'git --help'.\n\n    at Object.action (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/src/lib/plugins/error-detection.plugin.ts:42:29)\n    at PluginStore.exec (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/src/lib/plugins/plugin-store.ts:54:29)\n    at /usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:124:42\n    at new Promise (<anonymous>)\n    at GitExecutorChain.handleTaskData (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:121:14)\n    at GitExecutorChain.<anonymous> (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:97:40)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/simple-git/dist/cjs/index.js:52:24)"
       }
DEBUG: Failed to look up git-tags package git+https://bitbucket.some-org/scm/some-project/some-template.git (repository=some-project/some-repo, baseBranch=doc/bounded-contexts, packageFile=test-bc-a/.copier-answers.yml, 

[reitzig]

In our case, plain https://(...).git works.

I have to retract that statement in its generality. While Renovate is able to extract and lookup, actually performing the update fails. Renovate does not seem to forward HTTPS credentials to copier, or at least not in a way that copier understands:

Command failed: copier update --skip-answered --defaults --answers-file .copier-answers.yml --vcs-ref v2.2.0
Traceback (most recent call last):

<snip>

plumbum.commands.processes.ProcessExecutionError: Unexpected exit code: 128
Command line: | /usr/bin/git clone --no-checkout https://bitbucket.some.org/scm/some-project/some-repo.git /tmp/copier.vcs.clone.trm6f1wz --filter=blob:none
Stderr:       | Cloning into '/tmp/copier.vcs.clone.trm6f1wz'...
              | fatal: could not read Username for 'https://bitbucket.some.org': No such device or address

Funny enough, we end up with "not quite empty" PRs:

Anyway. Is credential forwarding part of this ticket, anyway, or would you like a separate discussion/issue, @rarkins ?

Addendum: After some tinkering outside of Renovate, I think it should be possible to configure a suitable Git credential helper in (our custom extension of) the Renovate container image. That way, any tool that delegates to the Git CLI would work as expected. While it would be great if Renovate (or the image) would be set up to do that by default, we see the potential for a workaround, at least in on-prem settings.

[reitzig]

I don't know since when, but today Renovate -- presumably the copier manager? -- does add some Git configuration as environment variables, something like

"GIT_CONFIG_KEY_3": "url.https://**redacted**@bitbucket.some.org/scm/.insteadOf",
"GIT_CONFIG_VALUE_3": "ssh://**redacted**@bitbucket.some.org:7999/",
"GIT_CONFIG_KEY_4": "url.https://**redacted**@bitbucket.some.org/scm/.insteadOf",
"GIT_CONFIG_VALUE_4": "ssh://**redacted**@bitbucket.some.org:7999/",
"GIT_CONFIG_KEY_5": "url.https://**redacted**@bitbucket.some.org/scm/.insteadOf",
"GIT_CONFIG_VALUE_5": "https://bitbucket.some.org/scm/",

which I interpret as: "if we can make either of HTTPs or SSH work, use that" (which may be completely off, I'm not familiar with the url.<url>.insteadOf config of Git).

Whatever the exact semantics of this trio of config settings, it works (for us).

We now have both HTTPS and SSH credentials configured in our Renovate deployment, and I can't tell which is ultimately being used by copier, but it does run successfully. If the HTTPS credentials were used, this ticket would be fixed (well, git+https may not work, haven't checked); if the SSH credentials were used, we have a workaround.