Intermittent CI failures due to Bazaar Launchpad upstream request timeouts

[ichard26]

I've seen numerous random CI failures like this one. Here's an another example.

The test failing is always test_vcs_url_urlquote_normalization.

pip/tests/functional/test_install.py

Lines 485 to 502 in 285ff72

	def test_vcs_url_urlquote_normalization(
	script: PipTestEnvironment, tmpdir: Path
	) -> None:
	"""
	Test that urlquoted characters are normalized for repo URL comparison.
	"""
	script.pip(
	"install",
	"-e",
	"{url}/#egg=django-wikiapp".format(
	url=local_checkout(
	"bzr+http://bazaar.launchpad.net/"
	"%7Edjango-wikiapp/django-wikiapp"
	"/release-0.1",
	tmpdir,
	)
	),
	)

INFO     pip._internal.vcs.bazaar:bazaar.py:40 Checking out http://bazaar.launchpad.net/%7Edjango-wikiapp/django-wikiapp/release-0.1 to /tmp/pytest-of-runner/pytest-1/popen-gw2/test_vcs_url_urlquote_normaliz0/cache/release-0.1
INFO     pip.subprocessor:subprocess.py:119 Running command bzr checkout --lightweight --quiet http://bazaar.launchpad.net/%7Edjango-wikiapp/django-wikiapp/release-0.1 /tmp/pytest-of-runner/pytest-1/popen-gw2/test_vcs_url_urlquote_normaliz0/cache/release-0.1
INFO     pip.subprocessor:subprocess.py:158 brz: ERROR: Connection error: while sending POST /~django-wikiapp/django-wikiapp/release-0.1/.bzr/smart: [Errno 110] Connection timed out
ERROR    pip.subprocessor:subprocess.py:197 bzr checkout --lightweight --quiet http://bazaar.launchpad.net/%7Edjango-wikiapp/django-wikiapp/release-0.1 /tmp/pytest-of-runner/pytest-1/popen-gw2/test_vcs_url_urlquote_normaliz0/cache/release-0.1 exited with 3

A telltale sign is a singular job failing that takes >12 minutes longer than usual (as each of the four retries timeouts after ~270 seconds). This has been quite annoying. Is there any more reliable public Bazaar instance we can use? Actually, we can probably simply switch to using a public GitHub repository as the test doesn't need to use Bazaar AFAIK. The git blame goes all the way back to 2010.

[notatallshaw]

Doing a bit of digging around, I found that Bazaar is no longer maintained and a fork called Breezy was made several years ago. If you install bzr on Ubuntu you get Breezy not Bazaar.

I suspect that the service at "http://bazaar.launchpad.net" will continue to degrade, I found some emails in their mailing list from 2023 about the service: https://lists.ubuntu.com/archives/bazaar/2023q4/076422.html

I wonder if there is a better place to point a test repo to, I have asked on the Breezy bug tracker (not sure if that's the right place but I do see people responding there as recently as October 2024): https://bugs.launchpad.net/brz/+bug/2093807

[pfmoore]

Maybe we should simply drop support for Bazaar, if it's no longer supported. From what I can see, Breezy uses a different command name (brz rather than bzr) so our current code doesn't support Breezy, and no-one has requested that it gets added, suggesting that there's little or no interest in practice.

[jelmer]

FWIW Breezy installs "bzr" as alias for "brz" on most platforms, and the command-line is backwards compatible.

This command works for me FWIW (with "bzr" referring to breezy 3.3.9) and is pretty quick:

% /usr/bin/bzr --version
Breezy (brz) 3.3.9
...
% time /usr/bin/bzr co --lightweight --quiet http://bazaar.launchpad.net/%7Edjango-wikiapp/django-wikiapp/release-0.1
...
/usr/bin/bzr co --lightweight --quiet   0.23s user 0.07s system 18% cpu 1.657 total

[jelmer]

I suspect that the service at "http://bazaar.launchpad.net" will continue to degrade, I found some emails in their mailing list from 2023 about the service: https://lists.ubuntu.com/archives/bazaar/2023q4/076422.html

That post is related to the breezy website, not to Launchpad - which has been functioning fine and which should continue to do so.

[notatallshaw]

Hi @jelmer thanks for posting here, and thanks for clearing up about the websites.

Responding to your question on laundpad (though let me know if you'd prefer to discuss there), the ask is: what is the best way to test against a sample breezy/bazaar repo?

As we are seeing recent CI failures against http://bazaar.launchpad.net/~django-wikiapp/django-wikiapp/release-0.1/changes, and our CI is not run that frequently (often it's not run for over 24 hours). This is fairly high cost to us because the CI pipeline takes ~30 mins to run, and everyone here is a volunteer, so no one really has time to babysit it.

If the answer is launchpad, who would we contact to ask about service reliability?

[jelmer]

Responding to your question on laundpad (though let me know if you'd prefer to discuss there), the ask is: is what is the best way to test against a sample breezy/bazaar repo?

Launchpad is probably the largest code hoster for Bazaar branches, so I would recommend that.

If the answer is launchpad, who would we contact to ask about service reliability?

The launchpad team; they can be contacted using:

• Launchpad Answers (if you have a Launchpad account): https://answers.launchpad.net/launchpad/+addquestion
• The launchpad-users mailing list: http://lists.ubuntu.com/mailman/listinfo/launchpad-users
• #launchpad channel for user questions, #launchpad-dev for development. Both are on irc.libera.chat

As we are seeing recent CI failures against http://bazaar.launchpad.net/~django-wikiapp/django-wikiapp/release-0.1/changes, and our CI is not run that frequently (often it's not run for over 24 hours).

What's the exact URL you're hitting (/changes involves a redirect from the repository browser, so that adds another component that needs to be functioning)?

(In general, I'm also happy to help debug Bazaar/Breezy issues related to pip if you @ me, I don't follow pip issues proactively)

[notatallshaw]

What's the exact URL you're hitting (/changes involves a redirect from the repository browser, so that adds another component that needs to be functioning)?

I'm pretty sure this is the subprocess pip spawns in the CI:

bzr checkout --lightweight --quiet http://bazaar.launchpad.net/%7Edjango-wikiapp/django-wikiapp/release-0.1 /tmp/tmp0ozuxcfq/cache/release-0.1

Excluding the last argument, which I just filled in there, but would be some temporary directory the CI creates.

[jelmer]

What version of breezy/bazaar is this running? Either way, I'm not really aware of recent issues that would cause occasional timeouts so my guess is that this is server side.

[notatallshaw]

I'm guessing Breezy 3.3.5 as it is installed during the workflow on Ubuntu 22.04/24.04 using apt-get update/install and that's the version I get when I installed from Ubuntu locally: https://github.com/pypa/pip/blob/main/.github/workflows/ci.yml#L135

Yeah, I'll ask launchpad if they have any known service realiablity issues recently, thanks for your help.