python 3 port plan

[PeterSurda]

Using the principles of continuous delivery, port to python 3 should progress as following:

• the first PR will add allow travis to run python3 while successfully doing nothing. The pybitmessage binary will check if it's running python 3 and if yes, exit with return code 0. Unit tests will be decorated to skip all tests when using python 3. This should allow travis to succeed in python 3. Not sure what to do if requirements.txt diverge between python 2 and python 3.
• gradually tests will be updated to allow running in python 3
• maybe new tests will be written to cater for areas that currently don't have tests to facilitate gradual porting
• once everything is working correctly in python 3, as well as binary builds for the main platforms, python 2 support will be dropped

[g1itch]

the first PR will add allow travis to run python3 while successfully doing nothing. The pybitmessage binary will check if it's running python 3 and if yes, exit with return code 0. Unit tests will be decorated to skip all tests when using python 3. This should allow travis to succeed in python 3.

In my ci branch I have python "3.6" in allow_failures and build itself succeeds if python3 part fails. But we all see the failures.

[g1itch]

Not sure what to do if requirements.txt diverge between python 2 and python 3.

Unlikely

[PeterSurda]

In my ci branch I have python "3.6" in allow_failures and build itself succeeds if python3 part fails. But we all see the failures.

This is interesting, unfortunately it doesn't help me here. I need something that allows a gradual compatibility with python 3 with as little human intervention as possible. If all the errors are just ignored, then a PR reviewer needs more time to figure out what's working and what not by looking at the detailed output. The creator of the PR also needs to spend more time evaluating whether he still needs to continue working or wait for a review. Whereas in my proposal, a failure means the PR is rejected and the PR creator has immediate feedback that the isn't finished, and a success means it's ready for a review. This way, PR after PR, python 3 support will be implemented.

I already tried several approaches for getting python 3 support intergrated but none produced a PR that can be merged. One of the developers produced a branch which has most of the functionality as well as most unit tests working (or at least it looks like it's working), but they don't have the proper quality and aren't chunked nicely for PRs. My proposal allows this to happen (refactor or write from scratch, depending on which makes more sense).

Then there are other minor issues, like travis2bash not understanding the full travis syntax, the non-developers not having proper feedback. In my proposal, pybitmessage could simply print "python3 version not ready yet, use python2" when run under python3.

[ghost]

@g1itch I don't have python2! It is not supported in any new OS. Not even in Debian 11!

I need a Debian 10 virtual machine with Gnome Boxes to run PyBitmessage. Unfortunately.

[g1itch]

@g1itch I don't have python2! It is not supported in any new OS. Not even in Debian 11!

I need a Debian 10 virtual machine with Gnome Boxes to run PyBitmessage. Unfortunately.

Well, for me Ubuntu Focal is recent enough distro and it has python2. Proper flatpack packaging will probably be enough to run current PyBitmessage code on Linux distros with no python2, though I would prefer merging qt5 code first.

[g1itch]

Then there are other minor issues, like travis2bash not understanding the full travis syntax, the non-developers not having proper feedback.

It seems that python: - "3.6" is OK for your buildbot tasks - it's simply ignored: buildbot/travis_focal succeed.

[PeterSurda]

That particular feature is supported. However the VMs don't have python 3.6 so the tests can't run. I'll add python 3.6 into the VMs and then rerun your tests, but it may take a couple of days.

[g1itch]

I'll add python 3.6 into the VMs and then rerun your tests, but it may take a couple of days.

There is no rush. If you take a close look into python 2.7 job of the PR you will see that it's not fully correct. Now I'm discovering some strange behavior of test-mode.

[ghost]

Python 3.9.1 (default, Dec  8 2020, 07:51:42) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Well, I only have Python 3.9 maybe I will flip that 9 to 6 somehow.

qt5 should be priority, just as to make sure that PyBitmessage works above Python 3.5 at least.

[g1itch]

just as to make sure that PyBitmessage works above Python 3.5 at least.

PyBitmessage currently does not support python3.

[PeterSurda]

@g1itch The VMs I use for buildbot now should support python3.6 through python3.10. No need to change anything, just put the desired version(s) into .travis.yml like you're doing in #1725. It will run all the tests for all the python versions listed.

[g1itch]

It seems I read here something you didn't write and maybe didn't even mean. Do you agree that if somebody wants to participate in the porting process, he should:

1. chose an entity: a structure, a module, a thread or a couple of them (for example objectProcessor works together with addressGenerator)
2. cover it by tests
3. when the tests pass with python2, edit the code to support python3

Furthermore, it's not related to the python3 porting process but it used to be so - in order to pass a review PR should fit the following criteria:

1. comply with PEP8
2. have one commit per change with descriptive messages
3. do not include unrelated changes
4. do not introduce unneeded extra files

The latest merge violates all the points. I'm surprised that it even passed pylint check.

[PeterSurda]

@g1itch Your proposals make sense. I appreciate you giving feedback to the other developers.

Unfortunately for personal reasons I have very little time to code review nowadays. I also didn't have any help from the developer that normally helps me in this process as she had time off for personal reasons too (for about 1.5 months). At the same time however, I'm paying developers to work on the project, and they either need feedback or their code merged, or I pay them to do nothing. The project is in an annoying phase when python2 compatibility is being dropped by more and more distros and dependencies, but python3 code isn't working sufficiently well to have the whole program run.

For these reasons I approved PRs I otherwise wouldn't have. Hopefully, now more python3 code can be worked on, and some tests that require full app can be refactored or created from scratch using mocks. Existing code can be cleaned up. And developers can work without having to wait for each other's PRs to be merged, so development can work in parallel.

I should have more time starting in about 3-4 weeks, around the end of May. Perhaps then we can make the process more strict again.

[g1itch]

Meanwhile the official doc explicitly mentions test coverage and the bad practice of using sys.version_info: https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection

As for requirements, you can separate them by version using environment markers e.g.

diff --git a/requirements.txt b/requirements.txt
index 899753cb..65fafe75 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,6 @@
 coverage
 python_prctl;platform_system=="Linux"
+PyQt5;python_version>="3.8"
 psutil
 pycrypto
 six

[tuxayo]

Is there a branch/repo/milestone to follow the python3 port?

[cornfeedhobo]

@g1itch are you still working on this?

[Neustradamus]

Have you progressed on it?

[cornfeedhobo]

After revisiting this 9 months later, I think it's safe to say that bitmessage is dying. There are few distros that still package pyqt4, leaving this to be deployable to few. Unless someone decides to take this on, probably best to archive your keys and move your chats elsewhere.

[PeterSurda]

The development is currently paused but should resume in a couple of months. While running from source may be problematic, you can use the binaries (like appimage) until the python3 port is done.

[cornfeedhobo]

@PeterSurda Is there a link to the AppImage that's published from an official build source, preferably signed?

[PeterSurda]

This is the latest one: https://appimage.bitmessage.org/releases/20230116/ . I understand it's a bit of indirect. Automated signing isn't implemented at the moment.

[cornfeedhobo]

@PeterSurda Thanks for following up. Good luck with the python3 migration. I'll check back in at the end of the year :)