Is it time for latest release of populse in pypi?

[servoz]

Nipype has just released its latest version (1.9.0) on pypi.

The latest versions, in pypi, of the populse libraries will soon be 1 year old. There have been no extraordinary changes in the master branches of these libraries. However, I think it would be a good idea to publish the latest versions of all the libraries.

As a reminder, currently in pipy :

populse-db 2.5.1

populse-mia 2.5.2

mia-processes 2.6.0

capsul 2.5.0

soma-base 5.2.0

soma-workflow 3.2.2

For Mia, there have been some bug fixes since the last release, so I think we can move to populse-mia 2.6.0. For mia-processes, new bricks and pipelines have been introduced, which implies more than a micro change, so 2.7.0 will be fine. For the other libraries, it seems to me that a micro change could be envisaged. This would result in :

populse-db 2.5.2

populse-mia 2.6.0

mia-processes 2.7.0

capsul 2.5.1

soma-base 5.2.1

soma-workflow 3.2.3

I can handle this job.

What do you think?

[denisri]

We could, indeed, but I am completely under water these days and cannot find time for it at the moment. Either if you can manage it on your side, it's good, or we'll have to wait for a few weeks for me...

[servoz]

No problem, I can do it.

Let's say we set a deadline of the end of 2024.

So next releases will be:
populse-db 2.5.2
populse-mia 2.6.0
mia-processes 2.7.0
capsul 2.5.1
soma-base 5.2.1
soma-workflow 3.2.3

I'll manage version changes in info and tags.

[servoz]

I haven't forgotten this deployment for Populse at all.

A big work is being done on Mia. When it's finished, we'll also have to adapt mia_processes.

This is a very time-consuming job.

When everything is finished, the version change for populse_mia and mia_processes will be major (V3).

We'll then have a chance to discuss the rest of populse (due for completion in the spring).

[servoz]

Populse_mia is now compatible with the populse_db 3.0 branch.

The transition to populse_db v3 took longer than expected, mainly because I took the opportunity to thoroughly review the populse_mia codebase. In the end, this refactoring was the most time-consuming part. A few modules still need further improvements, after which I will restart the unit tests and verify compatibility with mia_processes.

Speaking of mia_processes, I think a major cleanup —similar to what was done for populse_mia— would be beneficial. However, given the time required for populse_mia, I suggest postponing it until the right moment.

Several bugs, typos, and issues introduced by the recent code changes in Mia have already been fixed, but further testing is necessary before deployment. This naturally brings us to the topic of rolling out populse_mia.

Deployment Plan

As a reminder, we had planned to deploy the next release by the end of 2024, and I had offered to handle the process. However, due to the extensive refactoring, the timeline has slipped slightly. Given the progress so far, I propose rolling out this update now, keeping in mind the upcoming full transition to populse_db v3.

Current versions on PyPI:

populse-db 2.5.1 (Released: Dec 20, 2023)

populse-mia 2.5.2 (Released: Jan 12, 2024)

mia-processes 2.6.0 (Released: Apr 10, 2024)

soma-workflow 3.2.2 (Released: Dec 19, 2023)

capsul 2.5.0 (Released: Dec 19, 2023)

soma-base 5.2.0 (Released: Dec 19, 2023)

For populse-db, populse-mia, and mia-processes, the changes aren't major, so we could consider:

populse-db → 2.6.0

populse-mia → 2.6.0

mia-processes → 2.7.0

For soma-workflow, capsul, and soma-base, I don’t anticipate major changes either. Could @denisri and/or @sapetnioc suggest appropriate new versions for these three packages?

Merging Development Branches

Once ongoing work is completed, we can merge the development branches:

populse_mia db_v3 → master

capsul populsedb3 → master

mia_processes db_v3 → master

For populse_db, the 3.0 branch has diverged significantly from master, so a simple merge may not be sufficient. What’s the plan for soma-base and soma-workflow?

Next Deployment

populse-mia → v3

populse-db → v3

mia-processes → ? (Not a major change, but what version do we use?)

capsul → ? (A v3 is planned. What do we do in the meantime?)

soma-base → ?

soma-workflow → ? (If I understand correctly, it should be deprecated with capsul v3?)

Outstanding Issue

There’s an open ticket in populse_db. It looks like a write=True should likely be changed to False — this should be reviewed.

[denisri]

Hi,
I haven't talked much lately (for a number of months, at least) because I was too busy on other topics (and also was on vacation for a month), and have not participated the move to populse_db 3...

I'm not sure what you suggest exactly: doing a first "intermediate" release with bugfixes (populse_db 2.6, populse_mia 2.6, mia_processes 2.7) then merge populse_db3 branches and do another release ?
Or merge directly ?

For me I think I'm OK with the "intermediate" release at least.
For the populse_db3 branches I haven't tested capsul much in this config but as dependencies on populse_db are quite slight I guess a small test will be OK. For populse_mia and mia_processes, I haven't tested at all at the moment, but I cannot do it in the very short term (the time I have for coding between meetings and answering mails measures in minutes per day, max, at the moment...) but I agree to do it in "not so long".
For soma-workflow, there have been a few fixes (mainly for stability for large workflows), not important API changes, but it would be good to use the latest master branch also (which has version 3.3.0).
Otherwise, capsul v3 is in stand-by for the moment, since we don't have time to actively work on it now, and are trying to finish the move to conda/pixi distributions before continuing with capsul3 development. So for now, it's not ready, and will not be for at least a few more months.

[sapetnioc]

Good job for Mia adaptation to populse_db v3. We barely use PyPI since we started to use Pixi (i.e. Conda packages). Therefore we did not plan the releases for soma-base, capsul and soma-workflow. The relealse you propose are ok for me.

For the next deployment, we must discuss to see how to take into account the change we made in Conda package organization. populse_db is still independent and called "populse-db" (people tend not to use underscore in package names). soma-base and soma-workflow are now included in the "soma-python" package (with other Python only projects) ; soma-python depends on populse-db. All other base libraries that are not only in Python are in the "soma" package (that depends on soma-python). There is an "anatomist" package (depending on soma) and one package per toolbox (such as morphologist that depends on anatomist and soma).

[servoz]

My idea in a nutshell:

1- As the last version of populse is over a year old, I propose to publish on pypi the current master branches (without taking into account the latest changes linked to populse_db V3) in order to make a final 2.x version (for Mia, populse_db and mia_processes)

2- Then, in a few weeks' time (I've still got some work to do on Mia and mia_proceses, as I'm embarking on a complete rewrite of Mia at the same time as adapting it to populse_db V3): merge the db_v3 branches with the populse_mia and mia_processes masters. But in this case, we'll also have to merge the populsedb3 branch from capsul to master and the 3.0 branch to master for populse_db. For populse_db, given the differences between master and 3,0, I don't think a simple merge will be enough, we'll have to be a bit more aggressive ....

3- Then do a release of all that (for Mia we'll be at V3 because it's a major change, the old v2 databases won't work any more). This will be an opportunity to discuss abandoning pypi. I've only followed the work on Pixi from a distance, reading the various emails on the subject. I'd be happy to receive more information on the subject.

Points 2 and 3 will come later.

What I propose is to start by dealing with point 1:

-> deploy populse masters on pypi :

populse-db → 2.6.0

populse-mia → 2.6.0

mia-processes → 2.7.0

soma-workflow -> 3.3.0

capsul -> 2.6.0

soma-base -> 5.3.0

Are you ok with this ?

[denisri]

I'm OK.

[servoz]

OK, I'm running out of time, but I promise to sort out the first point by the end of next week at the latest!

[sapetnioc]

Ok. Capsul 2 is supposed to be ready for Populse_db 3 (in a specific branch). Therefore, once Mia is ready, the 3.0 branch will become the main branch (not a merge, a branch renaming).

[servoz]

Point 1 is now complete. We now have PyPi :
populse-db 2.6.0
populse-mia 2.6.0
mia-processes 2.7.0
soma-workflow 3.3.0
capsul 2.6.0
soma-base 5.3.0

[denisri]

Great ! Thanks !

[servoz]

It's time to move on to step 2.

I can take care of that.
I suggest in order:

1- soma_workflow and soma_base: nothing to do.
2- populse_db:
- do we keep a copy of the current master branch, for example, in a branch called V2 (or whatever)?
- Merge the 3.0 branch with master
3- capsul:
- do we keep a copy of master?
- Merge the populsedb3 branch with master
4- mia_processes:
- merge db_v3 with master.
5- populse_mia:
- copy the current master into the V2 branch.
- merge db_v3 with master.
- delete db_v3.
-
It will certainly be necessary to manage the requirements of the versions to be used.
I can do it as soon as possible.
What do you think?

[sapetnioc]

1- ok
2- We will not work on populse-db 2 anymore. If we ever need to do it, we will be able to create a 2 branch from latest 2.* tag.
3 - Same as populse-db but also delete branch populsedb3 afterward
4 - Also delete db_v3 branch ?
5 - ok

[denisri]

I let @sapetnioc answer because I haven't followed the populse_db v3 very thoroughly, but from my point of view I think it's OK.

[servoz]

So if I understand correctly, for capsul and populse_db, we merge the branches into master then delete the branches (3.0 for populse_db and for populsedb3 capsul) and that's it?

[sapetnioc]

If there is a tag corresponding to the latest state of the branches (for instance a tag created during the release on PiPy) yes. If such a tag doesn't exist we should keep a 2.0 branch, just in case.

[servoz]

Point 2 is now complete.

We may need to take a look at the unit tests and other small details.

[servoz]

Unit tests for all populse now work correctly.
Since the versions on pypi are lower than the prerequisites on GitHub, I cheated by lowering them on GitHub. It's really messy, and I'll do better tomorrow!

[servoz]

Point 2 is now (correctly) completed.
In master we have:
soma-base: 5.3.1 No version requirements for populse projects
soma-workflow: 3.3.1 No version requirements for populse projects
populse_db: 3.0.0 No version requirements for populse projects
capsul: 2.6.1 Version requirement: populse_db >=3.0.0
mia_processes: 2.7.1 Version requirement: populse_db >=3.0.0 and populse_mia >=3.0.0
populse_mia: 3.0.0 Version requirement: [populse_db >=3.0.0, capsul >=2.6.0, mia_processes >=2.7.0, soma-base >=5.3.0, soma-workflow >= 3.3.0]
All populse UTs are working properly.