Making the Cartesian config parser a standalone varianter for Avocado

[pevogam]

Hi all, I have often found myself in the need to quickly generate multiple test variants and so far have found the Avocado varianter plugins insufficient in expressiveness and functionality in comparison to the Cartesian configs currently supported only in Avocado VT. If I would like to have more regular non-VT tests (read: tests not needing virtualization) and would like to use the Cartesian configuration, it requires forceful coupling with dependencies and projects. What do you think about the possibility of separating the Cartesian configuration from the Avocado VT project, perhaps as a varianter for Avocado? I won't mind help maintaining such a project and I even have some features planned that could benefit its entire use.

@luckyh, @richtja, @clebergnu, @ldoktor: What do you think about this possibility?

[richtja]

Hi @pevogam I am not really familiar with Cartesian varianter in avocado-vt, therefore I am not able to estimate how difficult is to separate the varianter from avocado-vt. From the avocado side, I don't see a problem with having a Cartesian varianter as an optional plugin and It is definitely a welcome contribution. The only thing which concerns me is that it will add new dependence to the avocado-vt, because avocado can be installed without optional plugins.

[pevogam]

Hi @richtja, considering that aexpect is a dependency of avocado with small codebase that is quite useful on its own I guess we have already gone that path for one project that is different than avocado and avocado-vt but is under the same umbrella. Of course it would be a second project of that kind and I agree with you in my preference to stay away from dependency overhead as much as possible but I guess one could argue that there are multiple varianter plugins for avocado and this is simply the varianter plugin avocado-vt uses.

[ldoktor]

I'm not much involved in Avocado anymore so let me just ask some questions but don't take my opinions for granted. My main question would be, how many users would benefit out of it and whether someone would be prepared to maintain this in the long term (I know your history, Plamen, so I don't doubt the second part).

As for the benefit, I used to be a heavy user of cartesian, it's an amazing tool and if one understands it can really make magic. Similarly with the multiplexer, it was designed to allow almost the same things only in a structured manner. Is there a specific feature you miss in multiplexer that cartesian contains? Maybe it'd be easier to port that feature and use multiplexer instead.

Anyway if you decide you are willing to maintain it (either as a part of Avocado or as a standalone project) and would dedicate that time to create it, then the only problem is the review time of the Avocado team and I can not talk about them.

[pevogam]

Hi @ldoktor and thanks for chiming in despite many of these projects mostly being distant echoes of your past. Let me elaborate on your points:

My main question would be, how many users would benefit out of it and whether someone would be prepared to maintain this in the long term (I know your history, Plamen, so I don't doubt the second part).

I hope at the very least the Avocado VT community will benefit from better integration with Avocado as its code base will be somewhat reduced and further converted into usable plugins. On our side we already don't mind maintaining this repo as our integration test suite benefits from it immensely. Additionally, we plan to use this powerful varianter for the integration tests of further products that have swappable backends and dependencies and can benefit from better test code reusability. The rest is beyond me but hopefully with more documentation, testing, and separation of concerns such a varianter could pick the interest of more people.

As for the benefit, I used to be a heavy user of cartesian, it's an amazing tool and if one understands it can really make magic. Similarly with the multiplexer, it was designed to allow almost the same things only in a structured manner. Is there a specific feature you miss in multiplexer that cartesian contains? Maybe it'd be easier to port that feature and use multiplexer instead.

I had originally first tried the YAML to Mux plugin during our migration from Autotest to Avocado (here meaning migration away from our large production Cartesian config with hundreds of integration tests) and back then I recall that I encountered quite many missing features prompting me to go the Avocado VT route. Since it was years ago and I cannot recall what the problems were I took a fresh look today and I might be wrong in my refreshing reading of the documentation about it but I see:

1. No support for AND/OR operators on restrictions (e.g. differentiating among only=fedora..x86_64, only=fedora,x86_64 as it can only translate only=fedora.x86_64 into !filter-only : /fedora/x86_64). I assume it still supports restrictions of products (multiplexed subvariants) despite there being no example for it.

2. No support for conditionals, conditional parameters, and conditional restrictions like

variants:
    - a:
        param1 = a
    - b:
        param1 = b
b:
    param2=c
    only c (...)

3. No support for regex matched parameters like driver_image1(_vm.*)? ?= virtio. While appending could be done via lists, prepending or simple replacement using some_key${prev_key} could not be done either.

4. No support for suffix and join operators which are much used in our parametric object composition but one viable example for Avocado VT would be having tests with multiple vms of different guest OS and hardware.

Anyway if you decide you are willing to maintain it (either as a part of Avocado or as a standalone project) and would dedicate that time to create it, then the only problem is the review time of the Avocado team and I can not talk about them.

Indeed I don't mind and I have already created a repo with some initial small improvements. The question is mostly what the Avocado community thinks.

[luckyh]

Hi @pevogam I think this is a very interesting proposal! Have you had the draft code already for POC?

[pevogam]

I guess I wasn't clear - I want to actively work on this plugin at least for a while and perform lots of changes/updates and I need to be able to push and deploy frequently and this is not just a suggestion for moving repo and just maintaining the current code base. This is not about reducing the quality or avoiding scrutiny and pull request / review workflow. In the past years it could take me anywhere from a few weeks to more than a year to merge a single pull request with avocado so this is not something that is compatible with the rather long list of rather big changes I am planning. But we might as well have a separate repo for the plugin under a different organization, I just thought it would be better if everything is more integrated under the avocado umbrella.

[richtja]

I understand that merging PR to Avocado, especially with the lack of reviewers, can be time-consuming and create bottlenecks in your work and IMO this is the biggest struggle of upstream development. But I just still don't understand how moving the work to other repo will help if you don't want to remove pull request / review workflow. What could help, would be to bring more people to this (minimally, for reviews), which I think is possible since this has a common goal for avocado and avocado-vt.

Of course, if you want to do pushes without PRs and have more control over the project, that is a different debate then and IIANM we don't have such repo in avocado organization for now. Therefore, it will be a much bigger organization change.

There is also a possibility to do most of the main changes in your own repo and move it to avocado for maintenance when it will be done.

[pevogam]

I understand that merging PR to Avocado, especially with the lack of reviewers, can be time-consuming and create bottlenecks in your work and IMO this is the biggest struggle of upstream development. But I just still don't understand how moving the work to other repo will help if you don't want to remove pull request / review workflow. What could help, would be to bring more people to this (minimally, for reviews), which I think is possible since this has a common goal for avocado and avocado-vt.

I agree that bringing more people to the boat is the crux of the problem, however unfortunately it remains an unsolved problem that would block development on the Cartesian config if we accept it as a counterargument. Then indeed what I mean is that if there are no reviewers in due time and all tests are passing (of which I plan to be more) I would tend to merge a pull request and block further development.

Of course, if you want to do pushes without PRs and have more control over the project, that is a different debate then and IIANM we don't have such repo in avocado organization for now. Therefore, it will be a much bigger organization change.

A separate repo within avocado must still be possible where I could be give the task of maintain and freedom to push more often and this is not something that could happen under the core avocado repo which was your originally suggested alternative.

There is also a possibility to do most of the main changes in your own repo and move it to avocado for maintenance when it will be done.

If I commit a lot of changes in a separate repo to later move for maintenance under avocado then I would also prefer (and I think it is most preferable for everyone) if I am able to maintain it also afterwards, especially considering a likely large number of changes introduced by me. So even in this case a separate repo would be better which results in next to no difference with the original proposal.

So I guess the question boils down to: separate avocado repo under the same or a different umbrella. Or did I miss something in your already quite good suggestions?

[richtja]

Yes, basically you are right. After what you said, I think the whole question is about approval policies. In Avocado, every change needs to have at lease one approval. So if we decide that we won't enforce this, then we can have this repo under the avocado umbrella. If we keep enforcing it, then I understand that you will move it somewhere else because it will slow you down.

I think it would be great to get opinion about this also from @luckyh, @clebergnu, @ldoktor.

[luckyh]

I would agree with @richtja that it'd better to comply the same review workflow as what we've set for other avocado repos, if we want the new repo to be kept in the avocado organization.

In a long-term perspective, there will be not that much new features/functionalities/unittests to be added into the Cartesian package, hence I'd confidently say it won't be a real issue which could significantly slow down the development progress if we choose such a "strict" approval policy.

However, I understood that there would be tens of code/PRs submission at the early stage of the new repo where a delayed review could be even seen as the major stumbling block. On the other hand, I do believe code reviews can help on sustaining the stability, so I had the same thought as @clebergnu commented below. I mean, we can have the only review instead of several ones, which just happens once the new package is ready for GA.

[pevogam]

One idea regarding all the module headers: the best approach we could come up with is to add copyright headers and the original authors listed when they were working on this as is done in the linux kernel. An example header from mm/memcontrol.c:

// SPDX-License-Identifier: GPL-2.0-or-later
/* memcontrol.c - Memory Controller
 *
 * Copyright IBM Corporation, 2007
 * Author Balbir Singh <[email protected]>
 *
 * Copyright 2007 OpenVZ SWsoft Inc
 * Author: Pavel Emelianov <[email protected]>
 *
 * Memory thresholds
 * Copyright (C) 2009 Nokia Corporation
 * Author: Kirill A. Shutemov
 *
 * Kernel Memory Controller
 * Copyright (C) 2012 Parallels Inc. and Google Inc.
 * Authors: Glauber Costa and Suleiman Souhlal
 *
 * Native page reclaim
 * Charge lifetime sanitation
 * Lockless page tracking & accounting
 * Unified hierarchy configuration model
 * Copyright (C) 2015 Red Hat, Inc., Johannes Weiner
 *
 * Per memcg lru locking
 * Copyright (C) 2020 Alibaba, Inc, Alex Shi
 */

[ldoktor]

Looking at cartesian_config.py I can see only :copyright: Red Hat 2008-2013. Sure there were other significant changes afterwards by RH as well as others, but I'd simply list there that it is refactored-from: https://github.com/avocado-framework/avocado-vt followed by the original copyright + your copyright with the current date.

Anyway I'm not a lawyer nor I favor licenses. But as long as you point to the original authors and keep the license I assume it should be fine.

[pevogam]

One additional idea I have once we establish enough many unit tests for acceptance testing is to convert some of the core functionality to Rust with the same python interface as exported python bindings. Any objections or affirmations are welcome and I hope this doesn't sound too far extreme even if it could sound far fetched until acceptance testing becomes possible for such a repo.

[clebergnu]

Hi @pevogam,

When it comes to the organization of the repo, it's a fair game to have a new repo under the "avocado-framework" organization, with relaxed code-review guidelines until the project matures to the point you consider it "GA".

Then, I'd recommend one of two things:

1. Make it a "first-class" Avocado plugin. The fact that the cartesian config doesn't have dependency on external Python libraries allows for that. If you're concerned about further development of the plugin, like waiting on reviews, you'd be free to count on say, a colleague from your company, to perform these reviews and still satisfy the review criteria.
2. Keep it in the "avocado-framework" organization, as a separate repo, but enforce the coding/review guidelines as the other projects do too.

What do you think?

[pevogam]

Hi Cleber! Right now I think option (1) is more flexible to the requirements of such a project. Of course if you happen to have anyone at Red Hat that would like to perform frequent reviews and spend more of their resources in this project, they are always welcome to do so and greatly appreciated. Otherwise I will look for a person within my own team that would hopefully agree to do the same so that we stay as close to Avocado's review principle as possible, even at this stage. Then of course once we reach a point of long term stability we could embrace the full code of conduct and avocado workflows. Does this sound good?

[pevogam]

In the meantime while waiting for further feedback and potential action I will push some further work to https://github.com/pevogam/cartconf. I intend to introduce some Rust conversions via acceptance testing there too.