Add breakdown of tests output for integration testing framework #5825
Conversation
blakerouse
added
Team:Elastic-Agent-Control-Plane
|
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
|
This pull request does not have a backport label. Could you fix it @blakerouse? 🙏
|
|
|
mergify
bot
added
the
backport-8.x
ycombinator
requested review from
pazone
and removed request for
andrzej-stencel
|
I understand that what you added in the description of the PR is the output of the new mage target, but maybe we can discuss the structure of the output a bit since there are bits that may benefit from further polishing
- id: linux-amd64-ubuntu-2404-default
os:
type: linux
arch: amd64
version: "24.04"
distro: ubuntu
batch:
group: default
stack:
version: 9.0.0-SNAPSHOT
tests:
- name: github.com/elastic/elastic-agent/testing/integration
tests:
- name: TestAPMConfig
stack: true
- name: TestDiagnosticsOptionalValues
stack: false
...
- name: TestPackageVersion
stack: false
sudo_tests:
- name: github.com/elastic/elastic-agent/testing/integration
tests:
- name: TestBeatsServerless
stack: true
- name: TestInstallWithoutBasePath
stack: false
...
- name: TestSwitchUnprivilegedWithBasePath
stack: false
- id: kubernetes-amd64-1310-basic-kubernetes
os:
type: kubernetes
arch: amd64
version: 1.31.0
docker_variant: basic
batch:
group: kubernetes
stack:
version: 9.0.0-SNAPSHOT
tests:
- name: github.com/elastic/elastic-agent/testing/integration
tests:
How is I know that the proposed format is what the integration framework currently use internally but his may be an opportunity to clean up the structure a bit and I hope we can work a bit on that |
|
Every question you proposed is possible, just need a format to be designed that you deam acceptable to reflect your questions. Based on the way your questions are worded you have a design in-mind, please propose it. The current format is the format I designed for the internal representation of the testing framework, it makes sense to me and to me is very easy to understand. I don't think me spending time on designing a different one is going to provide the result you are looking for, as you already have one in mind. There is also another format that is a layer up that doesn't add the Please provide a recommended layout that you find more appropriate and I am sure this output can be transformed into that output. |
|
@pazone since your code will ultimately be consuming the output being produced by the code in this PR, please post the ideal output that you'd like to see. You can use the output posted in this PR's description as a starting point and take a look at the discussion in the last couple of comments as well for any ideas. Once we have your desired output, that will serve as the interface between the integration testing framework and your BK pipeline generator code, and @blakerouse can make tweaks to the code in this PR as needed to produce that output. |
|
The ideal output would be It will be consumed by The OS/ARCH/Version can be determined automatically because we run it on the target VM. This allows us to use matrix step representation. If the breakdown is also needed for other purposes, we can use a less ideal solution: And the Anyway eventually we need to have this test list |
|
@pazone We need to agree on the exact output as it will serve as the contract between the integration testing framework and your BK pipeline generator code. Could you please provide a complete sample of the ideal output, similar to what @blakerouse has provided in this PR's description? Feel free to use that as a starting point and tweak it. This way, there will be no ambiguity about the contract. Thanks. |
|
@ycombinator The ^^^ That's the contract. I can bring more context, If the tests filtering isn't clear enough. |
|
@ycombinator @pazone I have updated this PR to match the discussed format during the meeting. I have updated the output in the PR description to show the updated output. |
|
|
@blakerouse @ycombinator I'm now creating a filter for the tests using this breakdown. Thanks |
|
Do I understand correctly that |
Yes, your understanding is correct. The tests in that group are all running |
|
@blakerouse @ycombinator I need help. I'm having problems getting current OS info without introducing additional packages. Do we have anything already implemented? I need the current OS info in this format for filtering: |
@pazone The format you posted looks identical to the format in this PR's (updated) description:
So I'm not sure what you're asking for. 🤔 |
|
@ycombinator I'm trying to make it work as described here: #5825 (comment). To do that, I have to determine current OS facts:
It's not trivial to implement this without using third-party packages. The question is: Have we already implemented anything similar? Maybe I can reuse something? |
|
Ok, found this: https://github.com/elastic/go-sysinfo |
What does this PR do?
Adds a
mage integration:breakdownandmage integration:breakdownMatrixtargets that write an intermediate output of the list of operating systems and which tests need to run on each version of those operating systems.Why is it important?
Allows an easy way to see which tests run on which os/arch/distro/version combinations, along with if a stack is required and if the test needs the stack or not.
Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files[ ] I have added tests that prove my fix is effective or that my feature works[ ] I have added an entry in./changelog/fragmentsusing the changelog tool[ ] I have added an integration test or an E2E testDisruptive User Impact
None
How to test this PR locally
mage integration:breakdownor
mage integration:breakdownMatrixEnable output