feat(doc): bao test framework #65

Diogo21Costa · 2023-07-14T13:12:26Z

PR Description

Description

This PR introduces a description of a testing framework for the Bao hypervisor project. The framework enables developers to conduct unit tests on system components and assess interactions between different layers by leveraging a Python-based test framework API that handles build system interaction, test setup generation, test runner invocation, and result logging.

Changes Made

Provided an overview of the framework's concepts, configuration options, and Nix recipes.
Explained the test definition process using the BAO_TEST macro and integrated assertion functions.
Recommended directory structure and integration instructions for the build system.
Included instructions for writing tests, running tests locally, and setting up CI with GitHub Actions.

The documentation provides a comprehensive guide on the purpose, features, and usage of the framework. It includes step-by-step instructions for integrating the framework and utilizing it effectively for testing.
Please review this PR and provide your feedback.

josecm · 2023-07-26T09:31:32Z

I believe you forked your feature branch from the wrong branch. The first commit, which is not part of this PR, is not in the main branch.

danielRep

Broadly speaking, I feel that the only problem of the document is redirecting the information for a specific use-case where the test framework can be used. I would strive to broadly explain the framework without being to specific to a particular repository where it was tested or it is already applied. My comments will evidence the places where I felt that.

source/development/testing.rst

danielRep · 2023-07-26T09:40:40Z

source/development/testing.rst

+logs the test results. This typically includes information such as the test
+status (pass/fail) and any error messages.
+
+.. figure:: img/framework-overview.png


I would try to use the .svg format for the image. Despite we don't have any guidelines specific for image formats, it seems that this particular image as a really small resolution. On draw.io you can export it as an .svg file.

source/development/testing.rst

danielRep · 2023-07-26T12:37:04Z

source/development/testing.rst

+- **platform**: The target platform to perform the tests. See
+  `Appendix I <#appendix-i>`__ to check supported platforms.


Here you are being to specific. I would say something as "Please consult the documentation of the MUT to see the supporting platforms". That information seems more plausible to be located there.

danielRep · 2023-07-26T12:37:45Z

source/development/testing.rst

+  `Appendix I <#appendix-i>`__ to check supported platforms.
+
+- **nix_file**: A nix recipe that describes the software stack to be used.
+  This will select a pre configured bao.config or point to one of your own.


Suggested change

This will select a pre configured bao.config or point to one of your own.

This will select a pre-configured ``.config`` or point to one of your own.

danielRep · 2023-07-26T12:38:27Z

source/development/testing.rst

+
+- **testes/suites**: The suites and tests to be run.
+
+- **log_level**: Test related configurations, e.g., log level ...etc.


Suggested change

- **log_level**: Test related configurations, e.g., log level ...etc.

- **log_level**: Test-related configurations, e.g., log level, verbosity, etc.

danielRep · 2023-07-26T12:39:34Z

source/development/testing.rst

+Appendix I
+----------
+
+.. table:: Supported Platforms
+   :widths: 25 25 50
+
+   +---------------------+-------------------+----------------------+
+   |                     |     PLATFORM      |       ARCH           |
+   +=====================+===================+======================+
+   | QEMU Aarch64 virt   | qemu-aarch64-virt |        aarch64       |
+   +---------------------+-------------------+----------------------+


Like I mentioned before, this documentation should explain generally the testing framework and not being to specific regarding a particular repository. I would remove this.

danielRep · 2023-07-26T12:42:51Z

I believe you forked your feature branch from the wrong branch. The first commit, which is not part of this PR, is not in the main branch.

Just to reinforce this @Diogo21Costa

danielRep · 2023-08-11T12:24:36Z

source/glossary.rst

@@ -34,3 +34,6 @@ Glossary
        invited by a maintainer to oversee the development of one or multiple
        subsystems in a repository. More information
        :ref:`here <contrib_roles>`.
+
+    MUT
+        Module Under Test. More information :ref:`here <bao_tests>`.


Try to give a proper definition to the term. Something as "...refers to the specific component of the system that is currently being subjected to testing to ensure its proper functionality and correctness. " Or something more clear and specific of the testing framework.

Besides, a newline must be in-place between terms or this will trigger an error on the build.

Signed-off-by: Miguel Silva <[email protected]>

Signed-off-by: Diogo21Costa <[email protected]>

Co-Authored-By: Cristiano Rodrigues <[email protected]>

Co-Authored-By: Cristiano Rodrigues <[email protected]> Co-Authored-By: miguelafsilva5 <[email protected]>

Signed-off-by: ESCristiano <[email protected]>

Added a step regarding the source files that need to be included in the makefiles. Signed-off-by: Miguel Silva <[email protected]>

Added some words that appeared mostly on code blocks. Additionally, removed a line from the 1st paragraph that was supposed to be commented. Signed-off-by: Miguel Silva <[email protected]>

Moved the bao-tests/ and software-stack (TBD)/ folders inside the tests/ tree to minimize the overhead in the parent directory. Changed the example in step 3 of the how-to guide to reflect these changes. Signed-off-by: Miguel Silva <[email protected]>

Signed-off-by: ESCristiano <[email protected]>

josecm · 2023-09-15T12:32:30Z

I believe you forked your feature branch from the wrong branch. The first commit, which is not part of this PR, is not in the main branch.

I've removed that commit and rebased this on main.

josecm · 2023-07-27T15:26:51Z

source/development/testing.rst

+hypervisor, virtual machine (VM), and virtual machine monitor (VMM)), as well
+as assess interactions between different components/layers.


So, integration tests?

Some mention to regression testing and the fact this is purpose to integrate the CI pipeline also would be welcome

josecm · 2023-07-27T15:27:27Z

source/development/testing.rst

+In order to tackle the challenge of testing new functionalities on the Bao
+hypervisor, we are actively working on the development of a comprehensive
+testing framework. The primary objective of this framework is to enable


I don't think this should be said. The documentation should be written as if the
framework is operational.It should just start with something like "The primary
objective of the Bao Test Framework".

josecm · 2023-07-27T15:30:19Z

source/development/testing.rst

+Overview
+***********
+
+The test framework API is a Python tool that acts as the control center for the


Is it an API or a tool? This is really confusing.

josecm · 2023-10-26T17:29:17Z

I believe the documentation introduced in this PR will be developed in wip/testing, right @Diogo21Costa? If so, let's close this.

Diogo21Costa · 2023-10-27T12:00:23Z

I believe the documentation introduced in this PR will be developed in wip/testing, right @Diogo21Costa? If so, let's close this.

Yes, you're right @josecm.

Diogo21Costa requested review from danielRep, josecm and miguelafsilva5 as code owners July 14, 2023 13:12

josecm force-pushed the feat/testing branch 2 times, most recently from 0c981aa to 9d12c88 Compare July 26, 2023 09:29

danielRep self-assigned this Jul 26, 2023

danielRep requested changes Jul 26, 2023

View reviewed changes

danielRep reviewed Aug 11, 2023

View reviewed changes

miguelafsilva5 and others added 18 commits September 15, 2023 12:56

feat(skeleton): initial commit with the skeleton of this section

1b6dcfd

Signed-off-by: Miguel Silva <[email protected]>

feat(skeleton): introduction draft

a26c7da

Signed-off-by: Diogo21Costa <[email protected]>

feat(skeleton): overview draft

4e3204b

Signed-off-by: Diogo21Costa <[email protected]>

feat(skeleton): concepts draft

3444bb9

Signed-off-by: Diogo21Costa <[email protected]>

feat(skeleton): test definition description draft

152fee1

Signed-off-by: Diogo21Costa <[email protected]>

feat(skeleton): test definition (include asserts)

44d2c44

Co-Authored-By: Cristiano Rodrigues <[email protected]>

feat(skeleton): directory structure

f159500

Co-Authored-By: Cristiano Rodrigues <[email protected]>

update(skeleton): fix overview figure

6748293

Co-Authored-By: Cristiano Rodrigues <[email protected]> Co-Authored-By: miguelafsilva5 <[email protected]>

feat(how-to): Add how to section and fix some lint erros

5422b06

Signed-off-by: ESCristiano <[email protected]>

Fix: Delete incorrectly committed directory.

bb48fc1

feat(how-to): Added a new step to the How to Use section

98b4e57

Added a step regarding the source files that need to be included in the makefiles. Signed-off-by: Miguel Silva <[email protected]>

fix(ci): added words to spelling_worlist

2539c06

Added some words that appeared mostly on code blocks. Additionally, removed a line from the 1st paragraph that was supposed to be commented. Signed-off-by: Miguel Silva <[email protected]>

feat(guidelines): Add contributing guidelines to test definition

681fdda

fix: Remove Nomenclature

182c5d5

Signed-off-by: ESCristiano <[email protected]>

fix: Add bao-test terms to glossary

2776f66

Signed-off-by: ESCristiano <[email protected]>

fix: Update Headings

cf6d05d

Signed-off-by: ESCristiano <[email protected]>

fix: Remove TrailingWhitespaces

23d087f

Signed-off-by: ESCristiano <[email protected]>

josecm force-pushed the feat/testing branch from 565ad87 to 23d087f Compare September 15, 2023 12:31

josecm marked this pull request as draft September 15, 2023 14:15

josecm reviewed Sep 15, 2023

View reviewed changes

josecm mentioned this pull request Oct 26, 2023

feat(doc): include bao test framework introduction #71

Open

josecm closed this Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(doc): bao test framework #65

feat(doc): bao test framework #65

Diogo21Costa commented Jul 14, 2023

josecm commented Jul 26, 2023

danielRep left a comment

danielRep Jul 26, 2023

danielRep Jul 26, 2023

danielRep Jul 26, 2023

danielRep Jul 26, 2023

danielRep Jul 26, 2023

danielRep commented Jul 26, 2023

danielRep Aug 11, 2023

danielRep Sep 25, 2023

josecm commented Sep 15, 2023 •

edited

Loading

josecm Jul 27, 2023

josecm Jul 27, 2023

josecm Jul 27, 2023

josecm commented Oct 26, 2023

Diogo21Costa commented Oct 27, 2023

		- platform: The target platform to perform the tests. See
		`Appendix I <#appendix-i>`__ to check supported platforms.

	This will select a pre configured bao.config or point to one of your own.
	This will select a pre-configured ``.config`` or point to one of your own.


		- testes/suites: The suites and tests to be run.

		- log_level: Test related configurations, e.g., log level ...etc.

	- log_level: Test related configurations, e.g., log level ...etc.
	- log_level: Test-related configurations, e.g., log level, verbosity, etc.

		hypervisor, virtual machine (VM), and virtual machine monitor (VMM)), as well
		as assess interactions between different components/layers.

feat(doc): bao test framework #65

feat(doc): bao test framework #65

Conversation

Diogo21Costa commented Jul 14, 2023

PR Description

Description

Changes Made

josecm commented Jul 26, 2023

danielRep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielRep commented Jul 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josecm commented Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josecm commented Oct 26, 2023

Diogo21Costa commented Oct 27, 2023

josecm commented Sep 15, 2023 •

edited

Loading