Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mmtests update #469

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open

mmtests update #469

wants to merge 22 commits into from

Conversation

manul7
Copy link

@manul7 manul7 commented Oct 11, 2023 •
edited
Loading

I've updated shell script for launching mmtests as well as YAML config. New version gets multiple fixes and new option to control mmtest execution.

Tests

  1. Clean SUT, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 1 -s

Expected result

...
INFO: Installation skipped
+ pushd /mmtests
./mmtests.sh: line 283: pushd: /mmtests: No such file or directory
+ exit 1
  1. Clean SUT, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 1

Expected results

  1. Sys deps installed
  2. Perl deps installed
  3. MMTests source code checked out into /mmtests
  4. Target benchmark built
  5. Benchmark launched 1 time
  6. Results collected into /mmtests/output
...+ mv /tmp/tmp.aVWr6ZuULI /mmtests/output/BENCHMARKsqlite_CONFIGdb-sqlite-insert-small.json
  1. mmtests installed, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 3 -s

Expected results

  1. Deps installation skipped
  2. Source code checkout skipped
  3. Benchmark launched 3 times
  4. Collects results into /mmtests/output
+ mv /tmp/tmp.xHGS3Xh0sx /mmtests/output/BENCHMARKsqlite_CONFIGdb-sqlite-insert-small.json
  1. 3 iterations in /mmtests/work/log/config-db-sqlite-insert-small/
root@4d19d5392b66:/opt/linux/mmtests# ls /mmtests/work/log/config-db-sqlite-insert-small/
iter-0  iter-1  iter-2

@manul7 manul7 changed the title mmtests update WIP: mmtests update Oct 12, 2023
@manul7 manul7 force-pushed the master branch 6 times, most recently from 7095805 to fe2b2af Compare October 12, 2023 15:00
@manul7 manul7 changed the title WIP: mmtests update mmtests update Oct 12, 2023
@roxell
Copy link
Collaborator

roxell commented Oct 12, 2023

patch 77113cf ("mmtests.yml: parameters update") has multple changes that can be splitted up into multiple commits maybe or?

@manul7
Copy link
Author

manul7 commented Oct 13, 2023

patch 77113cf ("mmtests.yml: parameters update") has multple changes that can be splitted up into multiple commits maybe or?

Sure, np. Done

@manul7 manul7 force-pushed the master branch 2 times, most recently from 5ed0be3 to cbbcbfd Compare October 13, 2023 13:41
roxell
roxell reviewed Oct 17, 2023
View reviewed changes
automated/linux/mmtests/mmtests.yaml Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.yaml Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
@manul7 manul7 force-pushed the master branch 2 times, most recently from a24ed3d to a8d9917 Compare October 17, 2023 11:52
@manul7 manul7 force-pushed the master branch 9 times, most recently from 0b2bf52 to ea25796 Compare October 21, 2023 20:16
@manul7 manul7 changed the title mmtests update WIP: mmtests update Oct 22, 2023
@manul7 manul7 force-pushed the master branch 2 times, most recently from cf4e098 to 38bcda8 Compare October 26, 2023 15:21
@manul7 manul7 changed the title WIP: mmtests update mmtests update Oct 26, 2023
@manul7 manul7 marked this pull request as draft February 14, 2024 14:48
@manul7 manul7 force-pushed the master branch 2 times, most recently from 43f7c69 to 2ab73ba Compare February 29, 2024 15:48
Konstantin Belov added 10 commits March 5, 2024 19:51
automated: linux: mmtests: removed results processing
eeb630f 
Removed all parts related to results extraction and processing.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored parameters
8dbbcbd 
Moved parameters set up logic after script arguments processing.
This allows to calculate required parameters according arguments.
Added default values to parameters.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored perl deps installation
dfeee68 
Moved MMTest Perl dependencies installation into separate function.
This allows to control these deps independently from system packages

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored prepare_system
9a6c19b 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored run_test
8ac45b6 
Refactored function sets MMTEST_ITERATIONS, disables packages
installation before benchmarking procedure, increases priority.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored installation phase
b344b49 
Now operations, which required only while preparation are grouped
under installation branch and can be skipped if system has all
dependencies and binaries.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored help
fa01ec2 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: add results collection
a6d0953 
Now, by default MMTests results remain untouched after test run.
To collect results and system information, use -k option.

Added helper script collect.py, which collects system information, environment
variables, extracts results from raw MMTests results. Extracted and collected
information stored in output directory in JSON format.

If needed full archive with raw MMTests results can be created with -f option.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: test definition update
9ffd599 
Test definition was updated to accommodate recent changes in
mmtests.sh.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: results retrival
64b1134 
Signed-off-by: Konstantin Belov <[email protected]>
Konstantin Belov added 12 commits March 6, 2024 17:47
automated: linux: mmtests: new Perl dependency
b68d17b 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
13b4169 
Added SHA256 collection for kernel binary
Refactored kernel info func

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
06d76ce 
Added cloud instance type collection

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
e1b9d95 
Added benchmark's archive SHA256 collection

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
28515ad 
Packages list simplified

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: main script update
de1c099 
Removed explicit 'set +x' command to reduce output.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: added monitors control arg
4808317 
Added new argument -m to control monitors function
while benchmarking.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: added results dir arg
a6c7a2b 
Added new argument -w to control output dir location.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: checks & logs
f109cab 
Added additional check for config presence
Added clearer log messages

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: tasksel
3598083 
Added tasksel call for benchmark launch to
bind benchmark to single cpu. This allows to reduce
variance of the results.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: control file update
610ccca 
Added monitors control in test definition file

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: reworked parse_boottime
7cb9a4d 
New implementation is more robust and covers s, ms, us

Signed-off-by: Konstantin Belov <[email protected]>
@roxell roxell marked this pull request as ready for review August 2, 2024 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roxell roxell roxell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@manul7 @roxell