From 957b0823245fe88c9017bb2453dc0e2c2ba94879 Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Mon, 28 Apr 2025 11:46:40 -0600 Subject: [PATCH 1/9] Update installation page --- docs/install.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/install.rst b/docs/install.rst index f3d964a3f..aa5f959da 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -76,7 +76,7 @@ We recommend the following directory structure for Pavilion. The ``/pavilion/config`` directory will contain all of your site specific tests, plugins, and test source files. If you organize your tests in separate repositories, you may check them out here, and simply link the -appropriate files into ``config/tests`` and ``config/test_src``. +appropriate files into ``config/suites``. Alternatively, you can make the entire config directory, source tarballs and all, its own repository. From af18f785cc901c2d35b15ed0e1d0a9e5270e38a6 Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Mon, 28 Apr 2025 11:58:54 -0600 Subject: [PATCH 2/9] Update getting started page --- docs/basics.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/basics.rst b/docs/basics.rst index 5c49e120c..361bfac53 100644 --- a/docs/basics.rst +++ b/docs/basics.rst @@ -77,8 +77,7 @@ Pavilion doesn't come with any tests itself (though it does come with example tests in ``examples/``); it's just a system for running them on HPC clusters. Each test needs a configuration script, and most will need some source files. Both of these will live in one of -your :ref:`config.config_dirs` under the ``tests/`` and ``test_src/`` -sub-directories. +your :ref:`config.config_dirs` under the ``suites/`` sub-directory. Test configs tell pavilion what environment it needs to build and run your test, the commands to run it, how to schedule it on a cluster, and @@ -144,6 +143,11 @@ current host, which is provided via a built-in providing your own sys\_var plugin, typically to use more colloquial names for systems. +.. note:: + It is also possible to specify host configs on a per-suite basis using a slightly + different format. For more information on suite-specific host configs, see the + section on :ref:`tests.suites`. + Running tests ~~~~~~~~~~~~~ @@ -165,7 +169,7 @@ suite). Did you forget what you named them? That's ok! Just ask Pavilion. 1 tests started as test series s33. If you want to run every test in the suite, you can just give the suite -name. You can also run whatever combinations of tests you want. You also +name. You can also run whatever combinations of tests you want. You can also list tests in a file and have Pavilion read that. .. code:: bash @@ -183,7 +187,7 @@ Test Status ^^^^^^^^^^^ If you want to know what's going on with your tests, just use the -``pav status`` command. +``pav status`` command. .. code:: bash @@ -201,7 +205,7 @@ Test Series and ID's ~~~~~~~~~~~~~~~~~~~~ From the above, you may have noticed that each test gets a series id -like ``s24`` and a test id like ``41``. You can use these id's to +like ``s24`` and a test id like ``41``. You can use these ID's to reference tests or suites of tests to get their status, results, and logs through the pavilion interface. The ID's are unique for a given Pavilion :ref:`config.working_dir` but they will @@ -214,6 +218,9 @@ Pavilion builds a mapping of result keys and values for every test that runs. You can view the results of any tests using the ``pav results`` command. +runs. You can view the results of any tests using the ``pav results`` +command. + .. code:: bash $ pav results @@ -239,7 +246,7 @@ Every test has a results object that contains a variety of useful, automatically populated keys. Additional keys can be defined through :ref:`result parsing and result evaluations `. -Results are saved alongside each test, as well being written to a +Results are saved alongside each test, as well as being written to a central result log that is suitable for import into Splunk or other tools. From a91a68eba7589f19fd6919efc11a4d57e1af0bc2 Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Mon, 28 Apr 2025 12:07:29 -0600 Subject: [PATCH 3/9] Update advanced features page --- docs/advanced.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/advanced.rst b/docs/advanced.rst index 94c68dd94..d88f7d6de 100644 --- a/docs/advanced.rst +++ b/docs/advanced.rst @@ -16,8 +16,8 @@ In addition to host config files, you can provide mode config files that you can apply to any test when you run it. They have the same format as the host configs, but multiple can be provided per test. -Unlike host configs, mode configs apply _last_ in the process, overriding -values set be host configs and the test itself. +Unlike host configs, mode configs apply *last* in the process, overriding +values set by host configs and the test itself. For example, the following mode file could be used to set a particular set of slurm vars: @@ -38,7 +38,7 @@ Test Series A test series is a well defined group of tests that are designated to be run together. A series is created automatically whenever you use ``pav run``, but you can also explicitly define series using a -``configs/series/.yaml`` file. This allows you defined +``configs/series/.yaml`` file. This allows you to define relationships and dependencies between the tests, among other things. See :ref:`tutorials.series`. From 61e5ecb222d3a6d94786c762c3b5f013f565907d Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 14:42:10 -0600 Subject: [PATCH 4/9] Minor edits to configuration page --- docs/config.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/config.rst b/docs/config.rst index bd8608766..d1fffb92b 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -21,10 +21,10 @@ Finding the Pavilion.yaml file Pavilion looks for its main ``pavilion.yaml`` config in the following hierarchy and uses the first one it finds. -- The user's `~/.pavilion` directory. +- The user's ``~/.pavilion`` directory. - The directory given via the ``PAV_CONFIG_DIR`` environment variable. -The pavilion.yaml file can configure additional locations to look for test, +The ``pavilion.yaml`` file can configure additional locations to look for test, mode, and host configs, as well as plugins using the ``config_dirs`` option. The ``~/.pavilion`` directory is only searched for ``pavilion.yaml`` by default, but searches for other configs there can be turned on in ``pavilion From 4945313eb6bd5ecf7cbc78b8dd0782b65f12242d Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 14:56:27 -0600 Subject: [PATCH 5/9] Minor edits to test run lifecycle page --- docs/test_run_lifecycle.rst | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/test_run_lifecycle.rst b/docs/test_run_lifecycle.rst index 62ce56693..8ee776396 100644 --- a/docs/test_run_lifecycle.rst +++ b/docs/test_run_lifecycle.rst @@ -17,8 +17,8 @@ You'll often run a test using a test command like this: $ pav run mytests.last_login -In this case, we have a ``mytests.yaml`` test suite file in our tests -directory, that looks like this: +In this case, we have a ``mytests.yaml`` test suite file in our ``suites`` +directory that looks like this: .. code-block:: yaml @@ -37,7 +37,7 @@ directory, that looks like this: # Five hours in seconds time_limit: '{{ 60^2 * hours }}' - # Always skip gary. + # Always skip jerry. not_if: '{{user}}': 'jerry' @@ -109,7 +109,7 @@ Finding Suite, Host, and Mode Configs Pavilion can be configured to look in multiple places for test configs (see :ref:`config.config_dirs`), and uses the first matching file found. In -case, we need a suite config named ``tests/mytests.yaml`` and a host config +our case, we need a suite config named ``suites/mytests.yaml`` and a host config named ``hosts/tester.yaml``. Pavilion will load the yaml from each of these files, and use it to construct our raw test config. @@ -239,7 +239,7 @@ permutations, and generates test run objects and directories. 1. The available variable values are collected for each of the four variable types and put in a single *variable manager* for each test (:ref:`tests.variables`). -2. These, along with the ``permute_on`` value for a test, is used to compute +2. These, along with the ``permute_on`` value for a test, are used to compute a unique collection of variable values for each :ref:`Test Permutation`. Each of these will result in a separate *Test Run*. @@ -301,7 +301,7 @@ Deferred Variable Errors ######################## You may also see errors involving :ref:`tests.variables.deferred`. Some -sections, like 'build' and scheduler configuration sections, don't allow them. +sections, like ``build`` and ``scheduler`` configuration sections, don't allow them. .. code-block:: yaml @@ -329,8 +329,8 @@ that object and run the test. :alt: Creating a Test Run 1. The *Test Run* object is created from the config, which immediately grabs - the next available test_id number. The test run directory is then created - in a directory named for that number under `/test_runs/`. + the next available test ID number. The test run directory is then created + in a directory named for that number under ``/test_runs/``. 2. Everything needed to create the test run object is saved to the test's run directory, including the config, test variables, and any other attributes of the test. @@ -365,7 +365,7 @@ The following steps will be taken: 1) A ``kickoff`` script will be generated by the scheduler for each test run. - 1) The kickoff script will run `pav _run ` and set up + 1) The kickoff script will run ``pav _run `` and set up the basic Pavilion environment. #) The extension of the kickoff script is scheduler dependent. 2) The scheduler plugin will run the kickoff script such that @@ -398,9 +398,9 @@ Generating the Run Script Run scripts are generated mostly identically to build scripts, and consist of the same basic components. -1) Manage modules as described in the 'run.modules' options. -2) Manipulate environment variables as set in the 'run.env' options. -3) Run all the commands in `run.cmds`. +1) Manage modules as described in the ``run.modules`` options. +2) Manipulate environment variables as set in the ``run.env`` options. +3) Run all the commands in ``run.cmds``. For example: @@ -441,7 +441,7 @@ At this point, Pavilion simply runs the test's ``run.sh`` script. Like with building, Pavilion will only timeout a test if it doesn't produce output at least once every ``run.timeout`` seconds. -The return value from the ``run.sh`` script is saved in the 'return_value' +The return value from the ``run.sh`` script is saved in the ``return_value`` result, and may be used later when we gather the test results. If this step seems overly simple, it is! Most of the work running Pavilion @@ -455,6 +455,6 @@ This is described fully in :ref:`results.basics`. Mark the Test as Complete ~~~~~~~~~~~~~~~~~~~~~~~~~ -Finally, the test is marked as complete by saving a 'RUN_COMPLETE' file in the +Finally, the test is marked as complete by saving a ``RUN_COMPLETE`` file in the test's run directory. Pavilion uses this to quickly determine which tests -might still be running. \ No newline at end of file +might still be running. From 4323612837c59f54cb1ffa0a6e8af832cb433105 Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 15:17:56 -0600 Subject: [PATCH 6/9] Minor edits to test format page --- docs/tests/format.rst | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/tests/format.rst b/docs/tests/format.rst index fb024d7e6..709243c2c 100644 --- a/docs/tests/format.rst +++ b/docs/tests/format.rst @@ -11,7 +11,7 @@ This page contains in-depth documentation on the test format. Tests and Suites ---------------- -Each Suite is a yaml file (with a ``.yaml`` extension) which can contain +Each suite is a yaml file (with a ``.yaml`` extension) which can contain multiple tests. Suite files must reside in ``/suites/``, where ```` is one of your :ref:`config.config_dirs`. Tests in a suite can be run as a group or independently, and can even inherit @@ -136,10 +136,10 @@ All Pavilion (non-structural) test config values are interpreted as strings. YAML provides several different data types, but Pavilion forcibly -converts all of them to strings. The bool True becomes "True", 5 becomes -the string "5", and so on. This done mostly because it enables Pavilion +converts all of them to strings. The bool ``True`` becomes ``"True"``, ``5`` becomes +the string ``"5"``, and so on. This is done mostly because it enables Pavilion variable substitution in any config value. Some Pavilion scheduler and -result parser plugins ask for integer or other specific data types in +result parser plugins ask for integers or other specific data types in their configs. It's up to those plugins to interpret those values and report errors. @@ -196,7 +196,7 @@ Items in the config that can take a list can be extended by a list variable. Hidden Tests ------------ -Tests can be hidden by starting their name with an underscore '_' character. +Tests can be hidden by starting their name with an underscore ``_`` character. This is often useful when you have a base test that others inherit from, but the base test is never supposed to run on its own. @@ -239,6 +239,10 @@ You can specify the OS config with the ``'-o'`` option to the ``pav run``. pav run -o another_os my_tests +.. note:: + It is also possible to define host, OS, and mode configs on a per-suite + basis. See the :ref:`tests.suites` section for an overview of this feature. + Host Configs ------------ @@ -259,10 +263,6 @@ You can specify the host config with the ``-H`` option to the pav run -H another_host my_tests -.. note:: - An optional alternate format for host, OS, and mode configs exists when using suite - directories. See the following section for an overview of this format. - Format ~~~~~~ @@ -325,12 +325,12 @@ Rules of Inheritance ~~~~~~~~~~~~~~~~~~~~ 1. Every field in a test config can be inherited (except for - inherits\_from). -2. A field that takes a list (modules, cmds, etc.) are completely + ``inherits\_from``). +2. A field that takes a list (``modules``, ``cmds``, etc.) is completely overwritten by a new list by default. (In the above example, the single command in the fs test command list overwrites the entire original command list.) -3. You can add a plus to any 'list' key to extend that list with all The +3. You can add a plus to any list key to extend that list with all the given items instead. 4. A test can inherit from a test, which inherits from a test, and so on. @@ -381,17 +381,17 @@ order. 6. Any mode configs in the order specified. 2. Tests are filtered down to only those requested. -3. Command line overrides ('-c') are applied. +3. Command line overrides (``-c``) are applied. 4. Permutations are resolved. 5. Variables in the chosen scheduler config section are resolved. (You - should't have ``sched`` variables in these sections.) + shouldn't have ``sched`` variables in these sections.) 6. Variables are resolved throughout the rest of the config. This results in the semi-final test config. :ref:`tests.variables.deferred` can't be resolved until we're on the allocation. Once there, we'll finish resolving those, and resolve any parts of the config that used them. Parts of -the config that are required before kicking off the test (like the build and -scheduler sections), can't use deferred variables. +the config that are required before kicking off the test (like the ``build`` and +``schedule`` sections) can't use deferred variables. Top Level Test Config Keys -------------------------- @@ -400,8 +400,8 @@ inherits\_from ~~~~~~~~~~~~~~ Sets the test (by test base name) that this test inherits from *which must be* -*a test from this file*. The resulting test will be composed of all -keys in the test it inherits from, plus any specified in this test +*a test from the same file*. The resulting test will be composed of all +keys in the test it inherits from, plus any keys specified in this test config. See :ref:`tests.format.inheritance`. subtitle @@ -436,7 +436,7 @@ list of mappings (with the same keys) of strings. See the scheduler ~~~~~~~~~ -Sets the scheduler for this test. Defaults to 'raw'. It's recommended to +Sets the scheduler for this test. Defaults to ``raw``. It's recommended to set this in your host configs. build @@ -474,7 +474,7 @@ These sub-sections defines conditions under which tests are skipped. See :ref:`tests.skip_conditions` for the sub-section keys and usage. - +schedule ~~~~~~~~~~ Settings for controlling test scheduling. See ``pav show sched --config`` for From 5c6400bcfc85927cdf64a2f81bd96302ef713611 Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 15:51:56 -0600 Subject: [PATCH 7/9] Refine suite config page --- docs/tests/suites.rst | 50 +++++++++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/docs/tests/suites.rst b/docs/tests/suites.rst index c98dd7d10..ccb008c8b 100644 --- a/docs/tests/suites.rst +++ b/docs/tests/suites.rst @@ -13,10 +13,11 @@ This section details the organization of test suites. Suite Directories ----------------- -In addition to simple tests, Pavilion provides the option to organize suites using a directory -structure, such that the suite directory contains not only the test config, but any host, mode, -and OS configs associated with the test, as well as test source code. This provides a convenient -way of collecting the test code and data in a single location. This is useful, for instance, if the +In addition to single-file test suites, Pavilion provides the option to organize more complex +suites using a directory structure. Under this structure, each suite directory contains not only +the suite config YAML file, but also any host, mode, and OS configs associated with the suite, +along with any source code required by the suite's tests. This provides a convenient way of +collecting the test code and data in a single location. This is useful, for instance, if the user wishes to version control each test separately and use it as a Git submodule in a larger project. @@ -33,14 +34,15 @@ complexity of the suite: the ``suites`` directory. This method is useful for tests that do not require source code or other files. -2. Suites consisting of multiple files, including additional configs and test source, should be -placed under a subdirectory of ``suites``, ``suites//``. +2. Suites consisting of multiple files, including additional configs and source code, should be +placed under a subdirectory of ``suites``, ``suites//``. Arbitrary files may be placed +in this subdirectory, but the subdirectory **must** contain at minimum a suite config named +``suite.yaml``. -In the latter case, arbitrary files may be placed in the subdirectory, but it **must** contain a -suite config named ``suite.yaml``. In both cases, the suite config follows the format described in -the Test Format section. Note the difference in file name between the two organization methods: in the -first, the name of the file is the name of the suite; in the second, the file name is the generic -``suite.yaml``, and Pavilion derives the suite name from its containing directory. +In both cases, the suite config follows the format described in the :ref:`tests.format` section. +Note the difference in file name between the two organization methods: in the first, the name of +the file is the name of the suite; in the second, the file name is the generic ``suite.yaml``, and +Pavilion derives the suite name from the file's parent directory. .. admonition:: Deprecation Warning :class: warning @@ -57,16 +59,22 @@ Host, OS, and Mode Configs -------------------------- When using the first organization method, host, OS, and mode configs must be placed under their -respective subdirectories under the user's config directory and must be named with the name of -their associated host, operating system, or mode. - -When using the second, suite directory method of organization, auxiliary configs must be placed -in the suite directory alongside the suite config and must be named ``hosts.yaml``, ``os.yaml``, or -``modes.yaml`` according to their config type. **Note that these file names are plural**. These -configs override configs placed in the ``hosts``, ``os``, and ``modes`` subdirectories under the -configuration directories. - -The suite directory method of organization allows for more flexibility in these auxiliary configs. +respective subdirectories (``hosts``, ``os``, or ``modes``) under the user's config directory +and must be named according to their associated host, operating system, or mode. + +When using the second, suite directory method of organization, suite-specific auxiliary configs may +optionally be placed in the suite directory alongside the suite config itself. These configs may be +used to define test-specific options and variables that vary by host or OS, without cluttering +global host, OS, or mode configs with test-specific variables. If used, these files must be named +``hosts.yaml``, ``os.yaml``, or ``modes.yaml``, depending on their config type. **Note that these +file names are plural**. These configs override configs placed in the ``hosts``, ``os``, and +``modes`` subdirectories under the configuration directories. + +This method is especially useful in combination with the submodule-based structure described above, +since it allows host-specific test options and variables to be version controlled along with the +suite config. + +The directory method of organization allows for more flexibility in these auxiliary configs. Specifically, multiple hosts, OSs, or modes may be specified in a config file. A single host, OS, or mode can be selected on test invocation by passing the name of the host, OS, or mode to the appropriate flag (either ``-H``, ``-o``, or ``-m``). The following is an example of a host file From 97d10c64c55b79fe9010bd2d9b3a90e6bb5a431c Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 16:14:48 -0600 Subject: [PATCH 8/9] Further refine suite organization page --- docs/tests/suites.rst | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/tests/suites.rst b/docs/tests/suites.rst index ccb008c8b..874e4d67f 100644 --- a/docs/tests/suites.rst +++ b/docs/tests/suites.rst @@ -65,20 +65,22 @@ and must be named according to their associated host, operating system, or mode. When using the second, suite directory method of organization, suite-specific auxiliary configs may optionally be placed in the suite directory alongside the suite config itself. These configs may be used to define test-specific options and variables that vary by host or OS, without cluttering -global host, OS, or mode configs with test-specific variables. If used, these files must be named -``hosts.yaml``, ``os.yaml``, or ``modes.yaml``, depending on their config type. **Note that these -file names are plural**. These configs override configs placed in the ``hosts``, ``os``, and -``modes`` subdirectories under the configuration directories. +global host, OS, or mode configs with test-specific variables. These configs extend, rather than +override, configs placed in the ``hosts``, ``os``, and ``modes`` directories. -This method is especially useful in combination with the submodule-based structure described above, -since it allows host-specific test options and variables to be version controlled along with the -suite config. +This method is especially useful in combination with the submodule-based test structure described +above, since it allows host-specific test options and variables to be version controlled along with +the suite config while maintaining independence from global configs. -The directory method of organization allows for more flexibility in these auxiliary configs. -Specifically, multiple hosts, OSs, or modes may be specified in a config file. A single host, OS, -or mode can be selected on test invocation by passing the name of the host, OS, or mode to the -appropriate flag (either ``-H``, ``-o``, or ``-m``). The following is an example of a host file -containing multiple host entries (OS and mode formats are analogous): +If used, these suite-specific configs must be named ``hosts.yaml``, ``os.yaml``, or ``modes.yaml``, +depending on their config type. **Note that these file names are plural**. + +The directory method of organization allows multiple hosts, OSs, or modes to be specified in a +single config file. Pavilion discovers host and OS config files automatically based on the detected +host and OS, but specific hosts and OSs, as well as modes, can be selected on test invocation by +passing the name of the host, OS, or mode to ``pav run`` using the appropriate flag (either ``-H``, +``-o``, or ``-m``). The following is an example of a host file containing multiple host entries (OS +and mode formats are analogous): .. code-block:: yaml @@ -93,5 +95,7 @@ containing multiple host entries (OS and mode formats are analogous): foo: bar Note that this format differs from the format used by configs located in the ``hosts``, ``os``, and -``modes`` directories; in this case, individual configs within each config file derive their names -from top-level keys rather than from the name of the config file. +``modes`` directories; in this case, the config file consists of a series of key–value pairs, where +the keys are the names of hosts (or OSs or modes) and the values are individual host configs. This +is necessary even in configs containing only a single host, since configs derive their names from +the top-level keys. From f05d2c9903a2cf1c2dd4d5ef29dfafae189d085b Mon Sep 17 00:00:00 2001 From: Hank Wikle Date: Wed, 30 Apr 2025 16:29:44 -0600 Subject: [PATCH 9/9] Refine building tests page --- docs/tests/build.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/tests/build.rst b/docs/tests/build.rst index e893a188a..b73b14c54 100644 --- a/docs/tests/build.rst +++ b/docs/tests/build.rst @@ -72,9 +72,13 @@ This build option tells Pavilion when to download test source. Finding Source Files ~~~~~~~~~~~~~~~~~~~~ -The primary way to specify source in Pavilion is with the ``source_path`` -build option. Additionally you can bring additional files into your build -directory with the ``extra files`` option. +If a suite is organized as directory, all references to files within a test are interpreted as +relative to the suite directory. If your tests requires source code or other files, this is the +recommended method of organizing your test suite. + +It is also possible to use source code outside of the suite directory using the ``source_path`` +build option. Additionally you can bring additional files into your build directory with the +``extra files`` option. .. code:: yaml