Unit Tests

Unit Tests

For each model of the Buildings library, there is an example model that illustrates its use and that is used for running automatic unit test. It is recommended to run the unit tests frequently when modifying the library to make sure no model breaks as changes are being made.

Example models that can be used for unit tests are stored in the directory Buildings/[package name]/Examples, where [package name] is the name of the package that contains the model that is to be tested. The scripts that run these examples are stored in the directory Buildings/Resources/Scripts/[simulator name]/[package name]/Examples. For now, the only simulator that is supported is Dymola, but we plan to add support for other simulation environments when they support Modelica.Fluid and Modelica.Media (or the subset of these libraries that are used in the Buildings library).

The next sections describe how to run the unit tests.

Software requirements

The scripts require Python 2.6 or 2.7. Python 3.x is not yet supported. In addition to buildingspy, the Python packages http://matplotlib.sourceforge.net/ and http://numpy.scipy.org/ are required.

Executing the unit tests

On Linux, execute

cd [path_to_your_library]/bie/BuildingsPy
source setenv.sh
cd [path_to_your_library]/bie/modelica/Buildings
../../bin/runUnitTests.py [-b]

The script setenv.sh sets the PYTHONPATH system variable to the directory BuildingsPy. The optional flag -b runs the script in a batch mode. In this mode, no user input is required if the unit test results changed. No changes in the test results will be stored, but warning messages will be written to the console.

On Windows, set the PYTHONPATH system variable to the directory BuildingsPy. Execute from a dos prompt

cd [path_to_your_library]\bie\modelica\Buildings
..\..\bin\runUnitTests.py [-b]

The script runUnitTests.py calls a python script that

1. creates temporary directories for each processors,
2. copies the directory Buildings into these temporary directories,
3. creates run scripts that run all unit tests,
4. runs these unit tests,
5. compares all variables that are plotted by an *.mos script to reference results that are stored in Buildings/Resources/ReferenceResults/[simulator name]/. a. If the results changed by more than 1E-3 absolute or relative error, a warning is written to the console and a plot with the reference results and the new results is shown to the user. Unless the script is run with the -b flag, the user is asked whether to accept the changes. b. If the results are the same, no message will be written to the console.
6. collects the dymola log files from each process,
7. writes the combined log file unitTests.log in the current directory, and exits with the message 'Unit tests completed successfully.' or with an error message.

If no errors occurred during the unit tests, then the script returns 0. Otherwise, it returns a non-zero exit value.

How to include models as part of the unit tests

To enable an automatic unit test, a model developer needs to

1. provide a Modelica model in the Examples package,
2. provide a Modelica script in Buildings/Resources/Scripts/Dymola/.../Examples. This script needs to run the model and it needs to plot results.
3. commit to git the new reference results that are generated in Buildings/Resources/ReferenceResults/Dymola when running the unit tests.

The Modelica script must be in such a way that it can be run from the top-level directory of the ''Buildings'' library. See any of the example files that are in the directories Examples of the library.

For convenience, although not required for automatic unit testing, each example model should have an entry of the form

   model Damper
    annotation(Diagram, 
      __Dymola_Commands(file="modelica://Buildings/Resources/Scripts/Dymola/Fluid/Actuators/Examples/Damper.mos" 
                             "Simulate and plot"));

This will add an item to Dymola's pull-down menu. The Modelica script needs to contain a plot command that plots model results. Only plotted results will be included in when comparing the new results with reference results. For example, the script to run the unit test for the model Damper is as follows:

simulateModel("Buildings.Fluid.Actuators.Examples.Damper", method="dassl", resultFile="Damper");
 createPlot(id = 4,
   position = {73, 9, 598, 390},
   x = "res.y",
   y = {"res.m_flow", "res.v"},
   range = {0.15, 0.6, 11.0, 7.0},
   autoscale = true,
   autoerase = true,
   autoreplot = true,
   grid = true,
   color = true,
   filename = "Damper.mat",
   leftTitleType = 1,
   bottomTitleType = 1);

The scripts need to be stored in the sub-directories of Buildings/Resources/Scripts/Dymola that mirror the library package hierarchy. For example, for the damper model, the script is stored as Buildings/Resources/Scripts/Dymola/Fluid/Actuators/Examples/Damper.mos

When running the unit tests, all variables and parameters that appear in a plot command as the y variable will be compared to reference results that are stored in the directory Buildings/Resources/ReferenceResults/[simulator name]/. If a unit test has no reference results, then the user is asked whether reference results should be stored. Also, if the simulation results are different from the reference results, the user is asked whether the new results should become the reference results. If the user accepts the new results, then they will be written to the directory Buildings/Resources/ReferenceResults/[simulator name]/.

For example, in the above script Damper.mos, the line

  y = {"res.m_flow", "res.v"}

causes res.m_flow and res.v to be compared to values that are stored in Buildings/Resources/ReferenceResults/Dymola/Buildings_Fluid_Actuators_Examples_Damper.txt.

Finally, a model developer needs to commit the new reference results to git. This can be done using the command

   git add Buildings_Fluid_Actuators_Examples_Damper.txt
   git commit -m "Added new reference results." Buildings_Fluid_Actuators_Examples_Damper.txt
   git push

Note that if other reference result files changed because of the new contribution, then these other changes must only be committed if they are intentional and if the reason for the change can be explained.

To avoid introducing bugs in the library, do not commit changes that you cannot explain.