FileCheck is an LLVM utility that works by running a user-specified command (typically, a compiler pass through the dynamatic-opt tool) on each unit test present in a file and checking the output of the command (printed on stdout) against a pre-generated expected output expressed as a sequence of CHECK*: ... assertions. Test files are made up one or more unit tests that are each checked independently of the others. Each unit test is considered passed if and only if the output of the command matches the output contained in its associated CHECK assertions. The file is considered passed if and only if all unit tests contained within it passed.
We give an example test file (modeled after the real unit tests for the constant pushing pass located at test/Transforms/push-constants.mlir) and explain its content below.
// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
// RUN: dynamatic-opt --push-constants %s --split-input-file | FileCheck %s
// CHECK-LABEL: func.func @simplePush(
// CHECK-SAME: %[[VAL_0:.*]]: i32) -> i32 {
// CHECK: %[[VAL_1:.*]] = arith.constant 10 : i32
// CHECK: %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
// CHECK: cf.cond_br %[[VAL_2]], ^bb1, ^bb2
// CHECK: ^bb1:
// CHECK: %[[VAL_3:.*]] = arith.constant 10 : i32
// CHECK: return %[[VAL_3]] : i32
// CHECK: ^bb2:
// CHECK: %[[VAL_4:.*]] = arith.constant 10 : i32
// CHECK: %[[VAL_5:.*]] = arith.subi %[[VAL_4]], %[[VAL_4]] : i32
// CHECK: return %[[VAL_5]] : i32
// CHECK: }
func.func @simplePush(%arg0: i32) -> i32 {
%c10 = arith.constant 10 : i32
%eq = arith.cmpi eq, %arg0, %c10 : i32
cf.cond_br %eq, ^bb1, ^bb2
^bb1:
return %c10 : i32
^bb2:
%sub = arith.subi %c10, %c10 : i32
return %sub : i32
}
// -----
// CHECK-LABEL: func.func @pushAndDelete(
// CHECK-SAME: %[[VAL_0:.*]]: i1) -> i32 {
// CHECK: cf.cond_br %[[VAL_0]], ^bb1, ^bb2
// CHECK: ^bb1:
// CHECK: %[[VAL_1:.*]] = arith.constant 0 : i32
// CHECK: return %[[VAL_1]] : i32
// CHECK: ^bb2:
// CHECK: %[[VAL_2:.*]] = arith.constant 1 : i32
// CHECK: return %[[VAL_2]] : i32
// CHECK: }
func.func @pushAndDelete(%arg0: i1) -> i32 {
%c0 = arith.constant 0 : i32
%c1 = arith.constant 1 : i32
cf.cond_br %arg0, ^bb1, ^bb2
^bb1:
return %c0 : i32
^bb2:
return %c1 : i32
}- The
// RUN: ...statement at the top of the file contains the command to run for each unit test (here, for eachfunc.func). At test-time, the%sis replaced by the name of the test file. Here, the Dynamatic optimizer runs the--push-constantspass on each unit test and the transformed IR (printed to stdout bydynamatic-opt) is fed toFileCheckfor verification. // -----statements separate unit tests. They are read by the--split-input-filecompiler flag (provided by theRUNcommand) which wraps each unit test into an MLIRmodulebefore feeding each module to the specified pass(es) independently of one another.- Each
func.funcmodels a standard MLIR function, with its body enclosed between curly brackets, Here, eachfunc.funcrepresents a different unit test, since the constant pushing pass operates within the body of a single function at a time. - The
CHECK-LABEL,CHECK-SAME, andCHECKassertions represent the expected output for each unit test. They use some special syntax and conventions to verify that the output of each unit test is the one we expect while allowing some cosmetic differences between the expected and actual outputs that have no impact on behavior.FileCheck's documentation explains how each assertion type is handled by the verifier. The section below explains how you can generate these assertions automatically for your own unit tests.
Unit tests are a very useful way to check the behavior of a specific part of the codebase, for example, a transformation pass. They allow us to verify that the code produces the right result in small, specific, and controlled scenarios that ideally fully cover the design under test (DUT). Furthermore, unit tests are very easy to write and maintain with the FileCheck LLVM utility, making them a requirement when contributing non-trivial code to the project. We go into how to write you own unit tests and automatically generate FileCheck annotations (i.e., CHECK assertions) for them below.
As their name suggests, unit tests are meant to test one unit of functionality. Typically, this means that the DUT must be as minimal as possible while remaining practical to analyze (e.g., there is no need to test each individual function). In most cases this translates to testing a single compiler pass in isolation, for example, the constant pushing (--push-constants) pass. Each unit test should aim, as much as possible, to evaluate a single behavior of the DUT. Consequently, it is good practice to make unit tests as small as possible for testing for a desired functionality. Doing so makes it easier for future readers to understand (1) what behavior the unit test checks for and (2) where to look in the code if a test starts failing.
TODO | Formalize list of unit tests to have for a pass, an operation, etc.
Once you have written your own unit tests, all that remains to do is generate FileCheck annotations that will allow the latter to verify that the output of the DUT matches the expected one. Let's take the example test file given above without FileCheck annotations as an example and go through the process of generating assertions for its two unit tests. We start from a test file containing only the input code that will go through the constant pushing pass as well as a // ----- marker to later instruct the Dynamatic optimizer to split the file into separarte MLIR modules in this location.
func.func @simplePush(%arg0: i32) -> i32 {
%c10 = arith.constant 10 : i32
%eq = arith.cmpi eq, %arg0, %c10 : i32
cf.cond_br %eq, ^bb1, ^bb2
^bb1:
return %c10 : i32
^bb2:
%sub = arith.subi %c10, %c10 : i32
return %sub : i32
}
// -----
func.func @pushAndDelete(%arg0: i1) -> i32 {
%c0 = arith.constant 0 : i32
%c1 = arith.constant 1 : i32
cf.cond_br %arg0, ^bb1, ^bb2
^bb1:
return %c0 : i32
^bb2:
return %c1 : i32
}Test files need to be located in the the test folder of the repository. Constant pushing is a transformation pass, so we store it as test/Transforms/example.mlir.
From the top level of the repository, assuming you have already built the project, you can now run:
./build/bin/dynamatic-opt test/Transforms/example.mlir --push-constants --split-input-file | circt/llvm/mlir/utils/generate-test-checks.py --source=test/Transforms/example.mlir --source_delim_regex="func.func"Let's break this command down, token by token:
./build/bin/dynamatic-optruns any (sequence of) compiler pass(es) defined by Dynamatic on a source MLIR file passed as argument and prints the transformed IR on standard output.test/Transforms/example.mlirindicates the file containing the IR you want to transform using the constant pushing pass.--push-constantsinstructs the optimizer to run the constant pushing pass.--split-input-fileinstructs the compiler to wrap each piece of code separated by a line containing only// -----into an MLIR module.|pipes the standard output of the command on its left (i.e., the input code transformed by the constant pushing pass) to the standard input of the command on its right (i.e., the code to transform intoFileCheckassertions).circt/llvm/mlir/utils/generate-test-checks.pytransforms the IR it is given on standard input into a sequence ofCHECKassertions and prints them to standard output.--source=test/Transforms/example.mlirindicates the source unit test file for which assertions are being generated, and is used to print the source code of each unit test below its corresponding assertions after transformation on standard output--source_delim_regex="func.func"indicates a regex on which to split the source code. Each split of the source code will be grouped with its correspondingCHECKassertions in the output, and splits will be displayed one after the other. Here, since each standard MLIR function represents a unit test, we split on afunc.func.
After running the command, the following should be printed to standard output.
// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
// The script is designed to make adding checks to
// a test case fast, it is *not* designed to be authoritative
// about what constitutes a good test! The CHECK should be
// minimized and named to reflect the test intent.
// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
// RUN: dynamatic-opt --push-constants %s --split-input-file | FileCheck %s
// CHECK-LABEL: func.func @simplePush(
// CHECK-SAME: %[[VAL_0:.*]]: i32) -> i32 {
// CHECK: %[[VAL_1:.*]] = arith.constant 10 : i32
// CHECK: %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
// CHECK: cf.cond_br %[[VAL_2]], ^bb1, ^bb2
// CHECK: ^bb1:
// CHECK: %[[VAL_3:.*]] = arith.constant 10 : i32
// CHECK: return %[[VAL_3]] : i32
// CHECK: ^bb2:
// CHECK: %[[VAL_4:.*]] = arith.constant 10 : i32
// CHECK: %[[VAL_5:.*]] = arith.subi %[[VAL_4]], %[[VAL_4]] : i32
// CHECK: return %[[VAL_5]] : i32
// CHECK: }
func.func @simplePush(%arg0: i32) -> i32 {
%c10 = arith.constant 10 : i32
%eq = arith.cmpi eq, %arg0, %c10 : i32
cf.cond_br %eq, ^bb1, ^bb2
^bb1:
return %c10 : i32
^bb2:
%sub = arith.subi %c10, %c10 : i32
return %sub : i32
}
// -----
// CHECK-LABEL: func.func @pushAndDelete(
// CHECK-SAME: %[[VAL_0:.*]]: i1) -> i32 {
// CHECK: cf.cond_br %[[VAL_0]], ^bb1, ^bb2
// CHECK: ^bb1:
// CHECK: %[[VAL_1:.*]] = arith.constant 0 : i32
// CHECK: return %[[VAL_1]] : i32
// CHECK: ^bb2:
// CHECK: %[[VAL_2:.*]] = arith.constant 1 : i32
// CHECK: return %[[VAL_2]] : i32
// CHECK: }
func.func @pushAndDelete(%arg0: i1) -> i32 {
%c0 = arith.constant 0 : i32
%c1 = arith.constant 1 : i32
cf.cond_br %arg0, ^bb1, ^bb2
^bb1:
return %c0 : i32
^bb2:
return %c1 : i32
}It is now fundamental that you manually check the generated assertions and verify that they match the output that you expect from the DUT. Indeed, at this point no verification of any kind has happened. The previous command simply ran the constant pushing pass on each unit test and turned the resulting IR into CHECK assertions, which will from this moment forward be considered the expected output of the pass on the unit tests. At this time you are thus the verifier who needs to make sure these assertions showcase the correct and intended behavior of the DUT.
Once you are confident that the DUT's output is correct on the unit tests, you can overwrite the content of test/Transforms/example.mlir with the command output (skipping the NOTE on the first line and the following commented out paragraph). If you now go to the build directory at the top level of the repository and run ninja check-dynamatic, you unit tests should be executed, checked, and (at this point) pass.
Congratulations! You have now
- created good unit tests to make sure a part of the codebase works as intended and,
- set up an easy way for you and future developers of Dynamatic to make sure it keeps working as we move forward!
The assertion generation script (circt/llvm/mlir/utils/generate-test-checks.py) sometimes generates CHECK assertions that FileCheck is then unable to verify, even when running ninja check-dynamatic immediately after creating assertions (which, logically, should always verify). The issue arises in some cases with functions of more than two arguments and has a simple formatting fix. For example, consider the following unit test with its associated automatically generated assertions (body assertions skipped for brevity).
// CHECK-LABEL: handshake.func @duplicateLiveOut(
// CHECK-SAME: %[[VAL_0:.*]]: i1,
// CHECK-SAME: %[[VAL_1:.*]]: i32,
// CHECK-SAME: %[[VAL_2:.*]]: i32,
// CHECK-SAME: %[[VAL_3:.*]]: none, ...) -> none {
// [...]
// CHECK: }
func.func @duplicateLiveOut(%arg0: i1, %arg1: i32, %arg2: i32) {
cf.cond_br %arg0, ^bb1(%arg1, %arg2, %arg1: i32, i32, i32), ^bb1(%arg2, %arg2, %arg2: i32, i32, i32)
^bb1(%0: i32, %1: i32, %2: i32):
return
}The unit test above reports a matching error near %[[VAL_2:.*]]: i32 and fails to verify regardless of the function body assertions' correctness. Merging the second and third function argument on a single line as follows solves the issue.
// CHECK-LABEL: handshake.func @duplicateLiveOut(
// CHECK-SAME: %[[VAL_0:.*]]: i1,
// CHECK-SAME: %[[VAL_1:.*]]: i32, %[[VAL_2:.*]]: i32,
// CHECK-SAME: %[[VAL_3:.*]]: none, ...) -> none {
// [...]
// CHECK: }
func.func @duplicateLiveOut(%arg0: i1, %arg1: i32, %arg2: i32) {
cf.cond_br %arg0, ^bb1(%arg1, %arg2, %arg1: i32, i32, i32), ^bb1(%arg2, %arg2, %arg2: i32, i32, i32)
^bb1(%0: i32, %1: i32, %2: i32):
return
}