An introduction of how to run scenario tests can be found in Scenarios Tests Guide. The current document has specific instruction to run:
- python3 or newer
- dotnet runtime 3.0 or newer
- terminal/command prompt in Admin Mode (for collecting kernel traces)
- clean state of the test machine (anti-virus scan is off and no other user program's running -- to minimize the influence of environment on the test)
These performance tests use the built runtime test directory Core_Root for the crossgen tool itself and other runtime assmblies as compilation input. Core_Root is an intermediate output from the runtime build, which contains runtime assemblies and tools.
You can skip this step if you already have Core_Root. To generate Core_Root directory, first clone dotnet/runtime repo and run:
src\tests\build.cmd Release <arch> generatelayoutonlythe instruction of building coreclr tests, which creates Core_Root directory.
If the build's successful, you should have Core_Root with the path like:
runtime\artifacts\tests\coreclr\<OS>.<Arch>.<BuildType>\Tests\Core_RootSame instruction of Scenario Tests Guide - Step 1.
Crossgen Throughput is a scenario test that measures the throughput of crossgen compilation. To be more specific, our test implicitly calls
.\crossgen.exe <assembly to compile>with other applicable arguments and measures its throughput. We will walk through crossgen compiling System.Private.Xml.dll as an example.
Ensure you've first followed the preparatory steps.
For Crossgen Throughput scenario, unlike other scenarios there's no need to run any precommand (pre.py). Just switch to the test asset directory:
cd crossgen
Now run the test, in our example we use System.Private.Xml.dll under Core_Root as the input assembly to compile, and you can replace it with other assemblies under Core_Root.
python3 test.py crossgen --core-root <path to core_root>\Core_Root --single System.Private.Xml.dllThis will run the test harness Startup Tool, which runs crossgen compilation in several iterations and measures its throughput. The result will be something like this:
[2020/09/25 09:54:48][INFO] Parsing traces\Crossgen Throughput - System.Private.Xml.etl
[2020/09/25 09:54:49][INFO] Crossgen Throughput - System.Private.Xml.dll
[2020/09/25 09:54:49][INFO] Metric |Average |Min |Max
[2020/09/25 09:54:49][INFO] ---------------|---------------|---------------|---------------
[2020/09/25 09:54:49][INFO] Process Time |7295.825 ms |7295.825 ms |7295.825 ms
[2020/09/25 09:54:49][INFO] Time on Thread |7276.019 ms |7276.019 ms |7276.019 msSame instructions as Scenario Tests Guide - Step 4.
Compared to Crossgen Throughput scenario, Crossgen2 Throughput Scenario measures more metrics, which are:
- Process Time (Throughput)
- Loading Interval
- Emitting Interval
- Jit Interval
- Compilation Interval
Steps to run Crossgen2 Throughput scenario are very similar to those of Crossgen Throughput. In addition to compilation of a single file, composite compilation is enabled in crossgen2, so the test command is different.
Ensure you've first followed the preparatory steps.
Same as Crossgen Throughput scenario, there's no need to run any precommand (pre.py). Just switch to the test asset directory:
cd crossgen2For scenario which compiles a single assembly, we use System.Private.Xml.dll as an example, you can replace it with other assembly under Core_Root:
python3 test.py crossgen2 --core-root <path to core_root>\Core_Root --single System.Private.Xml.dllFor scenario which does composite compilation, we try to compile the majority of runtime assemblies represented by framework-r2r.dll.rsp:
python3 test.py crossgen2 --core-root <path to core_root>\Core_Root --composite <repo root>/src/scenarios/crossgen2/framework-r2r.dll.rspNote that for the composite scenario, the command line can exceed the maximum length if it takes a list of paths to assemblies, so an .rsp file is used to avoid it. --composite <rsp file> option refers to a rsp file that contains a list of assemblies to compile. A sample file framework-r2r.dll.rsp can be found under crossgen2\ folder.
The test command runs the test harness Startup Tool, which runs crossgen2 compilation in several iterations and measures its throughput. The result should partially look like:
[2020/09/25 10:25:09][INFO] Merging traces\Crossgen2 Throughput - Single - System.Private.perflabkernel.etl,traces\Crossgen2 Throughput - Single - System.Private.perflabuser.etl...
[2020/09/25 10:25:11][INFO] Trace Saved to traces\Crossgen2 Throughput - Single - System.Private.etl
[2020/09/25 10:25:11][INFO] Parsing traces\Crossgen2 Throughput - Single - System.Private.etl
[2020/09/25 10:25:15][INFO] Crossgen2 Throughput - Single - System.Private.CoreLib
[2020/09/25 10:25:15][INFO] Metric |Average |Min |Max
[2020/09/25 10:25:15][INFO] ---------------------|---------------|---------------|---------------
[2020/09/25 10:25:15][INFO] Process Time |13550.728 ms |13550.728 ms |13550.728 ms
[2020/09/25 10:25:15][INFO] Loading Interval |1090.205 ms |1090.205 ms |1090.205 ms
[2020/09/25 10:25:15][INFO] Emitting Interval |1330.489 ms |1330.489 ms |1330.489 ms
[2020/09/25 10:25:15][INFO] Jit Interval |9464.402 ms |9464.402 ms |9464.402 ms
[2020/09/25 10:25:15][INFO] Compilation Interval |12827.350 ms |12827.350 ms |12827.350 mspython3 post.pyThe size on disk scenario for crossgen/crossgen2 measures the sizes of generated ready-to-run images. These tests use the precommand to generate a ready-to-run image, then run the size-on-disk tool on the containing directory.
Ensure you've first followed the preparatory steps.
cd crossgen|crossgen2
python3 pre.py crossgen|crossgen2 --core-root <path to core_root> --single System.Private.Xml.dll--single takes any framework assembly available in core_root.
For scenario which compiles a single assembly, we use System.Private.Xml.dll as an example, you can replace it with other assembly under Core_Root:
python3 test.py sod --dirs ./crossgen.outThe size-on-disk tool outputs an accounting of the file sizes under the crossgen output directory:
[2020/10/26 19:00:06][INFO] Crossgen2 Size On Disk
[2020/10/26 19:00:06][INFO] Metric |Average |Min |Max
[2020/10/26 19:00:06][INFO] -----------------------------------------|------------------|------------------|------------------
[2020/10/26 19:00:06][INFO] Crossgen2 Size On Disk |8412672.000 bytes |8412672.000 bytes |8412672.000 bytes
[2020/10/26 19:00:06][INFO] Crossgen2 Size On Disk - Count |1.000 count |1.000 count |1.000 count
[2020/10/26 19:00:06][INFO] .\crossgen\ |8412672.000 bytes |8412672.000 bytes |8412672.000 bytes
[2020/10/26 19:00:06][INFO] .\crossgen\ - Count |1.000 count |1.000 count |1.000 count
[2020/10/26 19:00:06][INFO] .\crossgen\System.Private.Xml.ni.dll |8412672.000 bytes |8412672.000 bytes |8412672.000 bytes
[2020/10/26 19:00:06][INFO] Aggregate - .dll |8412672.000 bytes |8412672.000 bytes |8412672.000 bytes
[2020/10/26 19:00:06][INFO] Aggregate - .dll - Count |1.000 count |1.000 count |1.000 countpython3 post.pyFor the purpose of quick reference, the commands can be summarized into the following matrix:
| Scenario | Asset Directory | Precommand | Testcommand | Postcommand | Supported Framework | Supported Platform |
|---|---|---|---|---|---|---|
| Crossgen Throughput | crossgen | N/A | test.py crossgen --core-root <path to Core_Root> --single <assembly name> | post.py | N/A | Windows-x64;Windows-x86 |
| Crossgen2 Throughput (single assembly) | crossgen2 | N/A | test.py crossgen2 --core-root <path to Core_Root> --single <assembly name> | post.py | N/A | Windows-x64;Linux |
| Crossgen2 Throughput (composite) | crossgen2 | N/A | test.py crossgen2 --core-root <path to Core_Root> --composite <path to .rsp> | post.py | N/A | Windows-x64;Linux |
| Crossgen Size on Disk | crossgen | pre.py crossgen --core-root <path to Core_Root> --single <assembly name> | test.py sod --dirs crossgen.out | post.py | N/A | Windows-x64;Linux |
| Crossgen2 Size on Disk | crossgen2 | pre.py crossgen2 --core-root <path to Core_Root> --single <assembly name> | test.py sod --dirs crossgen.out | post.py | N/A | Windows-x64;Linux |