A conondrum on how to avoid having to write long make command lines

[oharboe]

To build mock-array-big, I write a long command line:

make DESIGN_CONFIG=designs/asap7/mock-array-big/config.mk

Wouldn't it be nice if I could record that option somewhere, so I can simply write make?

Indeed, there are lots of commented out code in the ORFS Makefile to do just this:

OpenROAD-flow-scripts/flow/Makefile

Line 5 in df9a442

# DESIGN_CONFIG=./designs/nangate45/aes/config.mk

This got me thinking, what if there was a settings.mk file locally, that was added to .gitignore and -include settings.mk at the top of the Makefile? I could then plonk in my settings to that file.

I coded that up in https://github.com/Pinata-Consulting/OpenROAD-flow-scripts/tree/openroad-settings, and it worked.

Except now I want to add some more variables....

mock-array-big allows several other configuration variables that can be used to modify the dimensions of the design: width and height of array in number of elements, width of bus lines/data operated on inside the elements and also the pitch(distance) between two elements in the array.

Why not allow putting those into the settings.mk too?

This worked, except, the DIE/CORE_AREA are wrong.

The problem is that the DIE/CORE_AREA must be calculated and the area depends on the values set in settings.mk. However, the evaluation of $(shell ...) happens too early, the environment variables in settings.mk are not visible to the script executed below:

OpenROAD-flow-scripts/flow/designs/asap7/mock-array-big/config.mk

Line 13 in df9a442

export CORE_AREA = $(shell python3 designs/asap7/mock-array-big/core_area.py)

After learning more about make and how variables are expanded than I ever intended to learn, I've concluded that this nearly works, but can't be made to work without some further changes to the ORFS Makefile.

What is needed is that the CORE/DIE_AREA is evaluated when the targets execute. At that point, all the environment variables are set as expected. Indeed there are many other aspects of the design that are calcualted when the target is executed and these environment variables are set. For instance, MACRO_PLACEMENT_TCL is evaluated while the environment variables are available.

So what changes could be made to ORFS to make DIE/CORE_AREA computable based upon other environment variables?

A general mechanism could be introduced where for instance DIE_AREA and CORE_AREA are evaluated when the target is executed. A strawman proposal to give an idea of how such a utility function would:

• if the string ends in .cfg, read from the .cfg file
• if the string ends in .tcl, run the tcl source file and use the stdout
• if the string ends in .py, run the py source file and use the stdout
• if the string starts with {, interpret it as .tcl and run it as an inline Tcl snippet that returns a string that is used

This may seem like a lot of bother to avoid some typing, but it's not the typing that is bothering me. The problem is that I see that when one is trying to debug some step of ORFS flow, then it is easy to forget setting a variable on the command line. This can in turn result in hours long wild goose chases...

[rovinski]

Why not just generate the config.mk that you need and then run make e.g. using a shell script? Not all problems need to be solved in Make.

Or why not pass in the necessary variables as arguments to your .py script?

[oharboe]

Why not just generate the config.mk that you need and then run make e.g. using a shell script? Not all problems need to be solved in Make.

I considered that, but I was trying to stay within a single paradigm instead of adding more string and chewing gum. It is bad enough

Or why not pass in the necessary variables as arguments to your .py script?

That won't work, for the same make reasons work with settings.mk

[maliberty]

You can also just set it as an envar if you are working on the same design repeatedly.

export DESIGN_CONFIG=designs/asap7/mock-array-big/config.mk

[gadfort]

@maliberty they are probably a little out of date, as they were built against an older version of SC. Our preference would be to use the SC scripts directly (https://github.com/siliconcompiler/siliconcompiler) and make improvements there to support different use cases. Since SC encapsulates all the run information in the design manifest, instead of in environmental variables, it just a python script that sets up all the design information and SC handles the rest.

[oharboe]

Thanks! Does siliconcompiler support artifacts out of the box?

[gadfort]

@oharboe what do you mean by artifacts? if modules, yes, you can see one way to build the child modules here: https://github.com/siliconcompiler/zerosoc/blob/main/build.py

[oharboe]

In build systems like Bazel, artifacts and hashes play a crucial role in distributing builds and enabling efficient execution on various machines, including less powerful ones. Here's a brief explanation of their use:

Artifacts: In the context of a build system, artifacts refer to the compiled or built outputs of a project, such as executable binaries, libraries, or other generated files. These artifacts are generated during the build process and are essential for running the software.

By preserving and distributing artifacts, build systems like Bazel can avoid recompiling the entire project every time a build is executed. Instead, if the artifacts are already available, the build system can simply reuse them, saving significant time and resources. This is particularly beneficial when multiple developers or machines need to build the same project or when a build is repeated frequently.

Hashes: A hash function is a mathematical algorithm that takes an input and produces a fixed-size string of characters, known as a hash value or checksum. In the context of build systems, hashes are used to uniquely identify files, dependencies, and artifacts.

Build systems like Bazel calculate hashes for input files and dependencies involved in the build process. These hashes act as fingerprints of the files and are stored in a build graph or metadata. Whenever a build is executed, the build system checks the hashes of the input files and dependencies against the stored values.

If the hashes match, it means that the files haven't changed since the last build, and the build system can skip the compilation or generation steps associated with those files. This process, known as incremental build or cache hit, helps save time by avoiding unnecessary work. Only the modified or invalidated parts of the project are rebuilt, resulting in faster build times.

By leveraging artifacts and hashes, build systems like Bazel can distribute prebuilt artifacts across multiple machines, including less powerful ones. This allows developers to fetch and reuse the artifacts, reducing the build time and enabling faster builds even on limited hardware resources. Additionally, hashes help in accurately determining which parts of the build need to be executed, further optimizing the build process.

[gadfort]

@oharboe we don't hash the input files unless it's turned on, but it's on my roadmap to support that for run control, so that is currently something that would be missing. There is a resume feature that will restart and you can always specify the steps to run directly (assuming the previous task completed successfully). If there are specific functionality you want you can always open an issue on github. We've been working to make a lot of improvements recently and we use it ourselves for our chip designs.

[oharboe]

What I find frustrating with DIE/CORE_AREA, is that although I can calculate the positions of macros using environment variables from make, I can't calculate the areas.

It would be more consistent if both DIE/CORE_AREA could be calculated.

[maliberty]

Perhaps you want to use CORE_UTILIZATION & CORE_ASPECT_RATIO instead

[oharboe]

Perhaps you want to use CORE_UTILIZATION & CORE_ASPECT_RATIO instead

That doesn't work, I want the macros to be the same width, but vary the height, so it needs to be calculated.

[maliberty]

We could have something like CORE_UTILIZATION & CORE_WIDTH as that's enough to find the height.

[oharboe]

That doesn't resolve the problem: it has to be a calculation that is resolved when the target is executed, just like for macro placement with .tcl or python files for manual placement.

[oharboe]

Aha! I missed a spot during a review, which confused me w.r.t. when environment variables are evaluated: #1084

So now, it all makes sense and works.

I articulated #1085 to implement the settings.mk feature and demonstrate on mock-array-big. Pretty neat actually when exploring configurations...

See README.md on how settings.mk is used:

https://github.com/Pinata-Consulting/OpenROAD-flow-scripts/blob/openroad-settings/flow/designs/asap7/mock-array-big/README.md