[Autotuner] Plotting utility + test #2394

luarss · 2024-09-29T05:16:27Z

Plotting utility
README

docs/user/InstructionsForAutoTuner.md

tools/AutoTuner/src/autotuner/utils/plot.py

docs/user/InstructionsForAutoTuner.md

vvbandeira

Please add a call to this script in the test_helper.py so we test this script.

flow/test/test_autotuner.sh

vvbandeira · 2025-01-11T16:29:48Z

tools/AutoTuner/src/autotuner/utils/plot.py

+        except Exception as e:
+            print(f"Error in {fname}: {e}")
+            continue


Should we keep a list of the files that were not processed so we can print them at the very end in order for the error not to be buried in the logs? And should the script fail silently in this case?

And should the script fail silently in this case?

Just to clarify, this exception happens when any Autotuner run does not proceed to the "finish" stage, it does not fail silently

Should we keep a list of the files that were not processed so we can print them at the very end in order for the error not to be buried in the logs?

Sure I can add that.

vvbandeira · 2025-01-11T16:30:49Z

tools/AutoTuner/src/autotuner/utils/plot.py

+    z = np.polyfit(df["timestamp"], df[key], 1)
+    p = np.poly1d(z)


z I assume is the z-axis; what is p?

z is the np array of polynomial (ax+b) coefficients, and p is the actual polynomial function.

Then let's use more descriptive names, maybe:
z --> coeff
p --> poly_func

vvbandeira · 2025-01-11T23:11:00Z

tools/AutoTuner/src/autotuner/utils/plot.py

+    plt.boxplot(df[key])
+    plt.ylabel(key)
+    plt.title(f"{key} Boxplot")
+    plt.savefig(f"images/{key}-boxplot.png")


Let's also parametrize the output folder to save images from CI into the reports folder for the design. This way it will be automatically saved to the build artifacts.

vvbandeira · 2025-01-12T16:33:41Z

flow/test/test_autotuner.sh

+    --platform ${PLATFORM} \
+    --design ${DESIGN_NAME} \


At the beginning of this script, you are casting these values to be uppercase, which does not match the path since Linux is case-sensitive. From the CI:

02:32:36 Running Autotuner plotting smoke test 02:32:36 ls: cannot access './flow/logs/SKY130HD/gcd/*/': No such file or directory 02:32:36 No experiments found for plotting 02:32:36 + exit 0

If we call the plot script and do not find data to plot, this should be an error, right?

Yup, these are 2 separate errors, will fix them.

vvbandeira · 2025-01-15T16:11:04Z

flow/test/test_autotuner.sh

@@ -20,28 +22,23 @@ python3 -m unittest tools.AutoTuner.test.smoke_test_sweep.${PLATFORM}SweepSmokeT
 echo "Running Autotuner smoke tests for --sample and --iteration."
 python3 -m unittest tools.AutoTuner.test.smoke_test_sample_iteration.${PLATFORM}SampleIterationSmokeTest.test_sample_iteration

-if [ "$PLATFORM" == "asap7" ] && [ "$DESIGN" == "gcd" ]; then
+if [ "$PLATFORM" == "asap7" ] && [ "$DESIGN_NAME" == "gcd" ]; then


The first check will never be true because you are casting to uppercase on line 14.
Would it be simpler/easier if we just use the platform name everywhere and just remove the dashes if that is not supported by the unittest Python lib?

The main concern is that I prefer name of smoke tests to be capitalized. We could also do something like asap7SampleIterationSmokeTest, but might be less readable

Perhaps a UPPERCASE_PLATFORM would make the distinction clearer?

Modifying PLATFORM or another variable already in use somewhere else in the code can easily lead to confusion. Other options could be PLATFORM_UT where UT is 'under test' or TEST_PLATFORM. That said, I don't see any issue with asap7SampleIterationSmokeTest.

vvbandeira · 2025-01-15T16:11:49Z

flow/test/test_autotuner.sh

+  echo "No experiments found for plotting"
+  exit 0


This is still exiting with a success code.

This is returning 0 because I didn't want to error out for smoke tests that are not suitable for plotting. I will modify it to only run plot.py for valid smoke tests then.

We could then change line 31 to list either manually the tests we want or use a better regex in the experiment names to differentiate whether the plot is available or not.

tools/AutoTuner/src/autotuner/utils/plot.py

Signed-off-by: Jack Luar <[email protected]>

Signed-off-by: luarss <[email protected]>

Signed-off-by: Vitor Bandeira <[email protected]>

Signed-off-by: luarss <[email protected]>

Signed-off-by: Jack Luar <[email protected]>

- run plot.py for all log files in the autotuner supported design/platforms - parameterised img_dir, so we can store all AT image runs in build artifacts - argparse interface now takes in platform, design, experiment for easier usage - chdir before running plot.py - add docstrings - more error handling Signed-off-by: Jack Luar <[email protected]>

Signed-off-by: luarss <[email protected]>

Signed-off-by: Jack Luar <[email protected]>

luarss requested a review from vvbandeira September 29, 2024 05:16

luarss self-assigned this Oct 8, 2024

luarss added the autotuner Flow autotuner label Oct 8, 2024

vvbandeira requested changes Oct 10, 2024

View reviewed changes

docs/user/InstructionsForAutoTuner.md Outdated Show resolved Hide resolved

tools/AutoTuner/src/autotuner/utils/plot.py Outdated Show resolved Hide resolved

luarss force-pushed the at_plot_util branch from 8c62a06 to 3c61f40 Compare October 10, 2024 14:05

luarss requested a review from vvbandeira October 10, 2024 14:05

vvbandeira reviewed Oct 10, 2024

View reviewed changes

docs/user/InstructionsForAutoTuner.md Outdated Show resolved Hide resolved

vvbandeira requested changes Oct 10, 2024

View reviewed changes

luarss force-pushed the at_plot_util branch from ccb1aea to 27012ae Compare October 11, 2024 14:58

luarss changed the title ~~[Autotuner] Plotting utility~~ [Autotuner] Plotting utility + test Oct 11, 2024

luarss requested a review from vvbandeira October 11, 2024 14:58

luarss marked this pull request as draft October 16, 2024 09:30

luarss marked this pull request as ready for review October 17, 2024 00:35

luarss force-pushed the at_plot_util branch from 4aca251 to 6f31b0c Compare December 24, 2024 03:42

luarss force-pushed the at_plot_util branch from 6f31b0c to 4983d9b Compare January 10, 2025 14:19

luarss marked this pull request as draft January 10, 2025 17:36

luarss force-pushed the at_plot_util branch from 4983d9b to f0f9c9c Compare January 11, 2025 07:02

luarss marked this pull request as ready for review January 11, 2025 12:55

vvbandeira requested changes Jan 11, 2025

View reviewed changes

vvbandeira reviewed Jan 11, 2025

View reviewed changes

luarss requested a review from vvbandeira January 12, 2025 14:19

vvbandeira requested changes Jan 12, 2025

View reviewed changes

luarss requested a review from vvbandeira January 14, 2025 02:44

vvbandeira requested changes Jan 15, 2025

View reviewed changes

luarss and others added 6 commits January 16, 2025 14:32

update plotting util and readme

59af71c

Signed-off-by: Jack Luar <[email protected]>

make arg required and update docs

c4b7831

Signed-off-by: luarss <[email protected]>

Update docs/user/InstructionsForAutoTuner.md

989f88c

Signed-off-by: Vitor Bandeira <[email protected]>

add one call to test plot.py

a7eeaaf

Signed-off-by: luarss <[email protected]>

add test_autotuner call, matplotlib reqs

9eb4e16

Signed-off-by: Jack Luar <[email protected]>

temporarily disable unstable tests

6c550c3

Signed-off-by: Jack Luar <[email protected]>

luarss added 8 commits January 16, 2025 14:34

print all files at the end of load_dir

e98a5b3

Signed-off-by: Jack Luar <[email protected]>

fix lowercase platform

e320d88

Signed-off-by: luarss <[email protected]>

fix syntax for string join

f3b90e6

Signed-off-by: Jack Luar <[email protected]>

fix LOWERCASE_PLATFORM

d2ee915

Signed-off-by: Jack Luar <[email protected]>

Change qor key to metric

9175606

Signed-off-by: Jack Luar <[email protected]>

PLATFORM now strictly lowercase in scripts

869e575

Signed-off-by: Jack Luar <[email protected]>

change plot.py sys exits

ad154f5

Signed-off-by: Jack Luar <[email protected]>

luarss force-pushed the at_plot_util branch from 1052380 to ad154f5 Compare January 16, 2025 14:42

Fix genReport

9a6566c

Signed-off-by: Jack Luar <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Autotuner] Plotting utility + test #2394

[Autotuner] Plotting utility + test #2394

luarss commented Sep 29, 2024

vvbandeira left a comment

vvbandeira Jan 11, 2025

luarss Jan 11, 2025 •

edited

Loading

vvbandeira Jan 11, 2025

luarss Jan 11, 2025 •

edited

Loading

vvbandeira Jan 11, 2025

vvbandeira Jan 11, 2025

vvbandeira Jan 12, 2025

vvbandeira Jan 12, 2025

luarss Jan 13, 2025

vvbandeira Jan 15, 2025

luarss Jan 16, 2025 •

edited

Loading

vvbandeira Jan 16, 2025

vvbandeira Jan 15, 2025

luarss Jan 16, 2025

vvbandeira Jan 16, 2025

[Autotuner] Plotting utility + test #2394

Are you sure you want to change the base?

[Autotuner] Plotting utility + test #2394

Conversation

luarss commented Sep 29, 2024

vvbandeira left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luarss Jan 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luarss Jan 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luarss Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luarss Jan 11, 2025 •

edited

Loading

luarss Jan 11, 2025 •

edited

Loading

luarss Jan 16, 2025 •

edited

Loading