Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes #288, #289, inference_repo issue number 1335 #298

Merged
merged 8 commits into from
May 16, 2023
Merged

Fixes #288, #289, inference_repo issue number 1335 #298

merged 8 commits into from
May 16, 2023

Conversation

arjunsuresh
Copy link
Contributor

No description provided.

@github-actions
Copy link

github-actions bot commented Apr 12, 2023 •
edited
Loading

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@arjunsuresh arjunsuresh changed the title Fixes #289, inference_repo issue number 1335 Fixes #288, #289, inference_repo issue number 1335 Apr 12, 2023
@arjunsuresh
Copy link
Contributor Author

arjunsuresh commented May 1, 2023 •
edited
Loading

I ran the checker in the inference_results_3.0 repository and below are the results.

[2023-05-01 22:28:36,662 submission_checker.py:2651 INFO] ---
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/multistream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/offline
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/singlestream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/multistream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/offline
[2023-05-01 22:28:36,666 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/singlestream
[2023-05-01 22:28:36,666 submission_checker.py:2657 INFO] ---
[2023-05-01 22:28:36,666 submission_checker.py:2658 INFO] Results=7277, NoResults=7
[2023-05-01 22:28:36,666 submission_checker.py:2661 ERROR] SUMMARY: submission has errors

Even though 7 results are failed it is actually 2 unique results (others are inferred).

Both are from the same SUT and the uncertainties are happening at the beginning and end of the loadgen testing phase run.

@psyhtest
Copy link
Contributor

psyhtest commented May 2, 2023 •
edited
Loading

Checking the first log, the testing range is set to 0.2 Amps. The warnings have the following timestamps:

  • the first 9 seconds (samples):
02-26-2023 18:46:05.257: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:06.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:07.257: WARNING: Uncertainty 1.03%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:08.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:09.258: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:10.257: WARNING: Uncertainty 1.16%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:11.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:12.257: WARNING: Uncertainty 1.10%, which is above 1.00% limit for the last sample!
02-26-2023 18:46:13.257: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!
  • one sample 15 seconds later:
02-26-2023 18:46:28.257: WARNING: Uncertainty 1.02%, which is above 1.00% limit for the last sample!
  • the last 9 seconds:
02-26-2023 18:56:31.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:32.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:33.257: WARNING: Uncertainty 1.05%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:34.258: WARNING: Uncertainty 1.05%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:35.257: WARNING: Uncertainty 1.12%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:36.257: WARNING: Uncertainty 1.13%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:37.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:38.257: WARNING: Uncertainty 1.13%, which is above 1.00% limit for the last sample!
02-26-2023 18:56:38.995: Response to client sent: Stopping untimed measurement
02-26-2023 18:56:39.258: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!

Here are the corresponding lines from the testing spl.txt:

  • the first 9 seconds:
Time,02-26-2023 18:46:05.257,Watts,2.416000,Volts,251.180000,Amps,0.035140,PF,0.273700,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:06.257,Watts,2.990000,Volts,251.220000,Amps,0.039280,PF,0.303100,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:07.257,Watts,3.069000,Volts,251.190000,Amps,0.039830,PF,0.306800,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:08.257,Watts,2.442000,Volts,251.220000,Amps,0.035410,PF,0.274500,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:09.257,Watts,2.418000,Volts,251.160000,Amps,0.035240,PF,0.273200,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:10.257,Watts,2.377000,Volts,251.210000,Amps,0.034880,PF,0.271300,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:11.257,Watts,2.439000,Volts,251.180000,Amps,0.035360,PF,0.274600,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:12.257,Watts,2.652000,Volts,251.100000,Amps,0.036870,PF,0.286400,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:46:13.257,Watts,2.425000,Volts,251.140000,Amps,0.035220,PF,0.274100,Mark,2023-02-26_18-34-49_testing
  • one sample 15 seconds later:
Time,02-26-2023 18:46:28.257,Watts,3.133000,Volts,251.410000,Amps,0.040360,PF,0.308800,Mark,2023-02-26_18-34-49_testing
  • the last 9 seconds:
Time,02-26-2023 18:56:30.257,Watts,3.369000,Volts,251.000000,Amps,0.041700,PF,0.321800,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:31.257,Watts,2.956000,Volts,250.980000,Amps,0.038670,PF,0.304500,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:32.257,Watts,2.950000,Volts,250.970000,Amps,0.038660,PF,0.304000,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:33.257,Watts,2.927000,Volts,251.070000,Amps,0.038510,PF,0.302700,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:34.257,Watts,2.895000,Volts,251.110000,Amps,0.038240,PF,0.301400,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:35.257,Watts,2.526000,Volts,251.040000,Amps,0.035730,PF,0.281600,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:36.257,Watts,2.487000,Volts,251.040000,Amps,0.035480,PF,0.279200,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:37.257,Watts,2.467000,Volts,251.020000,Amps,0.035310,PF,0.278300,Mark,2023-02-26_18-34-49_testing
Time,02-26-2023 18:56:38.257,Watts,2.492000,Volts,251.090000,Amps,0.035470,PF,0.279800,Mark,2023-02-26_18-34-49_testing

@dmiskovic-NV
Copy link
Contributor

TL;DR: If range is kept constant, lower power will have higher uncertainty

Uncertainty is sum of uncertainties that come from: set range for voltage and current (which are not scaled by measured power), measured value, power factor, and few more parasitic effects. In order to get uncertainty in %, value is divided by measured power, so it will naturally increase as power reduces

@arjunsuresh
Copy link
Contributor Author

Thank you @psyhtest for sharing the details. Actually there is a check to ensure the uncertainty reports are only considered during the loadgen run. And it is just one single sample which is failing this test for both the SUTs.

If I modify the check as follows
if start_load_time+TIME_DELTA_TOLERANCE < log_time < stop_load_time-TIME_DELTA_TOLERANCE:

it passes and the TIME_DELTA_TOLERANCE being used is 500ms. Would you recommend committing this change?

@psyhtest
Copy link
Contributor

psyhtest commented May 3, 2023

And it is just one single sample which is failing this test for both the SUTs.

Interesting. Where does this sample occur? When transitioning from idle to busy or vice versa, I guess?

@arjunsuresh arjunsuresh requested a review from a team as a code owner May 4, 2023 14:07
@arjunsuresh
Copy link
Contributor Author

yes @psyhtest it occured very close to the testing start - within 500ms interval. Just a guess - this could be due to this issue

@arjunsuresh arjunsuresh mentioned this pull request May 16, 2023
Closed
araghun
araghun approved these changes May 16, 2023
View reviewed changes
Copy link
Contributor

@araghun araghun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved in PowerWG 5/16. Pending item post merge is have additional documentation for prove changes.

@araghun araghun merged commit e3755e9 into mlcommons:master May 16, 2023
@github-actions github-actions bot locked and limited conversation to collaborators May 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@araghun araghun araghun approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@arjunsuresh @psyhtest @dmiskovic-NV @araghun