VITIS-10034 Add Usage metrics for all XRT objects #7788

rbramand-xilinx · 2023-11-07T15:16:36Z

Problem solved by the commit

Added new Usage metrics logger that logs info about XRT objects like device, buffer, hw context, kernel etc
This information is collected in thread safe manner and the info is dumped in JSON format to a file after application run finishes
The info collected and spec for this solution is captured in confluence page

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

NA

How problem was solved, alternative solutions (if any) and why they were rejected

The proposed solution is explained in above confluence page

Risks (if any) associated the changes in the commit

Low

What has been tested and how, request additional testing if necessary

Tested with a simple multi threaded bandwidth application and was able to generate JSON file properly with necessary info
The following is the JSON dump generated with application that has 5 threads :

{
    "thread 0": {
        "device": {
            "device_index": "0",
            "bdf": "0000:3b:00.1",
            "bos_peak_count": "3",
            "global_bos": {
                "total_count": "1",
                "size": "67108864 bytes",
                "avg_size": "67108864 bytes",
                "peak_size": "67108864 bytes",
                "bytes_synced_to_device": "0 bytes",
                "bytes_synced_from_device": "0 bytes"
            },
            "hw_context": [
                {
                    "id": "0",
                    "xclbin_uuid": "b3b30b4a-5ded-3ee7-88a9-f9d0859459ea",
                    "bos": {
                        "total_count": "2",
                        "size": "134217728 bytes",
                        "avg_size": "67108864 bytes",
                        "peak_size": "67108864 bytes",
                        "bytes_synced_to_device": "67108864 bytes",
                        "bytes_synced_from_device": "67108864 bytes"
                    },
                    "kernels": [
                        {
                            "name": "bandwidth",
                            "num_of_args": "4",
                            "num_total_runs": "1",
                            "avg_run_time": "172652 us"
                        }
                    ]
                }
            ]
        }
    },
    "thread 1": {
        "device": {
            "device_index": "0",
            "bdf": "0000:3b:00.1",
            "bos_peak_count": "3",
            "global_bos": {
                "total_count": "1",
                "size": "67108864 bytes",
                "avg_size": "67108864 bytes",
                "peak_size": "67108864 bytes",
                "bytes_synced_to_device": "0 bytes",
                "bytes_synced_from_device": "0 bytes"
            },
            "hw_context": [
                {
                    "id": "0",
                    "xclbin_uuid": "b3b30b4a-5ded-3ee7-88a9-f9d0859459ea",
                    "bos": {
                        "total_count": "2",
                        "size": "134217728 bytes",
                        "avg_size": "67108864 bytes",
                        "peak_size": "67108864 bytes",
                        "bytes_synced_to_device": "67108864 bytes",
                        "bytes_synced_from_device": "67108864 bytes"
                    },
                    "kernels": [
                        {
                            "name": "bandwidth",
                            "num_of_args": "4",
                            "num_total_runs": "1",
                            "avg_run_time": "84010 us"
                        }
                    ]
                }
            ]
        }
    },
    "thread 2": {
        "device": {
            "device_index": "0",
            "bdf": "0000:3b:00.1",
            "bos_peak_count": "3",
            "global_bos": {
                "total_count": "1",
                "size": "67108864 bytes",
                "avg_size": "67108864 bytes",
                "peak_size": "67108864 bytes",
                "bytes_synced_to_device": "0 bytes",
                "bytes_synced_from_device": "0 bytes"
            },
            "hw_context": [
                {
                    "id": "0",
                    "xclbin_uuid": "b3b30b4a-5ded-3ee7-88a9-f9d0859459ea",
                    "bos": {
                        "total_count": "2",
                        "size": "134217728 bytes",
                        "avg_size": "67108864 bytes",
                        "peak_size": "67108864 bytes",
                        "bytes_synced_to_device": "67108864 bytes",
                        "bytes_synced_from_device": "67108864 bytes"
                    },
                    "kernels": [
                        {
                            "name": "bandwidth",
                            "num_of_args": "4",
                            "num_total_runs": "1",
                            "avg_run_time": "342594 us"
                        }
                    ]
                }
            ]
        }
    },
    "thread 3": {
        "device": {
            "device_index": "0",
            "bdf": "0000:3b:00.1",
            "bos_peak_count": "3",
            "global_bos": {
                "total_count": "1",
                "size": "67108864 bytes",
                "avg_size": "67108864 bytes",
                "peak_size": "67108864 bytes",
                "bytes_synced_to_device": "0 bytes",
                "bytes_synced_from_device": "0 bytes"
            },
            "hw_context": [
                {
                    "id": "0",
                    "xclbin_uuid": "b3b30b4a-5ded-3ee7-88a9-f9d0859459ea",
                    "bos": {
                        "total_count": "2",
                        "size": "134217728 bytes",
                        "avg_size": "67108864 bytes",
                        "peak_size": "67108864 bytes",
                        "bytes_synced_to_device": "67108864 bytes",
                        "bytes_synced_from_device": "67108864 bytes"
                    },
                    "kernels": [
                        {
                            "name": "bandwidth",
                            "num_of_args": "4",
                            "num_total_runs": "1",
                            "avg_run_time": "261316 us"
                        }
                    ]
                }
            ]
        }
    },
    "thread 4": {
        "device": {
            "device_index": "0",
            "bdf": "0000:3b:00.1",
            "bos_peak_count": "3",
            "global_bos": {
                "total_count": "1",
                "size": "67108864 bytes",
                "avg_size": "67108864 bytes",
                "peak_size": "67108864 bytes",
                "bytes_synced_to_device": "0 bytes",
                "bytes_synced_from_device": "0 bytes"
            },
            "hw_context": [
                {
                    "id": "0",
                    "xclbin_uuid": "b3b30b4a-5ded-3ee7-88a9-f9d0859459ea",
                    "bos": {
                        "total_count": "2",
                        "size": "134217728 bytes",
                        "avg_size": "67108864 bytes",
                        "peak_size": "67108864 bytes",
                        "bytes_synced_to_device": "67108864 bytes",
                        "bytes_synced_from_device": "67108864 bytes"
                    },
                    "kernels": [
                        {
                            "name": "bandwidth",
                            "num_of_args": "4",
                            "num_total_runs": "1",
                            "avg_run_time": "84019 us"
                        }
                    ]
                }
            ]
        }
    }
}

Documentation impact (if any)

NA

gbuildx · 2023-11-07T17:59:40Z

Build Passed!

stsoe

This is very good Rahul. Nice clean code.
Lots of comments. Let me know what you think, not all may be possible. We can meet and discuss.
Thank you for making this a manageable pull request. The changes are about what I can comprehend in one sitting :-)

src/runtime_src/core/common/api/hw_context_int.h

src/runtime_src/core/common/api/kernel_int.h

src/runtime_src/core/common/shim/buffer_handle.h

src/runtime_src/core/common/api/xrt_hw_context.cpp

src/runtime_src/core/common/usage_metrics.h

src/runtime_src/core/common/api/xrt_kernel.cpp

rbramand-xilinx · 2023-11-08T06:21:09Z

This is very good Rahul. Nice clean code. Lots of comments. Let me know what you think, not all may be possible. We can meet and discuss. Thank you for making this a manageable pull request. The changes are about what I can comprehend in one sitting :-)

Thanks Soren :)

gbuildx · 2023-11-08T21:14:17Z

Build failed :(

stsoe

Some more changes. Please review.

src/runtime_src/core/common/usage_metrics.h

src/runtime_src/core/common/usage_metrics.cpp

src/runtime_src/core/common/api/xrt_hw_context.cpp

src/runtime_src/core/common/usage_metrics.h

src/runtime_src/core/common/usage_metrics.cpp

src/runtime_src/core/common/api/xrt_kernel.cpp

src/runtime_src/core/common/device.h

gbuildx · 2023-11-09T16:47:12Z

Build Passed!

stsoe

I am sorry to drag this on. But it is making good progress. Just need to get everything perfect :-)

src/runtime_src/core/common/usage_metrics.cpp

gbuildx · 2023-11-10T15:49:46Z

Build failed :(

src/runtime_src/core/common/usage_metrics.cpp

gbuildx · 2023-11-10T19:19:27Z

Build Passed!

src/runtime_src/core/common/usage_metrics.cpp

Signed-off-by: rbramand <[email protected]>

> Refactor code Signed-off-by: rbramand <[email protected]>

Signed-off-by: rbramand <[email protected]>

stsoe

Thank you for working through all the changes. Looks good now. Reads well IMO.

gbuildx · 2023-11-11T00:09:41Z

Build failed :(

stsoe · 2023-11-11T06:29:51Z

retest this please

gbuildx · 2023-11-11T06:59:46Z

Build failed :(

chvamshi-xilinx · 2023-11-11T15:27:10Z

retest this please

gbuildx · 2023-11-11T17:27:52Z

Build failed :(

manikandan-xilinx · 2023-11-11T17:29:32Z

retest this please. (PACT board test is failing repeatedly. So witching back from it )

gbuildx · 2023-11-11T20:34:28Z

Build Passed!

stsoe

Hi @rbramand-xilinx
Please check https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463167&mergedDefectId=330809
Comments added

stsoe · 2023-11-13T00:39:59Z

src/runtime_src/core/common/usage_metrics.cpp

+}
+
+usage_metrics_logger::
+~usage_metrics_logger()


We need try / catch to resolve

https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463167&mergedDefectId=330809

https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463171&mergedDefectId=330815

https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463146&mergedDefectId=330816

stsoe · 2023-11-13T00:45:49Z

src/runtime_src/core/common/usage_metrics.cpp

+    m_dev_map[dev_id] = {};
+    try {
+      auto bdf = xrt_core::query::pcie_bdf::to_string(xrt_core::device_query<xrt_core::query::pcie_bdf>(dev));
+      m_dev_map[dev_id].bdf = bdf;


std::move(bdf)
https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463145&mergedDefectId=330811

stsoe · 2023-11-13T00:47:56Z

src/runtime_src/core/common/usage_metrics.cpp

+namespace {
+// global variables
+static std::mutex m;
+static uint32_t thread_count;


static std::atomic<uint32_t> thread_count {0};
https://scan9.scan.coverity.com/reports.htm#v36614/p13446/fileInstanceId=123592600&defectInstanceId=9463130&mergedDefectId=330812

stsoe · 2023-11-13T00:49:21Z

src/runtime_src/core/common/usage_metrics.cpp

+~usage_metrics_logger()
+{
+  {
+    std::lock_guard<std::mutex> lk(m);


Move before 337 now that thread_count is atomic

rbramand-xilinx requested review from stsoe, vboggara-xilinx and rozumx as code owners November 7, 2023 15:16

rbramand-xilinx requested a review from chvamshi-xilinx November 7, 2023 15:18

rbramand-xilinx force-pushed the usage_metrics branch from 579af83 to deda6d7 Compare November 7, 2023 15:25

stsoe requested changes Nov 7, 2023

View reviewed changes

rbramand-xilinx force-pushed the usage_metrics branch from deda6d7 to 871693a Compare November 8, 2023 19:56

rbramand-xilinx requested a review from stsoe November 8, 2023 19:59

rbramand-xilinx force-pushed the usage_metrics branch from 871693a to e6d318a Compare November 8, 2023 20:43

stsoe requested changes Nov 8, 2023

View reviewed changes

rbramand-xilinx force-pushed the usage_metrics branch from e6d318a to 39f89af Compare November 9, 2023 13:14

rbramand-xilinx requested a review from stsoe November 9, 2023 13:17

rbramand-xilinx force-pushed the usage_metrics branch from 39f89af to fd4898d Compare November 9, 2023 13:36

stsoe requested changes Nov 9, 2023

View reviewed changes

rbramand-xilinx force-pushed the usage_metrics branch from fd4898d to 9a87575 Compare November 10, 2023 12:22

stsoe force-pushed the usage_metrics branch from 9a87575 to 1884d29 Compare November 10, 2023 16:01

stsoe requested changes Nov 10, 2023

View reviewed changes

stsoe reviewed Nov 10, 2023

View reviewed changes

src/runtime_src/core/common/usage_metrics.cpp Outdated Show resolved Hide resolved

stsoe reviewed Nov 10, 2023

View reviewed changes

src/runtime_src/core/common/usage_metrics.cpp Outdated Show resolved Hide resolved

stsoe reviewed Nov 10, 2023

View reviewed changes

src/runtime_src/core/common/usage_metrics.cpp Outdated Show resolved Hide resolved

stsoe reviewed Nov 10, 2023

View reviewed changes

src/runtime_src/core/common/usage_metrics.cpp Outdated Show resolved Hide resolved

rbramand-xilinx force-pushed the usage_metrics branch 2 times, most recently from 46ab03d to eb8dcbe Compare November 10, 2023 21:15

stsoe reviewed Nov 10, 2023

View reviewed changes

src/runtime_src/core/common/usage_metrics.cpp Outdated Show resolved Hide resolved

rbramand added 6 commits November 11, 2023 03:17

Initial changes for usage metrics logger

7f964fb

Signed-off-by: rbramand <[email protected]>

Refactor to reduce overhead

cccf80f

Signed-off-by: rbramand <[email protected]>

Add initial json parser

a949094

Signed-off-by: rbramand <[email protected]>

Add changes to record kernel start and end timestamps

20497a5

Signed-off-by: rbramand <[email protected]>

Address comments on PR

003cc4d

> Refactor code Signed-off-by: rbramand <[email protected]>

Refactor code

f3888ff

Signed-off-by: rbramand <[email protected]>

rbramand-xilinx force-pushed the usage_metrics branch from eb8dcbe to f3888ff Compare November 10, 2023 21:59

stsoe approved these changes Nov 10, 2023

View reviewed changes

chvamshi-xilinx approved these changes Nov 11, 2023

View reviewed changes

stsoe merged commit b041f00 into Xilinx:master Nov 11, 2023
3 checks passed

stsoe reviewed Nov 13, 2023

View reviewed changes

This was referenced Nov 13, 2023

Fix coverity scan issues #7797

Merged

Fix divide by zero exception when collecting usage metrics #7805

Merged

rbramand-xilinx mentioned this pull request Dec 5, 2023

Create file with unique name to dump usage metrics #7831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VITIS-10034 Add Usage metrics for all XRT objects #7788

VITIS-10034 Add Usage metrics for all XRT objects #7788

rbramand-xilinx commented Nov 7, 2023 •

edited

Loading

gbuildx commented Nov 7, 2023

stsoe left a comment •

edited

Loading

rbramand-xilinx commented Nov 8, 2023

gbuildx commented Nov 8, 2023

stsoe left a comment

gbuildx commented Nov 9, 2023

stsoe left a comment

gbuildx commented Nov 10, 2023

gbuildx commented Nov 10, 2023

stsoe left a comment

gbuildx commented Nov 11, 2023

stsoe commented Nov 11, 2023

gbuildx commented Nov 11, 2023

chvamshi-xilinx commented Nov 11, 2023

gbuildx commented Nov 11, 2023

manikandan-xilinx commented Nov 11, 2023

gbuildx commented Nov 11, 2023

stsoe left a comment

stsoe Nov 13, 2023

stsoe Nov 13, 2023

stsoe Nov 13, 2023

stsoe Nov 13, 2023

VITIS-10034 Add Usage metrics for all XRT objects #7788

VITIS-10034 Add Usage metrics for all XRT objects #7788

Conversation

rbramand-xilinx commented Nov 7, 2023 • edited Loading

Problem solved by the commit

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

How problem was solved, alternative solutions (if any) and why they were rejected

Risks (if any) associated the changes in the commit

What has been tested and how, request additional testing if necessary

Documentation impact (if any)

gbuildx commented Nov 7, 2023

stsoe left a comment • edited Loading

Choose a reason for hiding this comment

rbramand-xilinx commented Nov 8, 2023

gbuildx commented Nov 8, 2023

stsoe left a comment

Choose a reason for hiding this comment

gbuildx commented Nov 9, 2023

stsoe left a comment

Choose a reason for hiding this comment

gbuildx commented Nov 10, 2023

gbuildx commented Nov 10, 2023

stsoe left a comment

Choose a reason for hiding this comment

gbuildx commented Nov 11, 2023

stsoe commented Nov 11, 2023

gbuildx commented Nov 11, 2023

chvamshi-xilinx commented Nov 11, 2023

gbuildx commented Nov 11, 2023

manikandan-xilinx commented Nov 11, 2023

gbuildx commented Nov 11, 2023

stsoe left a comment

Choose a reason for hiding this comment

stsoe Nov 13, 2023

Choose a reason for hiding this comment

stsoe Nov 13, 2023

Choose a reason for hiding this comment

stsoe Nov 13, 2023

Choose a reason for hiding this comment

stsoe Nov 13, 2023

Choose a reason for hiding this comment

rbramand-xilinx commented Nov 7, 2023 •

edited

Loading

stsoe left a comment •

edited

Loading