CUPTI Perfworks support

[tkonolige]

Hi All,

It is great to have a project that provides a toolkit for profiling. I'd like to integrate Timemory into TVM instead of having to reimplement all these profiling interfaces. However, it seems like Timemory does not current support CUDA 3000 series GPUs. I assume this is because of the new profiling APIs (CUPTI perfworks). Is anyone working on support? If not, I'd be happy to implement a component for it.

[jrmadsen]

However, it seems like Timemory does not current support CUDA 3000 series GPUs.

What issues have you had? I assume this is w.r.t. HW counters as the cupti_activity component should still work.

> Is anyone working on support?

Short answer: No. In part because when the APIs came out, it created a design problem. The other part is that, while I've developed a solution to this design problem, now I work at AMD these days so I can't really dedicate time to writing support for NVIDIA stuff.

Here's the background on the design problem: while I personally shudder when I look at the implementation of the cupti_activity and cupti_counters nowadays (they were some of the first truly complex components I wrote for timemory), a lot of the complexity arose from the premise that the components which stored data had a T get() const function which returned valid data. Thus there were essentially, two types of components:

1. Components which stored data, e.g. wall_clock, cupti_activity, etc., enabling:
   A. Retrieving the current value of a specific component instance at any point in the application via get() member function
   B. Retrieving the entire call-stack tree via storage<T>::instance()->get() (usually at the end of the application) and accessing all that data in those component instances
2. Components which just forwarded to other APIs and never made any data available

The problem was that because the new profiling APIs didn't make intermediate data available (i.e. you could only get the data when you finalized the entire API), a cupti_perfworks component (or similar) would, by necessity, have to always return zeros for 1.A and 1.B would also be completely populated with zeros until the data was available and that data was placed back into the storage call-stack (which, itself, wasn't straightforward).

> If not, I'd be happy to implement a component for it.

That would be great. Here's what I would recommend:

• Avoid trying to directly integrate within timemory (i.e. avoid putting your files in the source/timemory/components/cupti folder)
  • Small changes will take too long to re-compile and there really isn't much difference between a component defined within the timemory headers and within an external project (when using the template API)
  • Take an example from the cupti_samples (usually in /usr/local/cuda/extra/CUPTI/samples)
    • e.g. userrange_profiling, concurrent_profiling
• Create a standalone cmake project where timemory is an external project
  • i.e. find_package(timemory COMPONENTS cxx cuda cupti)
  • When you have something working, drop it into a new sub-directory in the examples folder and open a PR
• Move anything specific to CUPTI in the chosen example file into a cupti_perfworks.cpp file
• Create two components
  1. cupti_perfworks which stores no data and implements all the CUPTI API calls -- you use this component
     • inherits from component::base<cupti_perfworks, void>
     • Prioritize the finalization of the cupti_perfworks component
       • TIMEMORY_DEFINE_CONCRETE_TRAIT(fini_priority, component::cupti_perfworks, priority_constant<-4>)
  2. cupti_perfworks_data which can be hidden in a .cpp file (i.e. not publicly available) and holds the data
     • inherits from component::base<cupti_perfworks_data, T> where T whatever data-type is best suited for the results
• Write the prototype of the cupti_perfworks component in a cupti_perfworks.hpp file
• Implement the component member functions in cupti_perfworks.cpp and just call the functions they have already written in the proper places.
• Create a cupti_perfworks::global_finalize() static method, grab the data and iterate over it, adding the values to a cupti_perfworks_data component
  • Thus, until cupti_perfworks is finalized, there are no cupti_perfworks_data instances in storage<cupti_perfworks_data> that would misleadingly report zeros
  • When this component finalizes, it will produce the output data

Here's the gist of what you need in cupti_perfworks.hpp:

TIMEMORY_DECLARE_COMPONENT(cupti_perfworks)

TIMEMORY_DEFINE_CONCRETE_TRAIT(fini_priority, component::cupti_perfworks, priority_constant<-4>)

namespace tim
{
namespace component
{
struct cupti_perfworks : base<cupti_perfworks, void>
{
    static std::string label() { return "cupti_perfworks"; }

    static void global_init(); // place setup here (automatically called on first use of component)
    static void global_finalize(); // place teardown here

    void start();
    void stop(); 

    void set_prefix(const char* _v) { m_prefix = _v; }

private:
    const char* m_prefix = nullptr;
};
}
}

Here's the gist of what you need in cupti_perfworks.cpp:

TIMEMORY_DECLARE_COMPONENT(cupti_perfworks_data)

namespace tim
{
namespace component
{
struct cupti_perfworks_data : base<cupti_perfworks, std::vector<double>>
{
    // keep this the same
    static std::string label() { return "cupti_perfworks"; }

    void store(value_type _v) { set_accum(std::move(_v)); }
};
}
}

TIMEMORY_INITIALIZE_STORAGE(cupti_perfworks_data)

namespace
{
//
// ... CUPTI-specific sample code ...
//
}

//
//
//
void
tim::component::cupti_perfworks::start() { ... }

void
tim::component::cupti_perfworks::stop() { ... }

void
tim::component::cupti_perfworks::global_init() { ... }

void
tim::component::cupti_perfworks::global_finalize()
{
    static bool _once = false;
    if(_once) return;
    _once = true;

    using bundle_t = lightweight_tuple<cupti_perfworks_data>;
    // modify
    for(auto itr : ...)
    {
        bundle_t _v{ ... extract label from itr... };
        _v.push(); // create call-stack entry
        _v.start(); // needed for stop call
        _v.store( ... data from itr... );
        _v.stop(); // increments lap counter
        _v.pop(); // update call-stack entry
    }
}

And the gist of the main:

#include "cupti_perfworks.hpp"
#include <timemory/timemory.hpp>

namespace comp = tim::component;

using bundle_t = tim::component_tuple<comp::cupti_perfworks>;

int main(int argc, char** argv)
{
    tim::timemory_init(argc, argv);

    bundle_t _v{ "main" };
    _v.start();
    // ... etc.
    _v.stop();

    tim::timemory_finalize();
}

[tkonolige]

Are you certain that this causes incorrect results?

Sorry, what I meant was that this breaks any nested timing where the timers are not just CUPTI. In this example

tim::component_tuple<comp::wall_clock> bundle_outer("outer");
tim::component_tuple<comp::cupti_perfworks> bundle_inner("inner");

bundle_outer.start();
bundle_inner.start();
// Run kernels
bundle_inner.stop();
bundle_outer.stop();

bundle_outer will show the time it takes to do all the kernel replays instead of the time it takes to do a single run of the kernel.

I think that user-replay is most convenient for the use case I am looking at. I think using bundle::execute could work, but I would also be interested in an approach where the user must re-run the code themselves (like the api CUPTI provides).

[jrmadsen]

For the record, if you weren't aware, there are benefits to doing:

tim::component_tuple<comp::wall_clock, comp::cupti_perfworks> bundle("outer");

instead of what you have there because the bundles can handle prioritization b/t wall_clock and cupti_perfworks, e.g. if cupti_perfworks were to call cudaDeviceSychronize() during stop -- however with respect to:

this breaks any nested timing where the timers are not just CUPTI... will show the time it takes to do all the kernel replays instead of the time it takes to do a single run of the kernel.

That's generally fine, timing on the CPU is almost always at least slightly flawed when GPU kernels are profiled and, at the very least, it provides information on the overhead of the profiling. And, in general, if you really want the time to run a kernel, you should use the cupti_activity component instead of a wall-clock timer. One of these days, I should hopefully get around to appending the cupti_activity measurements to the wall-clock measurements -- which is something I've done with the roctracer component (AMD's ROCm equivalent to CUPTI activity) in another tool we will be releasing soon.

[tkonolige]

I guess I was concerned about nesting timers because I read that as one of the main goals of Timemory:

Support arbitrarily intermixing components:

• Instrument measurements of A, B, and C around arbitrary region 1
• Instrument measurements of A and C around arbitrary region 1.1 (nested with Section 1)
• Instrument measurements of C around arbitrary region 2
• Instrument measurements of D around arbitrary region 3
• No instrumentation around arbitrary region 4

(from https://timemory.readthedocs.io/en/develop/about.html)

Another disadvantage of using kernel replay is that CUPTI only supports it with autorange. That means we'll have to find some way of figuring out which CUPTI range corresponds the range defined by Timemory's start/stop. I don't have any ideas on how to do that. It might even be impossible.

[jrmadsen]

To me, this does not break that goal.

Another disadvantage of using kernel replay is that CUPTI only supports it with autorange. That means we'll have to find some way of figuring out which CUPTI range corresponds the range defined by Timemory's start/stop. I don't have any ideas on how to do that. It might even be impossible.

It's not impossible, just complicated, which, from experience, is the norm when dealing with CUPTI. It is one of my least favorite APIs ever.

IIRC, autorange just puts ranges around the kernels and the CUPTI Activity API will create a unique correlation ID for each kernel launch and there will be a callback when the buffer is full of the kernel start and end time. So to correlate the data, you need to record a start and stop timestamps or record which correlation IDs were emitted in between the start/stop in the cupti_perfworks component, store the timestamps and/or correlation IDs of the kernels provided by the CUPTI Activity API + any other info you need (such as the kernel name), and then in the post-processing stage, find the correlation-ID/timestamps matches.

[jrmadsen]

As soon as I get approval for open-sourcing the new tool I mentioned previously, I can point you to an example of how to do this correlation as the roctracer implementation is essentially the equivalent of the CUPTI Activity API.