Developing New Algorithms

[ecarl65]

I'm a total newb to MatX and CUDA in general. I'm a DSP engineer and program in C++ and Python, but just have a hole in my experience regarding GPUs. Anyway, I thought MatX might be a good way to dip my toes in the water.

I'm just wondering a little bit about the model. It's awesome that it has a lot of tools like filtering, FFTs, etc. Let's say I wanted to make a new algorithm, like a Hough transform, for example. I guess there are a couple of ways of going about it?

1. Program entirely using MatX, in developer-friendly manner that doesn't involve low level CUDA calls. Hope that the speed is decent. This is what MatX was designed for, presumably?
2. If you're experienced with CUDA read the CONTRIBUTING.md file and try to contribute a hough transform algorithm with appropriate low-level CUDA calls for addition into the baseline.

Do I have the general idea right?

[cliffburdick]

Hi @ecarl65, thanks for stopping by!

1. Yes, MatX is designed to be the same speed or faster than a novice to mid-level user writing their own CUDA kernels. We do this two ways: compile-time kernel fusion for element-wise kernels, and using popular CUDA libraries for optimized backends (cuBLAS, CUB, etc).

The first is important because using C++17 we're able to convert large algebraic expressions into a single kernel that should be as good or better than what you'd write yourself. It's also a lot more modular because writing a single kernel limits you to data types, sizes, strides, etc, whereas writing it generically in MatX allows you to change those parameters without changing the expression itself.

The second is important because in almost all cases you will not beat the CCCL or CUDA Math libraries, so we can utilize the same types you had in your algebraic expressions but translate it into the CUDA libraries. A good example of this is something like doing a sum(). For an optimal sum() function you need to use CUB. But CUB has its own API separate from other CUDA libraries that allows you to do a lot at compile-time. You can do that all yourself, or you can use something like MatX to do sum(out, in * 4 + sin(b)); and we convert that into an iterator used by CUB that computes the input section on-the-fly. You can do this yourself, but it ends up being a lot of code, and again, it's not modular.

This GTC talk goes into more depth: https://www.nvidia.com/en-us/on-demand/session/gtcfall21-a31410/

2. Yes, we're always open to new contributions. I can look into a Hough transform and see roughly what it would take and hopefully give some guidance.

[ecarl65]

Thanks so much! I look forward to reading this presentation. Also, this project is fantastic! This is exactly the sort of thing we need for easier and better adoption of CUDA technologies to people like me. How do we get this socialized to more people? I only happened to stumble on this from a link from a co-worker, but it seems like abstracting away a lot of the harder programming yet retaining similar execution times is a win-win and what people like me (engineers who aren't as savvy on the software development front) really need!

[cliffburdick]

Thanks so much! I look forward to reading this presentation. Also, this project is fantastic! This is exactly the sort of thing we need for easier and better adoption of CUDA technologies to people like me. How do we get this socialized to more people? I only happened to stumble on this from a link from a co-worker, but it seems like abstracting away a lot of the harder programming yet retaining similar execution times is a win-win and what people like me (engineers who aren't as savvy on the software development front) really need!

Thanks @ecarl65, much appreciated. Right now we rely on word-of-mouth beyond some internal uses we have, so you can help spread the word! We also have a sister project called cuSignal (https://github.com/rapidsai/cusignal) that's focused on GPU signal processing in Python. Many people find it helpful to prototype in Python before moving to C++, and we're happy to help with both.

I forgot to mention that if you think the performance of anything in MatX is not what you expect, please let us know. We expect to be native performance, and if not, then we'd like to know where we're failing.

[ecarl65]

Sounds great, I'll do that. I've already talked to my DSP friends inside and outside my company about it. But I'll start playing around with it soon as well. So far it looks fantastic, and I'm excited to use it. Of course, getting more and more features and approaching numpy functionality will only make it more useful. But that's a long term endeavor. Merely having filtering and FFTs gets me most of what I need, as well as matrix operations and a data type that just handles the CUDA parts.

[leofang]

We also have a sister project called cuSignal (https://github.com/rapidsai/cusignal) that's focused on GPU signal processing in Python. Many people find it helpful to prototype in Python before moving to C++, and we're happy to help with both.

FYI going forward cuSignal will be integrated into CuPy: https://docs.rapids.ai/notices/rsn0032/

[jwmelto]

I'll join in this discussion. I saw the linked presentation and thought it might make CUDA more accessible to my average developers. I started my CUDA journey trying to use Thrust, but it quickly became unwieldy.

I'm struggling for really useful documentation. Through trial and error, I'm making some progress, but I've spent too many hours trying to figure out what I think should be a basic operation.

I have some C-code that generates a signal replica. I call it in a loop to fill the rows of a 2D tensor. Now I want to extend this replica matrix multiple times (so I have N copies of each replica).

• is there a better way to fill each row than a loop that sets each element of the row from the "source" vector?
• how can I concatenate an entire matrix?

auto l1 = matx::linspace<0>({4}, 0, 5);
auto l2 = matx::linspace<0>({4}, 10, 40);

auto t1 = matx::make_tensor<int>( {2, 4} );
// t1[0] = l1;
// t1[1] = l2;

auto t2 = matx::make_tensor<int>( {2, 4*3} ); // I want 3 copies
// t2 = ???

// matx::print(t2) should produce
// 0 1 2 3 0 1 2 3 0 1 2 3
// 10 20 30 40 10 20 30 40 10 20 30 40

What am I missing?

[cliffburdick]

Hi @jwmelto, thanks for posting the question, and please don't hesitate to ask in the future if something is confusing.

For this problem:

auto t1 = matx::make_tensor<int>( {2, 4} );
// t1[0] = l1;
// t1[1] = l2;

You can do that in a single statement by using stack where you can stack l1 and l2 and assign directly to t1:

(t1 = stack(0, l1, l2)).run();

An example is here:

MatX/test/00_operators/OperatorTests.cu

Lines 3523 to 3531 in cda2122

	// example-begin stack-test-1
	// Stack 1D operators "t1a", "t1b", and "t1c" together along the first dimension
	auto op = stack(0, t1a, t1b, t1c);
	// example-end stack-test-1
	for(int i = 0; i < t1a.Size(0); i++) {
	ASSERT_EQ(op(0,i), t1a(i));
	ASSERT_EQ(op(1,i), t1b(i));
	ASSERT_EQ(op(2,i), t1c(i));

For your second question you would use repmat:

(t2 = repmat(t1, {1, 3})).run(); // Repeat `t1` 1 time in the first dimension and 3 times in the second

Let us know if you have any other questions.

[jwmelto]

Thank you! I knew the concepts had to exist, but just couldn't extract them from the documentation.

Are there plans for, or technical limitations precluding range-based assignment? e.g.,

std::vector<int> foo = {1,2,3,4,5};

auto t = matx::make_tensor<int>( { 5 } );
(t = foo).run();

(where any container-like thing (array, span, view) could be assigned)

What about filling containers in the opposite direction?