Host-Device Unnecessary Copying

[WillianJunior]

Hi guys,

I'm currently realizing a pipeline P1 on a static buffer B1 on the device over and over again until a certain condition is met. The condition is checked though another pipeline P2 which has B1 as an input and B2 as a smaller output buffer. When running P1, the only data movement of B1 is the initial host->device copy/allocation before the first realize, and a final device->host copy after a copy_to_host() call, as is expected.

For P2, I require a copy_to_host() of B2 after each of its realization for checking the end condition. As expected, after realizing P2, the device_dirty flag of B2 is set, allowing the copy_to_host() call. However, before the execution of P2 a host->device copy is always performed when it shouldn't. Interestingly, if I set B2.set_device_dirty() before the copy call, a device->host copy is performed before the host->device copy operation! Since the data on B2 is only dependent on B1, ideally it should only perform a single allocation/copy on device the first time (which I'm even doing outside the main loop), and a device->host copy at every copy_to_host() calls. I also tried setting B2.set_device_dirty(false) to no difference.

It is worth noting that both P1 and P2 use update definitions. For P1 the first definition is a Halide::undef() for in-place realization. For P2 the first definition is an initialization (i.e., P2(...) = 0) and the second actually computes the results based on B1. Just in case, bellow I've written a pseudo-code to better visualize what my algorithm does:

long B2Host[Size];
B2 = Halide::Buffer(B2Host); // Wrapping a host memory variable with a Halide::Buffer
B2.set_host_dirty();
B2.copy_to_device(target); // Required to pass the Halide::Target with the configured device
do {
    P1.realize(B1)

    // B2.set_device_dirty() // This results in the device->host->device copy
    P2.realize(B2);
    B2.set_device_dirty(false); // This makes no difference...
    B2.copy_to_host();

    condition = shouldEnd(B2Host); // CPU code
} while (condition is false)

B1.copy_to_host(); // Final result

I imagine that Halide might see the update definition and assume that it should copy the host data, even with the host_dirty flag not set. I'm curious since this does not happen for P1 (maybe something related to Halide:undef() and in-place realization?).

What can I do to impede Halide from performing the host->device copy of B2 before every realization?

Thanks for the help.

[abadams]

I don't understand why you have B2.copy_to_device just before inspecting B2 on CPU. Is that supposed to be B2.copy_to_host?

I'm not seeing any unusual copies when I attempt to reproduce this. Can you post a full repro? Here's what I have:

#include "Halide.h"

using namespace Halide;

int main(int argc, char **argv) {
    Target target("host-cuda-debug");

    Buffer<int> B1(128, 128);

    Func f, g;
    Var x, y;

    f(x, y) = undef<int>();
    f(x, y) = x + y;

    g(x, y) = 0;
    g(x, y) += B1(x, y);

    Var xi, yi;
    f.compute_root()
        .update()
        .gpu_tile(x, y, xi, yi, 16, 16);
    g.compute_root()
        .gpu_tile(x, y, xi, yi, 16, 16)
        .update()
        .gpu_tile(x, y, xi, yi, 16, 16);

    Pipeline P1(f), P2(g);

    int buf[128 * 128];
    Buffer<int> B2(buf, 128, 128);
    B2.set_host_dirty();
    B2.copy_to_device(target);  // Required to pass the Halide::Target with the configured device
    for (int i = 0; i < 10; i++) {
        std::cerr << "\n\n\n\n";
        P1.realize(B1, target);

        // B2.set_device_dirty() // This results in the device->host->device copy
        P2.realize(B2, target);
        //B2.set_device_dirty(false);  // This makes no difference...
        //B2.copy_to_device(target);
        B2.copy_to_host();

        // Some CPU code here
        std::cout << buf[17] << "\n";
    };

    B1.copy_to_host();  // Final result

    return 0;
}

[WillianJunior]

First, thanks for the help.

B2.copy_to_device() was indeed wrong, and should be copy_to_host(). This was updated on the original question.

I found the main problem: I was scheduling the update definition on CUDA, but not the pure definition, and so g(x, y) = 0; was executed on the host, to later have its values sent to the device.

[WillianJunior]

On another issue, I found that the execution of functions with update definitions were performed separately. With debugging ON two executions happened for P2 (note, I named Func g("g")):

CUDA: halide_cuda_run (user_context: 0x7ffd11f431d8, entry: kernel_g_s0_x_v0___block_id_x, blocks: 13x1x1, threads: 10x1x1, shmem: 0

CUDA: halide_cuda_run (user_context: 0x7ffd11f431d8, entry: kernel_g_s1_x_v0___block_id_x, blocks: 13x1x1, threads: 10x1x1, shmem: 0

Highlighted are what I assume are the names of the Func's, "g" and the definition or stage "s". Is this the default behavior of Halide, to perform a single halide_cuda_run() (for CUDA for instance) for each update definition? If so, is it possible to merge these into a single halide_cuda_run() call for all update definitions of a given pipeline?

[abadams]

That's a function of the schedule. The simplest way to run them together would be to use Func::in to make a wrapper Func, and replace:

    g.compute_root()
        .gpu_tile(x, y, xi, yi, 16, 16)
        .update()
        .gpu_tile(x, y, xi, yi, 16, 16);

    Pipeline P2(g);

with

    g.in()
        .compute_root()
        .gpu_tile(x, y, xi, yi, 16, 16);

    Pipeline P2(g.in());

[WillianJunior]

Thank you. This worked just fine, with a few caveats:

1. all definitions (pure and update) must iterate on the same Var (i.e., cannot use Var on pure def and RDom on update).
2. If I use RDom on the right side of the definition (e.g., g(x) += f(x, ry), being ry a RDom) I cannot reorder the variables (e.g., g.in().compute_root().reorder(ry, x)), returning a "could not find var ry$x to reorder in the argument list" error message.

Is it possible to accessry after wrapping g?

Also, from my understanding, when I use g.in(void) it wraps every update definition. This is useful if all update definitions have the same dimensionality. As such, if I decide to perform another reduction (e.g., from a 1D array to a scalar) I need to have another Func h which uses g as its input. It is clear that I can schedule g.in(h) to have them all running on GPU without data movement between stages. However, now I have 2 halide_cuda_run()calls, one for each top-level wrapper of each g and h. Is there another trick similar to wrapping all update definitions with in(), but to a global inter-stage level?

Thanks again for all the help.