Stringing together a bunch of GPUs for very high VRAM at low cost?

[gituser123456789000]

How would we go about stringing together a bunch of GPUs to share VRAM and be able to use very high settings in iw3, even if it is slower?

For example, could we string together 4 x 12GB 3060s for 48GB VRAM at a cost of around $1000 instead of $3000-4000+ for a single 48GB AI card?

Or 8 of them for 96GB VRAM for around $2000, instead of an AI card that costs tens of thousands of dollars, if it could even be found for sale anywhere..

As far as wattage.. 3060s are rated at only 170W max.. so 4 is 'only' 680W max, which isn't crazy.. and I'm sure they could be set to something like 60% Power with MSI Afterburner, which would suggest only 408W max.

I read that SLI is gone, not supported by newer cards, or DirectX 12 cards.. but I also read that some people do have multiple card setups, like a 4090 and two 3060s somehow..

I also read this:
"Sli fell out of favor mostly, with 30 series they did cancel it. Except for the 3090, it's the only one still capable of sli. But then you're talking heavy power draw and while gpu compute performance was doubled in parallel, vram was not. So even if you could sli 3060's, say they were the 12gb vram variety. You'd have 2x 3060's in compute performance and still 12gb vram, not 24gb. Vram didn't stack."

So according to that, 4 3090s for example could have the compute performance of 4 3090s.. but still only utilize 24GB of VRAM and not 96GB.

It would be interesting to hear from any knowledgeable person who comes through here, if there is a way.. because a 4x or 8x 3060 setup at $1,000 or $2,000 is not so much, compared to tens of thousands of dollars for single cards with this much VRAM.

I suppose the other possibility to increase resolution at the cost of performance, would be Tiling. Still interested to see what results Tiling could give.

I did testing earlier and confirmed that using setting resolution higher than your original content does in fact create better depth maps at better results.

For example, if you have a small 240x240 image... you can test the Default maps, 512 maps, and higher, like 1024 maps.. The 1024 maps are significantly more details, accurate, true to life.. and converted results show it. The output image will still be the original 240x240, but 3D effect is much better with the high resolution maps. That's why I'm thinking about stacking VRAM, so we could push map resolution as high as possible, even if speed isn't great, but you'd eventually get a great result instead of an out of VRAM crash

I also wonder if having more regular RAM makes you more capable in using higher resolution depth maps, since more regular RAM means more Shared Memory

[nagadomi]

Not directly related to the title, but I made improvements to handle the high resolution depth now.
On Linux, Depth Resolution=4000 with Any_V2_S and Depth Resolution=3000 with Any_V2_B can be processed with 6GB of VRAM.
However, depth becomes broken around over 2000.

Not tested on Windows GPU environment.

[gituser123456789000]

Not directly related to the title, but I made improvements to handle the high resolution depth now. On Linux, Depth Resolution=4000 with Any_V2_S and Depth Resolution=3000 with Any_V2_B can be processed with 6GB of VRAM. However, depth becomes broken around over 2000.

Not tested on Windows GPU environment.

It's funny, because I actually did the exact same thing an hour ago (1hour after your post, but I didn't see until after.)

I was running tests and saw that I could max out AnyS v2 and AnyB v2 with the 2048 limit, so I edited gui.py to allow for 4096 limit. AnyB v2 ended up still being maxed at 2048 in my limited testing with that, but I could push AnyS v2 somewhere between 2310-2560 res (S wasn't looking as good as L and B, so I haven't found the specific limit with S yet)

Those are crazy results on Linux.. 3000 res with Any_V2_B with only 6GB VRAM? wow

What resolution are you maxing out with for Any_V2_L? I can push about 3,700,000 pixels. The resolution depends on the aspect ratio, but I always run out of CUDA memory once I pass the 3.7 Million pixel range with a 24GB 4090 using Any_V2_L on Windows 10

You're doing excellent work, and thanks for the very fast and easy implementation of v2

[nagadomi]

When I made the above post, I made two changes.

1. Use FlashAttention without xformers in DINOv2, the backbone of Depth-Anything V1/V2 (this will greatly reduce VRAM usage on iw3, because nunif-windows-package does not install xformers)
2. Fixed a bug in row_flow_v3 that caused a CUDA error at high resolution

If you tried before this change, please update and try again.
Note that on Linux, python -m iw3.download_models is required for updates.

Also, I have now unlocked the 2048 limit for Max Depth Resolution in the GUI.

[gituser123456789000]

When I made the above post, I made two changes.

1. Use FlashAttention without xformers in DINOv2, the backbone of Depth-Anything V1/V2 (this will greatly reduce VRAM usage on iw3, because nunif-windows-package does not install xformers)

2. Fixed a bug in row_flow_v3 that caused a CUDA error at high resolution

If you tried before this change, please update and try again. Note that on Linux, python -m iw3.download_models is required for updates.

Also, I have now unlocked the 2048 limit for Max Depth Resolution in the GUI.

ABSOLUTE INSANITY!!!!! iw3 is extremely more efficient now! Wow.

Now just waiting for the Giant model. Crazy improvements.

A clip I was testing with 1428 depth resolution (which was the absolute max I could run at a snail's pace.. it took up all of my VRAM and all of my Shared Memory.. is now running at only 9GB VRAM. You absolutely destroyed expectations with this update. Amazing

[gituser123456789000]

When I made the above post, I made two changes.

1. Use FlashAttention without xformers in DINOv2, the backbone of Depth-Anything V1/V2 (this will greatly reduce VRAM usage on iw3, because nunif-windows-package does not install xformers)

2. Fixed a bug in row_flow_v3 that caused a CUDA error at high resolution

If you tried before this change, please update and try again. Note that on Linux, python -m iw3.download_models is required for updates.

Also, I have now unlocked the 2048 limit for Max Depth Resolution in the GUI.

This efficiency improvement was huge. I was first thinking in terms of pushing ultra high resolution depth maps, but now that we've figured out only 518 resolution is ideal.. the real benefits are being about to run a quality conversion without your system being bogged down while you wait...

..and being able to run multiple instances at the same time, while easily fitting them into VRAM (depending what GPU the user has)

I ran a test with low settings using Any S, Default maps, etc.. and I had 4 iw3 instances run at the same time with no issue, and they were only taking up 0.6 - 0.9GB of VRAM each (Any B and L models will use more VRAM), ~28% CPU usage.. and GPU usage seemed unaffected whether it was 1 instance running or 4. I think it was around 70%.

At some point, I wanted to see how far I could push it . I planned to try to run like 20-30 video conversions at once. 4 was easy, so it will be interesting to find the limit and see where the bottleneck is. I'm not sure if it will be GPU utilization.. or CPU.. or maybe hard drive space/speed in some way.

Essentially a big speed boost when you can run multiple projects at the same time.. at the same or similar speed as running a single project.