consumer hardware

[ehartford]

I want to do this but not on AWS, rather on consumer hardware.
All my friends have GPUs and we want to make a pool of our resource to train large models.
Is it possible?

[BinhangYuan]

Yes, it is possible, although we did not extensively test the performance under this setting.

First, you might need to configure the network; the goal is to make sure every node that will join the training can communicate with each other through a shared network interface, which is demanded by NCCL initialization. One way to achieve this is to set up a Swan VPN for UPD hole punching, you may refer to some of our scripts here https://github.com/DS3Lab/DT-FM/tree/master/udp_hole_punching_test (which originally target UPD hole punching for AWS instances across different regions), I would expect there will be some necessary modification of the scripts, but it would be straight-forward.

Once you settle down this network configuration, you should be able to run the systems, although you might need to modify some of the helper scripts to fit your local environment.