Feature : Enable Intel GPU/XPU finetuning and inference

[abhilash1910]

What does this PR do?

Motivation: Thanks for creating this repository. At Intel, we are using this repository for finetuning and inference for our GPUs /XPU (Ipex backend) and hence the reason for this PR.
This change would help us run LLama v2 without issues on our GPU devices. (Since accelerate/peft already has support of XPUs enabled)

Fixes # (issue) -NA

Feature/Issue validation/testing

Feature for finetuning and inference tested on our Data Centre GPUs with the change

for CLA purposes this would be on behalf of Intel.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Thanks for contributing 🎉!

[HamidShojanazeri]

@abhilash1910 thanks the PR, very interesting, would love to learn more about how we can test Intel GPUs/XPUs. I sent you a message on LD.

[chauhang]

@abhilash1910 Thanks for submitting this PR. Please attach the logs for the tests for verification that things are working on the Intel GPUs with the changes. We don't have a mechanism to test this.

Can you also please check if torch.set_default_device api can be used to simply the device setting logic for the XPU devices?

[abhilash1910]

@chauhang sure I will attach the logs; also as discussed with @HamidShojanazeri there is a plan to support device for integration testing for any future PRs from Intel side. I am closing on it this week internally , and eventually would like to discuss on the next steps.

[abhilash1910]

@chauhang @HamidShojanazeri logs for distributed finetuning on 8 ranks :
7B_8ranks_intel_xpu.log
Could you also share a tentative timing for a discussion sometime this week or next on possible next steps?

[HamidShojanazeri]

Thanks very much @abhilash1910, I am wondering if there is a way to add support/ get intel GPU memory stats as well the way we are getting for nvidia.

[abhilash1910]

@HamidShojanazeri yes the memory stats are also added , this can be found in the new log here:
7B_8ranks_intel_xpu.log
Also requesting re-review . Thanks for suggestions.