Windows Build Environment Settings BKC

[Stonepia]

This document is for environment settings for building PyTorch. If you are just using pip install for PyTorch, then all of the steps in this document are not needed. All you need is just pip install and run.

• 1. System Settings:
  • 1.1. Mandatory Step
    • 1.1.1. Long Path Enablement
  • 1.2. Suggested Step
    • 1.2.1. Developer Mode
• 2. Env Setting
  • 2.1. Option 1: Official Recommended Way
    • 2.1.1. Activate the env
    • 2.1.2. Setup build environment
    • 2.1.3. Build PyTorch
    • 2.1.4. Notes: Activate Env Manually in CMD
  • 2.2. Option 2: Use Powershell / pwsh
    • 2.2.1. Activate the Env
    • 2.2.2. Setup Build environment
    • 2.2.3. Build PyTorch
    • 2.2.4. Notes: Optional step to automatically source the env

1. System Settings:

1.1. Mandatory Step

1.1.1. Long Path Enablement

Since PyTorch will generate kernels with long file names, Long Path support must be enabled to avoid build errors.

Please refer to Maximum Path Length Limitation for details.

Open the Terminal (Admin) by right-clicking the Windows button, and execute the following:

# Enable long path for the system
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

# Enable long path for git
git config --system core.longpaths true
git config --global core.longpaths true

1.2. Suggested Step

1.2.1. Developer Mode

Windows has introduced a Developer Mode. It allows the user to create symlinks with regular users. This could help to avoid potential build error. This is not a mandatory step.

2. Env Setting

You generally need Visual Studio, oneAPI and conda in your system. Since there are a lot of documents already, this section will not cover them. Be sure to install Visual Studio before install oneAPI!

Just some commands for setting the env.

After installation, you need to activate the env in your terminal. There are two kinds of terminals in Windows: PowerShell and Command Prompt (cmd). I recommend using PowerShell for a modern development experience. Moreover, you could also try the latest PowerShell from PowerShell.

2.1. Option 1: Official Recommended Way

The official recommended way of running it would be:

2.1.1. Activate the env

1. Open the "Intel oneAPI command prompt for Intel 64 for Visual Studio 2022" shortcut from the start menu.
2. Activate conda env and run.

Normally these two steps should work. If there are some environment not activated. Please refer to Notes: Activate Env Manually in CMD section.

2.1.2. Setup build environment

You also need to set the additional env:

1. Setup the proxy:

# cmd
set https_proxy=http://ip:port
set http_proxy=http://ip:port

git config --global https.proxy http://ip:port
git config --global http.proxy http://ip:port

2. Setup the workaround flags for the build
   There is a build error in PyTorch#134989.

# cmd
set CMAKE_SHARED_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_MODULE_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_EXE_LINKER_FLAGS=/FORCE:MULTIPLE

3. Setup the AOT build target
   By default, the AOT target is in BuildFlags.cmake. Since this is for dev purpose, you don't need to build all of them.
   One could set according to the following.

# cmd
set TORCH_XPU_ARCH_LIST=mtl-h

One could find all the targets with the following:

ocloc ids --help

2.1.3. Build PyTorch

Please following the official doc for build. One thing worth noting is that, the long path might still be a problem.
Thus please put the pytorch folder in a shallow folder path (D:/pytorch for example). Otherwise, it might be wrong.

2.1.4. Notes: Activate Env Manually in CMD

In case one needs to activate the env manually:

Conda Env Activation
If your conda is not initialized, please try to initiate the conda first. You could use the following:

# cmd
"C:\Users\sdp\miniforge3\Scripts\conda.exe" init

Then close and reopen the Command Prompt (cmd).

Visual Studio Env Activation
In case one needs to activate the environment manually, one could activate it by:

# cmd
# Activate Visual Studio
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"

# Activate oneAPI
"C:\Program Files (x86)\Intel\oneAPI\<toolkit-version>\oneapi-vars.bat"

oneAPI Activation

# cmd
"C:\Program Files (x86)\Intel\oneAPI\<toolkit-version>\oneapi-vars.bat"

2.2. Option 2: Use Powershell / pwsh

PowerShell is more modern to use. Thus if there is no bug, you could try with PowerShell. It is also recommended to install the latest pwsh from PowerShell.

However, with PowerShell, one should activate all the environments manually.

2.2.1. Activate the Env

1. Setup conda env
   One should first setup the policy fo execute scripts.

Open the terminal(Admin) by right click the start menu:

# Execute in admin
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

Then open a normal PowerShell terminal and execute the following:

& "C:\Users\sdp\miniforge3\Scripts\conda.exe" init

Then close and reopen the PowerShell terminal.

2. activate the Visual Studio
   This is the option step. Since the Step 3 will automatically activate the Visual Studio env. One could also directly open the "Visual Studio Developer PowerShell" shortcut from the start value.

# Not necessarily needed. Just put it here for reference.
# Please directly use the "Visual Studio Developer PowerShell" shortcut
& 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat'

3. activate the oneAPI toolkit
   To activate oneAPI, execute the following:

# Execute if you are using the default PowerShell
cmd.exe "/K" '"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" && powershell'

# Use this if you installed the pwsh
cmd.exe "/K" '"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" && pwsh'

2.2.2. Setup Build environment

1. Setup the proxy:

# PowerShell
$env:https_proxy="http://ip:port"
$env:http_proxy="http://ip:port"

git config --global https.proxy http://ip:port
git config --global http.proxy http://ip:port

2. Setup the workaround flags for the build
   There is a build error in PyTorch#134989.

$env:CMAKE_SHARED_LINKER_FLAGS="/FORCE:MULTIPLE"
$env:CMAKE_MODULE_LINKER_FLAGS="/FORCE:MULTIPLE"
$env:CMAKE_EXE_LINKER_FLAGS="/FORCE:MULTIPLE"

3. Setup the AOT build target
   By default, the AOT target is in BuildFlags.cmake. Since this is for dev purpose, you don't need to build all of them.
   One could set according to the following.

# cmd
$env:TORCH_XPU_ARCH_LIST="mtl-h"

One could find all the targets with the following:

ocloc ids --help

2.2.3. Build PyTorch

Please follow the official doc for build.

2.2.4. Notes: Optional step to automatically source the env

One could create a profile of PowerShell and add the oneAPI source file to it. Note that this would greatly reduce the start time of the PowerShell.

In addition, It is NOT recommended to edit the profile of the default PowerShell, please install the pwsh and edit its profile.

notepad $Profile

Then add the following:

# Path to the oneAPI setvars script
$oneApiSetvarsPath = 'C:\Program Files (x86)\Intel\oneAPI\setvars.bat'

# Check if the setvars script exists
if (Test-Path $oneApiSetvarsPath) {
    # Source the oneAPI environment using cmd.exe and capture the output
    $output = cmd.exe /c "`"$oneApiSetvarsPath`" && set" | Out-String

    # Display the output
    # Write-Host $output

    # Parse the output and set the environment variables in the current session
    $output -split "`r`n" | ForEach-Object {
        if ($_ -match "^(.*?)=(.*)$") {
            [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2])
        }
    }

    # Check if the SETVARS_COMPLETED environment variable is set
    if ($env:SETVARS_COMPLETED) {
        Write-Host "oneAPI environment sourced successfully."
    } else {
        Write-Host "Failed to source oneAPI environment."
    }
} else {
    Write-Host "oneAPI setvars script not found at $oneApiSetvarsPath"
}