Create combined GPU tarball and zip file package #8827

chilo-ms · 2021-08-24T08:22:29Z

Integrate TensorRT EP libraries into existing GPU tarball and zip files.
Following three Artifacts are the final results of combined GPU package:
onnxruntime-linux-x64-gpu
onnxruntime-win-x64-gpu
onnxruntime-java-gpu

previous onnxruntime-xxx-gpu only contains cuda ep library, therefore rename to onnxruntime-xxx-cuda and act as intermediate artifact.

Also, this PR makes small code refactor on copying trt ep libraries for nuget package in c-api-artifacts-package-and-publish-steps-windows.yml.

jywu-msft · 2021-08-24T15:14:17Z

even though the Zip-Nuget-Java Packaging Pipeline reports successful , there are still failures ? Is this expected behavior?

Check failure on line 128 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L128
Error running codesign job: 1 of 1
Check failure on line 129 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L129
GuardianErrorExitCodeException: codesign completed with an Error exit code: 254. No available target files were found and the tool was set to fail under this condition
Check failure on line 132 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L132
Error running tool 1 of 1: codesign
Check failure on line 133 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L133
Error running codesign job: 1 of 1

chilo-ms · 2021-08-24T15:51:28Z

even though the Zip-Nuget-Java Packaging Pipeline reports successful , there are still failures ? Is this expected behavior?

Check failure on line 128 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L128
Error running codesign job: 1 of 1
Check failure on line 129 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L129
GuardianErrorExitCodeException: codesign completed with an Error exit code: 254. No available target files were found and the tool was set to fail under this condition
Check failure on line 132 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L132
Error running tool 1 of 1: codesign
Check failure on line 133 in Build log

@azure-pipelinesazure-pipelines
/ Zip-Nuget-Java Packaging Pipeline
Build log #L133
Error running codesign job: 1 of 1

I saw similar log errors on other build here:
https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=168830&view=results

snnn · 2021-08-24T16:43:05Z

I have a PR that has conflict with yours. #8754 I removed bundle_dlls_gpu.bat and put the logic in tools/nuget/generate_nuspec_for_native_nuget.py, for splitting out PDB files to a separated package, otherwise we almost hit the nuget.org 250MB file size limit.

snnn · 2021-08-25T00:34:40Z

The "No available target files were found and the tool" can be temporarily ignored. We can address it later.

snnn · 2021-08-25T00:36:19Z

tools/ci_build/github/azure-pipelines/c-api-noopenmp-packaging-pipelines.yml

@@ -357,7 +453,7 @@ jobs:
  - task: BatchScript@1
    displayName: 'Bundle Native NuGet and other binaries'
    inputs:
-      filename: $(Build.SourcesDirectory)\tools\ci_build\github\windows\bundle_dlls_gpu.bat
+      filename: $(Build.SourcesDirectory)\tools\ci_build\github\windows\bundle_dlls_gpu_nuget.bat


Why this line needs be changed?

This bat file is mainly for bundling libraries for nuget package, simply rename it for clarity.
Also, we need to bundle libraries for GPU zip file, so the name "bundle_dlls_gpu.bat" is better for this case.

snnn · 2021-08-25T00:40:02Z

tools/ci_build/github/windows/bundle_dlls_gpu.bat

@@ -3,20 +3,20 @@ REM Licensed under the MIT License.

 REM for available runtime identifiers, see https://github.com/dotnet/corefx/blob/release/3.1/pkg/Microsoft.NETCore.Platforms/runtime.json


How about this: we make this file solely for the new jobs you are adding? And remove the nuget part in this file.

Yes, I can remove this link.

snnn · 2021-08-25T07:20:53Z

tools/ci_build/github/azure-pipelines/c-api-noopenmp-packaging-pipelines.yml

@@ -130,14 +133,14 @@ jobs:
    displayName: 'Download Pipeline Artifact - Win x64'
    inputs:
      buildType: 'current'
-      artifactName: 'drop-onnxruntime-java-win-gpu-x64'
+      artifactName: 'onnxruntime-java-win-x64-cuda'


Please keep the prefix unchanged. The cpu one is still "drop-onnxruntime-java-win-x64". If you don't change that, it's inconsistent. But if you change that, you will need to change a lot of places in the c-api-cpu.yml. Given we only have 3 days left, I think it's not a good time to do this.

I updated the Artifact names that are related to "GPU" and kept cpu one unchanged.

Following three Artifacts are the final results of combined GPU package:
onnxruntime-linux-x64-gpu
onnxruntime-win-x64-gpu
onnxruntime-java-gpu

(previous onnxruntime-xxx-gpu only contains cuda ep library, therefore I rename to onnxruntime-xxx-cuda and act as intermediate artifact)

i think there is something wrong with onnxruntime-win-x64-gpu zip
the copy zip file step finds 0 files.

Starting: Copy zip file to: D:\a_work\1\a

Task : Copy files
Description : Copy files from a source folder to a target folder using patterns matching file paths (not folder paths)
Version : 2.190.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/copy-files

found 0 files
Finishing: Copy zip file to: D:\a_work\1\

Like I said, it is inconsistent. The GPU part of jar packaging looks for "onnxruntime-java-win-x64-cuda", but CPU looks for "drop-onnxruntime-java-win-x64". And you'd need to modify the scripts to be compatible with both.

we identified the issue, it's somewhere else.
But yes, it would be good to keep the naming convention as consistent as possible

jywu-msft · 2021-08-25T14:24:07Z

Integrate TensorRT EP libraries into existing GPU tarball and zip files.
The tarball and zip files will be generated and publish under following Azure Pipeline Artifacts:

onnxruntime-win-combined-gpu-x64

onnxruntime-linux-x64-combined-gpu

Note: Due to original Artifacts onnxruntime-linux-x64-gpu/onnxruntime-win-gpu-x64 act as intermediate file, as well as Azure Pipeline Artifact can't be overwritten, we're going to use the name "combined" as new Artifact name for combined GPU package.

Also, this PR makes small code refactor on copying trt ep libraries for nuget package in c-api-artifacts-package-and-publish-steps-windows.yml.

please edit the description to reflect the final implemenentation.
(the artifacts will not be named -combined-

jywu-msft · 2021-08-25T15:28:39Z

tools/ci_build/github/azure-pipelines/c-api-noopenmp-packaging-pipelines.yml

+    displayName: 'Copy zip file to: $(Build.ArtifactStagingDirectory)'
+    inputs:
+      SourceFolder: '$(Build.BinariesDirectory)\zip-artifacts'
+      Contents: 'onnxruntime-win-x64-gpu-*.zip'


something wrong here? It's not finding a file. for the copy.

snnn · 2021-08-25T18:09:49Z

tools/ci_build/github/windows/bundle_dlls_gpu_nuget.bat

@@ -0,0 +1,23 @@
+REM Copyright (c) Microsoft Corporation. All rights reserved.


FYI: I will delete this file in my PR #8754

the logic will need to be moved to new location then?

Yes. Instead of injecting the generated DLLs into a nuget package, we should put them in nuspec file and let nuget packaging tool handle it. For two reasons:

It's less risky. We don't manually modify the package

Nuget tool can help us split PDB files out.

chilo-ms added 22 commits August 23, 2021 18:26

Add onnxruntime_providers_shared.dll into gpu nuget package

9f46361

Modify for test

2b7f42e

Temporarily remove for test

9618e78

Modify for test

9d00b1d

Modify for test

afdc20a

Test packging Windows combined GPU

4ab4ba7

Test packging Windows combined GPU

613daab

Test packging Windows combined GPU

0a54637

Test packging Windows combined GPU

18011bb

modify for test

85943a9

modify for test

f178ec6

fix bug

c5f2a51

Modify for test

2228f8b

Modify for test

c17d41e

Modify for test

598e4fc

Modify for test

2ca1237

Modify for test

7945d11

Modify for test

60dad90

Modify for test

3bd46f4

Modify for test

e2b71ac

Prepare for PR

4867d73

Prepare for PR

dfff0a7

chilo-ms requested review from snnn and jywu-msft August 24, 2021 08:22

chilo-ms requested a review from a team as a code owner August 24, 2021 08:22

snnn reviewed Aug 25, 2021

View reviewed changes

Code refactor

c6325db

snnn previously approved these changes Aug 25, 2021

View reviewed changes

Rename proper Artifact name

0ec42b8

chilo-ms dismissed snnn’s stale review via 0ec42b8 August 25, 2021 03:42

chilo-ms added 3 commits August 24, 2021 21:27

Rename intermediate Artifact names

fb167e1

Revert Artifact Names

3c596bb

Rename Artifact Names

ea99a37

snnn reviewed Aug 25, 2021

View reviewed changes

chilo-ms added 7 commits August 25, 2021 01:42

Modify Artifact name

26c05a3

Modify Artifact name

3dc9a8d

Modify Artifact name

13d25f2

Update Java package

d7efb04

Update Java package

a958e2a

fix bug to change artifact name

0a0b970

Fix bug for the wrong file path

a35c596

jywu-msft reviewed Aug 25, 2021

View reviewed changes

chilo-ms added 3 commits August 25, 2021 10:35

Fix no fetching correct artifact and test

af4f000

temporarily modify for test

c224408

undo the change for test

86ce149

jywu-msft approved these changes Aug 25, 2021

View reviewed changes

snnn reviewed Aug 25, 2021

View reviewed changes

jywu-msft merged commit 32ecbf4 into master Aug 25, 2021

jywu-msft deleted the trt_ep_tarball_and_zip branch August 25, 2021 20:51

chilo-ms mentioned this pull request Aug 26, 2021

Fix issue of GPU tarball/zip/java package #8850

Merged

jywu-msft mentioned this pull request Aug 26, 2021

Onboard to cibuildwheel and improve macOS build #8754

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create combined GPU tarball and zip file package #8827

Create combined GPU tarball and zip file package #8827

chilo-ms commented Aug 24, 2021 •

edited

Loading

jywu-msft commented Aug 24, 2021

chilo-ms commented Aug 24, 2021

snnn commented Aug 24, 2021

snnn commented Aug 25, 2021

snnn Aug 25, 2021

chilo-ms Aug 25, 2021

snnn Aug 25, 2021

chilo-ms Aug 25, 2021

snnn Aug 25, 2021

chilo-ms Aug 25, 2021 •

edited

Loading

chilo-ms Aug 25, 2021 •

edited

Loading

jywu-msft Aug 25, 2021

snnn Aug 25, 2021

jywu-msft Aug 25, 2021

jywu-msft commented Aug 25, 2021

jywu-msft Aug 25, 2021

snnn Aug 25, 2021

jywu-msft Aug 25, 2021

snnn Aug 25, 2021 •

edited

Loading

		@@ -3,20 +3,20 @@ REM Licensed under the MIT License.

		REM for available runtime identifiers, see https://github.com/dotnet/corefx/blob/release/3.1/pkg/Microsoft.NETCore.Platforms/runtime.json

		@@ -0,0 +1,23 @@
		REM Copyright (c) Microsoft Corporation. All rights reserved.

Create combined GPU tarball and zip file package #8827

Create combined GPU tarball and zip file package #8827

Conversation

chilo-ms commented Aug 24, 2021 • edited Loading

jywu-msft commented Aug 24, 2021

chilo-ms commented Aug 24, 2021

snnn commented Aug 24, 2021

snnn commented Aug 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chilo-ms Aug 25, 2021 • edited Loading

Choose a reason for hiding this comment

chilo-ms Aug 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Starting: Copy zip file to: D:\a_work\1\a

Task : Copy files Description : Copy files from a source folder to a target folder using patterns matching file paths (not folder paths) Version : 2.190.1 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/copy-files

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jywu-msft commented Aug 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snnn Aug 25, 2021 • edited Loading

Choose a reason for hiding this comment

chilo-ms commented Aug 24, 2021 •

edited

Loading

chilo-ms Aug 25, 2021 •

edited

Loading

chilo-ms Aug 25, 2021 •

edited

Loading

Task : Copy files
Description : Copy files from a source folder to a target folder using patterns matching file paths (not folder paths)
Version : 2.190.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/copy-files

snnn Aug 25, 2021 •

edited

Loading