-
Notifications
You must be signed in to change notification settings - Fork 387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[build] Fix and extend build scripting #1212
base: master
Are you sure you want to change the base?
Conversation
a) Fixed various syntax issues, e.g. unquoted variables b) Unwrapped nested IF statements to ensure no jumping out of loops c) Extended colour pallette to stdout d) Introduced environment variable SKIP_SDV_ACTUAL to skip Static Driver Verifier (SDV) during analysis builds. CodeQL, Code Analysis (CA) and DVL operations are still executed. Default is to perform the SDV build (unchanged behaviour). e) Make actual SDV build conditional (Win10_SDV only) f) Introduced managed CodeQL package cache and test suite versioning control. If the CodeQL binary is detected at CODEQL_BIN a CodeQL build commences, otherwise it is skipped (unchanged behaviour). The package cache versions are set in the :config_ql_whcp function (new). Test suite versioning is via git hash of the WDK Developer Supplemental Tools repository (new, depends on git). The versions are determined by the value of the WHCP_LEVEL variable, which can be either WHCP_LEGACY or WHCP_24H2. Provision for future WHCP versions is templated as WHCP_NEXT. g) Introduced environment variable CODEQL_OFFLINE_ONLY to perform package cache and suite version checking but NOT download any needed updates. Exits on error expecting resolution. Requires git. h) Introduced environment variable CODEQL_RUN_BLIND to retain legacy behaviour and perform no package cache or suite version checking. It presumes the prerequisites are present. Removes git dependency. i) Pre-builds x86 viosock libraries when building virtio-win.sln or viosock.sln for amd64 in the circumstances the required x86 libraries do not already exist. j) If running in a Germanium EWDK environment, the script will instantiate a Cobalt EWDK environment to build Win10 SDV operations, and will return to Germanium EWDK when done. k) Use Germanium EWDK to sign Win11 drivers. l) Enforce echo off after calling SetupBuildEnv.cmd m) Provide some extra EWDK details when EnterpriseWDK=True n) Prettify last leg of build and provide SUCCESS / FAIL feedback. Signed-off-by: benyamin-codez <[email protected]>
|
Thanks for having a look, Kostiantyn..! First let me say, I've got some more changes to commit. So I'll do that first and then look at breaking it up. I'll likely put these two together to avoid merge dependencies:
I'll probably drop these together as they're in the same file [
Can do.
One scenario is when testing the build system. Without having to do the actual SDV to check the tail end of the build, this is a great time saver. It's also helpful when you have already done the SDV build and you don't want to clobber it on a re-run. Another scenario is if you want to build for any target and do code analysis but not SDV. I note your mention of using Win11_SDV, but sometimes you might want to build for Win10 and not wish to expose Win11-specific options. Obviously, these are all prototyping scenarios and not building for the WHCP.
That's certainly how we do it - and I'm in favour of that. However, I followed the MS documentation here and here in implementing this framework, and that really is what I am providing with this feature, a framework for future versions. if "%WHCP_LEVEL%"=="WHCP_LEGACY" (
set CODEQL_PACK_DEL=microsoft/[email protected]
set CODEQL_PACK_GET=codeql/[email protected],microsoft/[email protected]
)
if "%WHCP_LEVEL%"=="WHCP_24H2" (
set CODEQL_PACK_DEL=
set CODEQL_PACK_GET=codeql/[email protected],microsoft/[email protected]
)
if "%WHCP_LEVEL%"=="WHCP_NEXT" (
rem USE THIS AS A TEMPLATE FOR FUTURE VERSIONS
rem NOTE: Remember to populate the CODEQL_PACK_DEL variable for all WHCP_LEVELS above with __NEXTRANGE__ packs [per WHCP_LEGACY example]...
set CODEQL_PACK_DEL=
set CODEQL_PACK_GET=codeql/cpp-queries@__NEXTRANGE__,microsoft/windows-drivers@__NEXTRANGE__
) ...and then, with the introduction of say an imaginary WHCP_25H2 and updated CodeQL packs, we might have: if "%WHCP_LEVEL%"=="WHCP_LEGACY" (
set CODEQL_PACK_DEL=microsoft/[email protected],codeql/[email protected],microsoft/[email protected]
set CODEQL_PACK_GET=codeql/[email protected],microsoft/[email protected]
)
if "%WHCP_LEVEL%"=="WHCP_24H2" (
set CODEQL_PACK_DEL=codeql/[email protected],microsoft/[email protected]
set CODEQL_PACK_GET=codeql/[email protected],microsoft/[email protected]
)
if "%WHCP_LEVEL%"=="WHCP_25H2" (
set CODEQL_PACK_DEL=
set CODEQL_PACK_GET=codeql/[email protected],microsoft/[email protected]
)
if "%WHCP_LEVEL%"=="WHCP_NEXT" (
rem USE THIS AS A TEMPLATE FOR FUTURE VERSIONS
rem NOTE: Remember to populate the CODEQL_PACK_DEL variable for all WHCP_LEVELS above with __NEXTRANGE__ packs [per WHCP_LEGACY example]...
set CODEQL_PACK_DEL=
set CODEQL_PACK_GET=codeql/cpp-queries@__NEXTRANGE__,microsoft/windows-drivers@__NEXTRANGE__
) This would allow co-existence in the future regardless of whether or not the WHCP chooses to accept DVLs built with WHCP_25H2 for WHCP_24H2 or earlier targets. It also allows you to adjust rapidly should MS change their position. Similarly, to pick a CodeQL test suite, specify in the Architecturally, I'm curious as to whether you think it worthwhile to split this framework, or the whole CodeQL part off to separate file(s). The natural extension of this line of thinking would be to consider whether it is worthwhile to split the whole _SDV build process off to separate file(s) - and maybe rename to _ANALYSIS or similar in the process. This is probably worth considering now if I'm splitting up this PR... Best regards, |
Addendum to 7348ad4. 1. Refactors changes to the CustomBuildStep Command following code review. We now perform all steps regardless of Target OS. We also now create DVL files in the Project (driver) folder with version numbers in the file name for DVL files created with legacy EWDKs, and with the "latest" label for current EWDKs. These files are then copied to the relevant .\Install folders and renamed for extant WHCP submission operations. 2. New build\makeLegacyDVLs.bat now creates all legacy DVL files. This functionally replaces build\dvl1903.bat and extends it to service any legacy DVL build. It provides a framework to manage legacy DVL creation and a common place to lay out version-specific DVL operations and manipulations. Logging is also enabled to provide feedback when read at the tail of root\buildAll.bat. Please see PR virtio-win#1212 for details on these changes. Signed-off-by: benyamin-codez <[email protected]>
Addendum to da96c89. 1. Further changes to buildAll.bat a) Added @ prefix to FOR do commands b) Extended logging around creation of COMPAT (RS1/1607) DVL files c) Added SDV and Legacy DVL (DVL files created with legacy EWDKs) reporting. The latter queries the <PROJECT>.legacy_dvl_result.txt log file created by build\makeLegacyDVLs.bat and reports accordingly. Please see PR virtio-win#1210 for more information. d) Consolidates exiting function (whether success or fail) to :leave e) Creates a scheduled task to call build\clean_build_log.bat before leaving. 2. Created build\clean_build_log.bat to remove colour pallette artefacts from build_log.txt created during HCK/CI using Jenkins config. This script also deletes the scheduled task created in 1(d) above. Signed-off-by: benyamin-codez <[email protected]>
I've pushed those changes: Addendum to da96c89.
That log clean in (2) above, should complete while signing the drivers. There are presently 8 seconds of |
Addendum to da96c89 and 74f5391. 1. Removed packaging folder from root\buildAll.bat per PR virtio-win#1209. Signed-off-by: benyamin-codez <[email protected]>
@YanVugenfirer @kostyanf14 @ybendito @vrozenfe I have removed the packaging folder from Any final thoughts before I begin the process of splitting this up as discussed above...? |
I think you can split patches for now. |
1. Fixed various syntax issues, including an unquoted variable in conditional test (Debug or Release log file name), superfluous backslashes, spaces between errorlevel tests and tested functions, MSBuild syntax consolidaiton, prefixing FOR do commands with @, etc. 2. Unwrapped nested IF statements to ensure no jumping out of loops or other unexpected behaviours. 3. Updated usage, including Win11 target and explanation regarding analysis builds (build.bat only). 4. Added analysis build specific variables CODEQL_FAILED and SDV_FAILED to init area (build.bat only). 5. Refactored fail code, removing return failure detection in favour of in-line failure detection. 6. Updated labels to use snake style notation. 7. Refactored end of build_arch function to avoide "goto :eof" and instead use ":build_arch_done" to ensure POPD occurs and ":build_arch_skip" to check for success before ending the local environment (build.bat only). 8. Do conditional (if exists) removal of sdv and codeql_db folders (build.bat only). 9. Additional commenting and stdout output (logging). 10. Remove superfluous newline spacing. 11. Removed packaging folder from build per PRs virtio-win#1209 and virtio-win#1224 (buildAll.bat only). 12. Consolidated exiting function (whether success or fail) to :leave (buildAll.bat only). Split from PR virtio-win#1212. .,ueeeeeu, e$$$$$P?$$$$$b :$$$$" '$$$$c $$$$ $$$$c $$$6 $$$$$ $$$F $$$$" $$$F $$$F $$$$ $$$F $$$$ J$$F .ccCCCCCCCCc,. ?$$$ $$$$$$e ,CCCCCCCCCCCCCCCCCCc :$$$ `$$$$$" CCCCCCCCCCCCCC" CCCCCC. `$$$. CCCCCCCCCCC" .,d.`CCCCCC. $$$t .,uuuuu,. <CCC . `"",$$$$$$L CCCCCC ?$$$ ,HHHHHHHHHHHHHb, CCC $$$$$$$$$P"""" `CCCCC `$$$ 4F"" .,uuuuuuu,,,,. `CC '=- "$$$="""$$r C><C `$$$, zuHF""`.,,,,,,. ""??HH `CC bue$:$$$$bd$$$ C ? C $$$$ -=='.ee???$eeeec,,,.`=. `C $$$$ $$P???$$$ C cC ?$$$ ,H"" .e$P"""?heeee,,."" .'P .cc-=<Cc, cC'CC' `$$$ HH jF $$$b,/"""cd$$..,,) \ dP". .,cCCCP u `$$$. .HHH H':F $$$du,$$$$ ' ";l CCCCCCCCCCCCC'.$$ $$$h HHHF H H "$$$$$F d$ 4$$ H ,z "?CCCCCCCCP J$$ :: $ ?$$$ HHH'jF.H % $P" ...'" u$ J' .d$$$$F . ....zd$P" :::d$ :$$$ HHH H HF %%,%%" %' dH j$$$$$$$$ ::: ::::: $$ ? HF %%%%%.%. " % JHf :: $$$$$$$$$. :::::::::::::::d$$ .::::::::::::: '%%%%%%%%%%/ H" ::::: $$$$$$$$$$ ::::::::::::::d$$$ :::'``````````'::: `%%%%%%%%% JH dH :::::: ?$$$$$$$$$ `:::::::::::: $$$$ ..:::::::::::::::. %%%%%%/', H" H","" `::: ?$$$$$$$$$" :::::::::::`$$$$$ ... . ::: : . `: `eeeee;".F -.H dHHHHc. $$$$$F"`.=" `:::::::` '?$$$$ ::::: .ze$eu . `: =uuuu/"z".H" HHHHHHHHHHe. "" e$$$$$$b :::: ee,. '$$ ' z$$$$$$$$$$$$$ `.zeed?"".dHF HHHHHHHHHHHHHH $$$$$beeem ': ec,"$$$$ ' : 4$$$ . $$$$?? beeeeeedHF"".HHHHHF"" _,,.. `$$$$cze= ':,,`"$$$$$$$ : $$$$$b,,d$F , .HHF""".,zeHHHHF" ,i!"` .,;i!!!i;. ?$bee` ::`""9$$$$$$$ : $$$$$$$$$$$L., c,,eeHHHHHHHHF ,i!' .;!!!!!!!!!!!!!!. . ::::""??$$$$$$$ J$$$$$??$,`'". .HHHHHHHHHHHH" ,!!'.i!!!!!!!!!!!!!!!!!! \ ::::: d$bc."??$ "$$$$$$h .; .HHHHHHHHHHHF" ,!!',! !!!!!!''''` '''<!!i ? :::: d$$$$$ :$ "$$$$$$$F uHHHHHHHHHHH" ,!! ,!! !!!" !i ::: $$$$$ $$$ """ "HHHHHHHHH ,!!',!!!! !" .d$$$$ezece$$$, ! ::: d$$$$$' ...: :::::::: 3HHHHH" ,!!',!!!!! d$$$$$$$$$$$$$$$t > ::: d$$$$$ :::: ::` .::::: HHHHF ,!!' !!!!!! 9$$P""""?$$$$$$$$" ! ; `:: $$$$$$ : ::: .::::::::: ?HH" ,!!! !!!!!!! 3$$$.``''$$$F - ; ! ! `: $$$$$$`:: :: .::::::::::: H" ,!!!'<!!!!!!!i $$$$$$$$$$ teeJ' . ! ! : d$$$$$$`:: ` .:::::::::::::: " ,!!!! <!!!!!!!! $$$$$$P??'?$$ ! !i ` d$$$$$$L:: ::::::::::: `<!!! !!!!!!!!' '$$u,"""" ,' . ; ! `!. d$$$$$$$:: .::::::: aeeee$$$$$$e <!!!!!!!' '.^$$$$$$F' ; !! !!, $$$$$$$$`: ::::::::::`""777d$$$$$$u !!!!!! ' $$ee...e% < , !! `!!i$$$$$$$L : ``````::`` d$$$$$ ?$$$b `!!!' , . $$$$$$$$ . . !! !!!,$$$$$$$ : eccec,. ,zd$$$$$$$$$$>$L; !' ; , """"""" ; \ `! !!!,?$$$$$ ` $$$$$$$$d$$$$$$$$$P""""` . . ' !!!!!!!!! . `; ` ` !, !!i"$$$$ ` $$$$$$$$$$$$$$$$$F i!!' ,' .. .. ``!!! ! `; \ ` `\`!!.$$$L >>>"""""""??$$$$k ==" !!' ' , , d$$" u":". !! ! \ `; ` i!i,`!!, ?$F $$$$UUU$$$$u,"?$$$ =" r ' $F4 ".$$LJ 3$L !!,` ', \ !!i,.`!i.`!.$ $$$$$$$$$$$ec. d$$$ f '" $$F4.$"zP"$$b."b !!,` '! . :!!!!!!i.`!i """```"=??he, ?$$$b ,i'! ! d$$$$$d$b$$$$$$F?$ `!='' ..... `!,!i ,,,, ,,, ::: $$$$ `!i 'i" $$$$$$$$$$$$$$$c`" d$$$$$$bc, "?$$$$$$$.!,`!i $$$' $$$ '::::: '$$$> !!!!! `?$$$$$$$$$$$$F d$$$$$$$$$$F,u.'$$$$$$c!i`! : .$$$"x ::::: $$$ i!!!!!, $. ``"""""" .,z$$$P"?$$$",ed$$$b $$$$$$i!!! $e $$$" $R ::::: 3$$ !!!!!!!,`$$bd"uer .dP"""?$$. ?y, "4$$$$$$,`$$$$Fi!!! " d$$$ :::::::: )$$ !!!!!!i `$$bc . "?bey,. `d$$e `$$$$F $PP"i!!!! d$$F $r :::::: $$$ )!!!!!! `$$$L".W. "?$by,`"d$$$$$$c $$$" .eeeeu '! J$$" : .:::: . """ !!!!!!!i. `??$ .,._ `"3$$$$$$$$$b,,,d$$$$$$$$c "$" ::::::' : !!!!!!!!!!!!i;;i! ueee. `"?$$$$$$$$$$$$$$$$$$$$$$$$$$$F :.::::::: `:: :: <!!!!!!!!!!!!> z$$$$$$$$$$$be..`'<<<??$$$$$$$$$$$$$$$F.i :::::::: : :: :: !!!!!!!!!!!':$$$$$$$$$$$$$$$$$$beeeeeee,. ```"`,i!!! :::::: ::: ::: : !!!!!!!!!' . ""??$$$$$$$$$$$$$$PF"""'` .,,;;ii!!!!!!!! ::::: ::: `:: :: `!!!!!!! !!!!!i,,,,... .....,,,;;;ii!!!!!!!!!!!!!!!!! ::: :: :::: :: :: !!!!!' ,!!!!!!!!!!!!",!!!!!!!!!!!!!!!!!,`!!!!!!!!!!!!! :` ':: :::: :: :: !!!!' !!!!!!!!!!!!! ,!!!!!!!!!!!!!!!!!!!i!!!!!!!!!!!!! .:: :: :::: :: :: !!!' !!!!!!!!!!!!'.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! :::: :: :::: :: :: !!' !!!!!!!!!!!!',!!!!''.,;;;;;,.`'''''''`!!!!!!!!!!!! :::: :' :::: `: `: !'.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!i;,,..`'<!!!!! ::: :' ::::: :: :: .!!!! .!!!!!!!i!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!i;;,, ' ::`.: .:''` .. . `` . ' , '!!!!i!!!!>:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ...\..>\\.\\.\\\\ ,/ |, ''```''`<<''<'<<<<<<<!!!!!!!!!!!!!!!!!!!! `\>/`\\\>\%`%%"./'%..\=///,%//,,/', ,/'' ,/ , `'''''''''''''''''!!!!!!! //>;>;\.""./';./>.`\- |`/./'','./|'"/`\\</%/'//%','::::::::::::::::!!!!!!' \.:\'.'///'">>>`,`/'. '. ,` ,/',/',|%//<</,'<^< -=-<'\\'-::``,::'.!!!'` . /==>'/%.////;.//>>,/,;,.-.`\`,..\`<.\\=\\-^//=/<<\\%'- ' .,::` ..:' '`.::: '/'/''/'/>.''</>(\<|/%<%///>\`\%</>>\\%\\\>/\'\\\\< `.,'' ..::'`.:.::::'` /',/"><|,/','/'>,/,/'>/|/>\\`|"\|<``\`\"" \<```.,,'`` ..::::..:::: ::: ::::::::::::::::::::::...::::::::::::::::::::''` ...::::::::::::::: ::: ............................................. .:::::::::::::::::::: ::: ::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::: . Signed-off-by: benyamin-codez <[email protected]>
1. Use the Germanium EWDK Build Lab environment to sign Win11 drivers. We currently use the Cobalt EWDK to sign both Win10 and Win11 drivers. Split from PR virtio-win#1212. Signed-off-by: benyamin-codez <[email protected]>
1. Enforce echo off after calling SetupBuildEnv.cmd, which will otherwise spill noise into the build log. If data amongst this noise is required, best to report it specifically. 2. Provide some extra EWDK details when EnterpriseWDK=True. Here we include the BuildLab, VS year and dev CLI version as well as the Version_Number (EWDK Build Number). Split from PR virtio-win#1212. Signed-off-by: benyamin-codez <[email protected]>
1. Pre-builds x86 viosock libraries when building virtio-win.sln or viosock.sln for amd64 in the circumstances the required x86 libraries do not already exist. Split from PR virtio-win#1212. Signed-off-by: benyamin-codez <[email protected]>
@YanVugenfirer @kostyanf14 @ybendito @vrozenfe I've done the first round of splits... Please excuse seasonal ASCII art in PR #1231. |
Addendum to da96c89, 74f5391 and a92c45c. 1. Various syntax corrections discovered during split. Signed-off-by: benyamin-codez <[email protected]>
1. Use the Germanium EWDK Build Lab environment to sign Win11 drivers. We currently use the Cobalt EWDK to sign both Win10 and Win11 drivers. Split from PR #1212. Signed-off-by: benyamin-codez <[email protected]>
1. Enforce echo off after calling SetupBuildEnv.cmd, which will otherwise spill noise into the build log. If data amongst this noise is required, best to report it specifically. 2. Provide some extra EWDK details when EnterpriseWDK=True. Here we include the BuildLab, VS year and dev CLI version as well as the Version_Number (EWDK Build Number). Split from PR #1212. Signed-off-by: benyamin-codez <[email protected]>
Addendum to cec3f9c. 1. Refactored following review to a new file: prebuild_x86_libs.bat Split from PR virtio-win#1212. Signed-off-by: benyamin-codez <[email protected]>
Addendum to cec3f9c and f290abf. 1. Remove unrequired environment variable from build.bat Split from PR virtio-win#1212. Signed-off-by: benyamin-codez <[email protected]>
IF
statements to ensure no jumping out of loopsstdout
SKIP_SDV_ACTUAL
to skip Static Driver Verifier (SDV) during analysis builds. CodeQL, Code Analysis (CA) and DVL operations are still executed. Default is to perform the SDV build (unchanged behaviour).Win10_SDV
only) f)CODEQL_BIN
a CodeQL build commences, otherwise it is skipped (unchanged behaviour). The package cache versions are set in the:config_ql_whcp
function (new). Test suite versioning is via git hash of the WDK Developer Supplemental Tools repository (new, depends ongit
). The versions are determined by the value of theWHCP_LEVEL
variable, which can be eitherWHCP_LEGACY
orWHCP_24H2
. Provision for future WHCP versions is templated asWHCP_NEXT
.CODEQL_OFFLINE_ONLY
to perform package cache and suite version checking but NOT download any needed updates. Exits on error expecting resolution. Requiresgit
.CODEQL_RUN_BLIND
to retain legacy behaviour and perform no package cache or suite version checking. It presumes the prerequisites are present. Removesgit
dependency.viosock
libraries when buildingvirtio-win.sln
orviosock.sln
foramd64
in the circumstances the required x86 libraries do not already exist.echo off
after callingSetupBuildEnv.cmd
EnterpriseWDK=True