Change subprocesses to run at normal priority (#42715)

[Binary-Song]

Fixes microsoft/vcpkg#42715

The priority was changed to NORMAL_PRIORITY_CLASS so vcpkg can perform its tasks (building, zipping, etc) much faster.

[dg0yt]

I expected only a small effect in vcpkg-tool CI, but ... I see that Windows end-to-end test for this PR took only 15 min, vs. 25 min in previous runs.

FTR this might increase memory pressure, with much more active cores as described in microsoft/vcpkg#42715.

[autoantwort]

My experience is that ci times are not a reliable measurement because you never know under which conditions it was run. So local benchmarks would be more reliable.

[dg0yt]

True, but these test aren't massive workloads anyways.

[Binary-Song]

FTR this might increase memory pressure, with much more active cores as described in microsoft/vcpkg#42715.

That is true, but this can be solved by adjusting VCPKG_MAX_CONCURRENCY. On the other hand, there is currently no way of telling vcpkg to run build tools at normal priority, which is a huge waste on CI resources. In comparison, tools like cmake/ninja always build at normal priority, letting the user figure out how many cores to use without running out of mem, so I think it makes sense for vcpkg to do the same.

On a side note, zipping (the post build packaging phase) was ridiculously slow for llvm when run in low priority, that may be a 30 min vs 5 min difference (I will probably submit a benchmark when I have time).

[BillyONeal]

This was done intentionally in ras0219-msft/vcpkg@a24ccdf

@ras0219-msft Do you remember why this was set to IDLE_PRIORITY_CLASS? That seems wrong. I could imagine below normal but not idle. This might explain why we have seen such variable perf in CI.

[BillyONeal]

Is the problem fixed if you use BELOW_NORMAL rather than NORMAL? I can see an argument of not wanting to make interactive stuff on the system stuttery during a build being a reason to choose lower than normal, but idle may be triggering other forms of throttling like you describe in your issue filing.

[Binary-Song]

Unfortunately the improvement was not obvious. With VCPKG_MAX_CONCURRENCY set to 24, BELOW_NORMAL makes the compilers occupy around 10 cores, while NORMAL makes them occupy all 24 cores. I just changed the priority of Ninja.exe while it is running, and the effect was instant.

BELOW_NORMAL:


NORMAL:

[Neumann-A]

I think this is kind of intended. Otherwise machines become completely unresponsive.
Is this Win10?

I don't see the same problem with 24 AMD cores and Win11 building LLVM

[dg0yt]

How does Windows (10 vs. 11) handle power vs. efficiency cores for these scheduling levels?

[Binary-Song]

I have a win11 machine with 32 AMD cores in my workplace and it also has this issue.

[Binary-Song]

I think this is kind of intended. Otherwise machines become completely unresponsive.

I think that is a trade-off that should be left for the users to decide. Currently we already have VCPKG_MAX_CONCURRENCY for that.

[Thomas1664]

I think this is kind of intended. Otherwise machines become completely unresponsive.

I think setting process priority to idle was just a workaround for poor memory management by Windows. Also, this was changed to idle back in 2017 so I would expect this to be fixed in Windows by now. High memory usage is just a problem if you run out of swap. My MacBook sits constantly at 90% memory usage and still runs flawlessly.

Even if idle priority would still be necessary, I don't think locking users into slow builds is a good idea. At the bare minimum there should be an escape hatch.

[Binary-Song]

@BillyONeal What do you think? If you merge this, you are the hero that saves users millions of dollars 😉!

[Thomas1664]

@BillyONeal What do you think? If you merge this, you are the hero that saves users millions of dollars 😉!

Now they'll never merge this cause it means that Microsoft would loose millions of dollars

[Binary-Song]

@BillyONeal

[JavierMatosD]

This looks like the right change to me, but will bring up to the team.

[BillyONeal]

I think this is kind of intended. Otherwise machines become completely unresponsive.

That is the case. We want builds to yield to normal other work to keep the system responsive, though of course we don't want to leave perfectly good cores idle.

Is the behavior you observe affected by power management settings or something similar? It's hard to evaluate the risk vs. benefit of this change without a better understanding of why it's happening.

[BillyONeal]

Couple more questions:

• The CPU listed is a desktop part; is there somehow a battery connected to this system Windows is managing?
• Do you have power management settings turned on?

Could you try adding a call to SetProcessInformation which explicitly marks vcpkg's process as not Eco QoS?

//
// HighQoS
// Turn EXECUTION_SPEED throttling off. 
// ControlMask selects the mechanism and StateMask is set to zero as mechanisms should be turned off.
//

PowerThrottling.ControlMask = PROCESS_POWER_THROTTLING_EXECUTION_SPEED;
PowerThrottling.StateMask = 0;

SetProcessInformation(GetCurrentProcess(), 
                      ProcessPowerThrottling, 
                      &PowerThrottling,
                      sizeof(PowerThrottling));

[cenit]

microsoft/vcpkg#39935

this was also my observation, fully related to the same problem

[BillyONeal]

@cenit Would you be willing to try the SetProcessInformation workaround? I would try this but the problem doesn't repro for me on the 14900HX.

[cenit]

@cenit Would you be willing to try the SetProcessInformation workaround? I would try this but the problem doesn't repro for me on the 14900HX.

yes tomorrow I can test. Can you format your code as a patch file so that I can apply it without any possible mistake?

[BillyONeal]

@cenit Would you be willing to try the SetProcessInformation workaround? I would try this but the problem doesn't repro for me on the 14900HX.

yes tomorrow I can test. Can you format your code as a patch file so that I can apply it without any possible mistake?

#1570 ?

[cenit]

ok perfect, i'll let you know

[cenit]

OS Build: 19045.5247
CPU: i7-13850HX power plugged
Power mode: "Better performances"

Building ffmpeg with currently released vcpkg:

Building ffmpeg with noQoS vcpkg (#1570):

Building ffmpeg with normal-priority vcpkg (#1555)

Building ffmpeg with currently released vcpkg under wsl2:

The two PR are improving the situation only a little, maybe nothing if we consider errors in this very non-scientific measurement.
On the other hand, WSL2 on the same laptop clearly shows much better performances in building packages and fully usage of the CPU, with the currently released executable

[cenit]

in any case something changed in the windows kernel energy heuristic (please note all my data is related to win10, we still didn't move to 11), because i perfectly remember that the usual picture in the task manager was with only the efficiency cores taken by the compiler...

[Binary-Song]

@BillyONeal I tried. Didn't work either. Would you mind trying a demo that I wrote to see if the problem can be reproduced on your side? https://github.com/Binary-Song/vcpkg_test_project

[BillyONeal]

RE: #1555 (comment)

@cenit it looks like neither change fixes it in your case, but you also aren't seeing the 'things are only running on E cores' reported by @Binary-Song .

@Binary-Song No repro for me, sorry:

[cenit]

@cenit it looks like neither change fixes it in your case, but you also aren't seeing the 'things are only running on E cores' reported by @Binary-Song .

in microsoft/vcpkg#39935 i reported exactly that problem, that the build was happening only in efficiency cores. Maybe building ffmpeg now for the test was not the best choice due to the strange toolchain? i picked it because i needed it and it was big enough -.-" (but the title of my old issue clearly states vcpkg+cmake+ninja, so clearly i am stupid enough in not testing that... will do it tomorrow)
either that, or something happened in windows 10 in the meantime (my issue is quite old)
in any case, vcpkg on windows is clearly the slowest of the platforms

[BillyONeal]

@cenit Stupid question: Is the vcpkg directory where builds are happening excluded from Defender in your tests?

in any case, vcpkg on windows is clearly the slowest of the platforms

I agree that sucks but unfortunately given that this PR doesn't fix it that argues against taking it.

@Binary-Song I think adding a switch to do this might be acceptable but fixing one system and breaking responsiveness for everyone else doesn't seem like the correct tradeoff to me.

[Binary-Song]

@BillyONeal Yeah you are right. I tried this on the win 11 machine today and did not repro either. This is likely to be an OS bug that got fixed later. I will do some VM experiments about this. Adding a switch might be a reasonable solution if this is indeed an OS bug.

[cenit]

@cenit Stupid question: Is the vcpkg directory where builds are happening excluded from Defender in your tests?

in any case, vcpkg on windows is clearly the slowest of the platforms

I agree that sucks but unfortunately given that this PR doesn't fix it that argues against taking it.

@Binary-Song I think adding a switch to do this might be acceptable but fixing one system and breaking responsiveness for everyone else doesn't seem like the correct tradeoff to me.

on the laptop considered, there is CS and no way to change any settings.

Anyway, here there are the results when building llvm, and there is a difference wrt ffmpeg

Building llvm with currently released vcpkg:

Building llvm with noQoS vcpkg (#1570):

Building llvm with normal-priority vcpkg (#1555):

As you can see, only the normal-priority version really made a difference (and this is expected imho, since EcoQoS is not in the win10 api), but only in this case (previous experiment with ffmpeg port was much more dubious in results)

Can you explain why the llvm toolchain "sees" the different behavior while the ffmpeg one does not? Do we call external tools in vcpkg in different parts and so all of them should be upgraded to see the cpu usage improved also for ffmpeg?
In any case, is it a kernel bug for win10, since you all say that win11 does not reproduce these behaviors? What is the approach in this case?

[BillyONeal]

Given that it's clear we are unlikely to merge this in this form, I've set the PR to 'draft' status. We would be happy to accept adding an opt-in switch to turn on such behavior, but as proposed we are worried about breaking the responsiveness goals for which we intentionally run at lower priorities given that current versions of Windows do not seem to experience an issue.

[Binary-Song]

@BillyONeal How about adding an env var VCPKG_SUBPROCESS_PRIORITY? Allowed values: IDLE, BELOW_NORMAL, NORMAL, ABOVE_NORMAL, HIGH, REALTIME

[Binary-Song]

@BillyONeal Also, do I need to consider non-Windows?

[microsoft-github-policy-service]

@Binary-Song please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

[microsoft-github-policy-service]

@Binary-Song please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

[Binary-Song]

@microsoft-github-policy-service agree

[Binary-Song]

@BillyONeal