Making a standalone build to run on multiple PC's for comparison

[Andrew900460]

I've read past issues that sounded similar to this topic, but it appeared that certain things were out of date or just didn't give the full picture.
I do understand that BenchmarkDotNet creates a separate program that is ran for each benchmark, so I get that there are some difficulties with making standalone builds due to that.

But it does sound like it is possible one way or another to achieve, otherwise, somone woud've plainly said "no this is currently is not possible".
To clarify, I was making a standard [Benchmark] test to time certain functions. And I was reading that depending on how old your CPU is, certain x86 instructions have gotten faster in newer architectures. My CPU is a Skylake, and I have a friend who's CPU is a few years newer than mine. So I was just curious how different the benchmark results would be on his CPU.

So I tried making a build in Visual Studio, and tried various different settings, and he would always get some crash message.
So then I read another older closed issues saying somthing about a "[InProcess........]" attribute, but it was a longer name.
That attribute did not show up through intellisence, and I don't think the documentation links that were given in other issues gave any sort of example of how to even use such an attribute either way. So I took it that the page was changed over time.
So I just tried adding [InProcess], which seemed to change somthing, but it still didn't work after sending my friend a new build.

I just wish that if this was indeed possible, if there was a clear example of how this is meant to be done. Like I said, I looked at old issues and the doc links, but the small bits that talked about "InProcess..." didn't really give a concrete example on how to properly do it.
It could be just somthing's not configured in the build settings, which would be my fault.

If this isn't at all possible, I'd try to make my own micro benchmark, but it seems I would need somthing more precise than StopWatch.

[timcassell]

Since you want to compare the exact same instructions, I assume you're building a NativeAOT application, and not the default CoreCLR. [InProcess] attribute uses the InProcessEmitToolchain which does not support NativeAOT (it requires JIT). You want to use the InProcessNoEmitToolchain. You can do this by passing a config to the benchmark runner.

DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance))

It is a less commonly used toolchain and slightly less accurate than the other toolchains, hence why there is no attribute for it. I'm not entirely sure why there are no docs for it, but I assume it's because we do not want to suggest a less accurate toolchain to users. It's for niche scenarios such as yours.

[Andrew900460]

Since you want to compare the exact same instructions, I assume you're building a NativeAOT application, and not the default CoreCLR. [InProcess] attribute uses the InProcessEmitToolchain which does not support NativeAOT (it requires JIT). You want to use the InProcessNoEmitToolchain. You can do this by passing a config to the benchmark runner.

DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance))
It is a less commonly used toolchain and slightly less accurate than the other toolchains, hence why there is no attribute for it. I'm not entirely sure why there are no docs for it, but I assume it's because we do not want to suggest a less accurate toolchain to users. It's for niche scenarios such as yours.

To be fair, I'm not quite sure if the generated exe is AOT or JIT.
I have been doing "Build -> Publish Selection"
And I have tried various settings like "Self-contained", "Produce single file", "Enable ReadyToRun compilation", "Trim unsued code", etc.

But I will try to add your example code, and see if that fixes it.

couple minutes later

Yea so I took your code, assigned the return value to a variable and passed it to BenchmarkRunner.Run
Currently seeing if I can get a build to run on a computer other than the one the code is on.

some more time later

Didn't seem to work on another computer.
If this is indeed how it's meant to look:

ManualConfig config = DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance));
BenchmarkRunner.Run<MyBenchmark>(config);

Then it could just be a matter of me screwing up the build settings and doing somthing that the package doesn't support, but I don't need either way.

[timcassell]

Didn't seem to work

That's not very descriptive. It would help if you could describe what actually happens. What does the log say?

[timcassell]

https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot

[Andrew900460]

Didn't seem to work

That's not very descriptive. It would help if you could describe what actually happens. What does the log say?

Right, sorry.

So from the build folder, we ran the exe in "publish/win-x64"
And the console showed this exception:

Unhandled exception. System.UnauthorizedAccessException: Access to the path 'C:\Windows\system32\BenchmarkDotNet.Artifacts\results' is denied.
at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
at System.IO.Directory.CreateDirectory(String path)
at BenchmarkDotNet.Extensions.CommonExtensions.CreateIfNotExists(String directoryPath)
.....

And so on. There was more to the error message, but I had to manually type that all out ^
I will say that the stack trace was executing my BenchmarkRunner.Run() function.
The rest of the stack trace was internal BenchmarkRunner functions and such.
Until it eventually got to CommonExtensions.CreateIfNotExists(String directoryPath)

I mean, it is confusing to me that it would be trying to create a folder in system32 even on my computer where it does work.