Dotnet Test Return Code on Bitbucket Pipelines not correct interpreted

[fabEps]

Usage Information

Nuke 6.3.0 / SDK 7.0.100 / Framework net7.0 / OS Windows 10

Description

I observed a wrong behavior of the Bitbucket Pipelines Script Runner - I'm not sure if this is a bug:

Running tests in NUKE with DotNetTest won't indicate a failing build, if one of the unit tests is failing:

The reason is the returned exit code which is negative, but it should be a positive value to indicate BB a failed build. A workaround is currently adapting the build.ps1 script as follows:

if ($LASTEXITCODE) { 
	if($LASTEXITCODE -lt 0) {
		exit 1
	}
	exit $LASTEXITCODE 	

instead of the default logic:

if ($LASTEXITCODE) { 
	exit $LASTEXITCODE 	

I'm not sure if BB causes the problem or if it's possible to solve this on NUKE side.
Adapting each bootstrapper script file on creating a new build project could lead to missing this step.

Reproduction Steps

    Target RunTest => _ => _
        .DependsOn(CompileTests)
        .Executes(() =>
        {
            DotNetTest(t => t
                .SetConfiguration(Configuration)
                .EnableNoBuild()
                .SetResultsDirectory(TestResultDirectory)
                .CombineWith(TestProjects, (ts, p) => ts
                    .SetProjectFile(p)
                    .SetLoggers($"trx;LogFileName={p.Name}.trx")));
        });

Expected Behavior

Build should fail if unit test is failing:

Actual Behavior

Regression?

No response

Known Workarounds

if ($LASTEXITCODE) { 
	if($LASTEXITCODE -lt 0) {
		exit 1
	}
	exit $LASTEXITCODE 	

Could you help with a pull-request?

No

[matkoch]

Usually, anything that's not 0 should be considered a faulty execution. The majority of tools works this way. @lahma or @chaquotay uses/used Bitbucket, do you have any experience here?

[chaquotay]

We currently use Bamboo (also from Atlassian, but as far as I know not related to Bitbucket), so I don't have any Bitbucket experience 😕

[fabEps]

We're also using Bamboo for other projects (currently migrating from Bamboo to Bitbucket Pipelines), there it works like expected without adaptions.. Hence, it's Bitbucket Pipeline related.

[lahma]

We're running on bitbucket pipelines with nuke and failed tests error the pipeline. Maybe need to look into it more. Can those parallel tests affect final exit code?

[fabEps]

Hey @lahma, thanks for the input. Our script call is like:

clone:
  depth: full

pipelines:
  default:
    - parallel: # these 2 steps will run in parallel
        - step:
            name: Android build, test, publish
            clone:
              depth: full
            runs-on:
              - self.hosted
              - windows
              - maui.android
            script:
              - git branch
              - cd src
              - ./build.ps1 --target UploadAndroid --Configuration Release
            after-script:
              - trx2junit ./src/artifacts/TestResults/*.trx
            artifacts:
             - src/artifacts/*.zip
        - step:
            name: iOS build, test, publish
            clone:
              depth: full
            runs-on:
              - self.hosted
              - macos
              - maui.ios
            script:
              - git branch
              - cd src
              - security find-identity -v -p codesigning
              - chmod +x build.sh
              - ./build.sh --target UploadIos --Configuration Release
            artifacts:
              - src/artifacts/*.zip

Are there any differences to yours?

Only the first step (Android build) is executing UnitTests.

Here the additional log where I can see that NUKE build fails:

Thanks!

[lahma]

One difference for us is that we don't use an after-script, instead we inject test logger as environment variable so the output is produced without the extra conversion step:

script:
  - export POWERSHELL_EXE="/usr/bin/pwsh"
  - export TestLogger="junit"
  - export TestResultDirectory="test-results"
  - ./build.sh compile test

// file name must have .xml extension, we change it to it instead of trx if running on bitbucket
.SetLoggers($"{TestLogger};LogFileName={fileName}")

And our test project references JunitXml.TestLogger.

[fabEps]

... and I see, you are using cloud agents, we are using local Windows agents - correct?

[fabEps]

The after-script should not affect the build-result:
https://bitbucket.org/blog/after-scripts-now-available-for-bitbucket-pipelines

The commands in the after-script section run until one fails, and the result of this section does not affect the overall build status. For information, [..]

[lahma]

Yes we are running cloud agents and we don't have test parallelism on NUKE level (NUnit has parallel tests).

[fabEps]

https://confluence.atlassian.com/bbkb/pipelines-builds-always-succeeds-even-on-exit-code-status-on-non-zero-1142437808.html

This is cloud agent related, it's a unix command. As I mentioned before, it's a local Windows agent instance which leads to this behavior.
PowerShell/PowerShell#3415 -> Here they mentioned to use the exit code.
@matkoch how can I provide you more information? I'm still not sure if I need to change the configuration on BBP side or if this could be handled on NUKE side without adapting it each time for new projects.