RP shows tests as Passed even when several assertions failed

[Nikicyde]

Hey all!

Sorry in advance if I missed something in configuration, but I have a problem that my test is failing on several places, but it is shown in RP as passed.

I am running the tests from Azure DevOps with following command:

newman run $(serviceName)/$(serviceName)_$(devTeam).postman_collection.json \
    -g EnvironmentFiles/utils.postman_globals.json \
    -e EnvironmentFiles/$(postmanExecutionEnvironment).postman_environment.json \
    --reporters htmlextra,cli,junit,json-summary,@reportportal/agent-js-postman \
    --reporter-junit-export junitReport.xml \
    --reporter-htmlextra-export $(serviceName).html \
    --reporter-summary-json-export jsonReport.json \
    --reporter-@reportportal/agent-js-postman-debug=true \
    --reporter-@reportportal/agent-js-postman-endpoint=http://10.23.1.34:8080/api/v1 \
    --reporter-@reportportal/agent-js-postman-api-key=test2_o3hhOjKpRxeoPICvn-Qhfhmrb1U3aiIbZZhdmdQ-2PeqsAWIaMnc6AT6BsgPlbRw \
    --reporter-@reportportal/agent-js-postman-launch=$(serviceName) \
    --reporter-@reportportal/agent-js-postman-project=superadmin_personal \
    --reporter-@reportportal/agent-js-postman-attributes=Skynet \
    -x \
    --verbose

At the end of the test, I can see that there are several assertion error in Newman logs

But in RP, I see everything passed

In Postman, I have my assertions like this:

pm.test(requestName + ": Status code is 200", function () {
    pm.response.to.have.status(200);
});

pm.test(requestName + ": Correct data were set", function () {
    pm.expect(jsonData.status).to.equal(1);
    pm.expect(jsonData.registrationCode).to.equal(pm.environment.get('clientsCode'));
    pm.expect(jsonData.appName).to.equal("PostmanClient_" + suffix);
    pm.expect(jsonData.ipAddress).to.equal("192.168.0.1");
    pm.expect(jsonData.version).to.equal("1.0");
    pm.expect(jsonData.pcName).to.equal("pcName_" + suffix);
    pm.expect(moment(pm.environment.get('timestamp')).diff(moment((jsonData.codeDate).substr(0, 19)), "seconds")).not.to.be.above(2);
    pm.expect((toSeconds(jsonData.codeAge)) - (moment(timestampAct).diff(moment(pm.environment.get("timestamp")), "seconds" ))).to.be.below(1); //codeAge check
    pm.expect(Date.parse(jsonData.activationDate)).to.be.above(Date.parse(moment().subtract(11,'s'))) && pm.expect(Date.parse(jsonData.activationDate)).to.be.below(Date.parse(moment().add(11,'s')));
});

Not sure what am I doing wrong, thanks for any help!

[bart-the-wacky]

Hello. I'm facing a similar problem (assertion errors between real test results and RP test results (fails that appear as OK). Any update on this?

[lucasvannoort]

Hi, I'm facing the same issue.
I integrate this agent into a service test Postman project (launched by Newman).
The logs that I seen into my console beetween the result imported in Report Portal are not the same.
Like that technical describe by this issue.

[Nikicyde]

Hey guys, unfortunately, I was still not able to solve this.