[Issue] There are some redundant results in Runs

[AmyAn666]

Hello，
Thank you very much for this powerful tool, which provides strong support for our test scripts. However, I encountered the following two issues, I would appreciate for your help:

1. After the run is created, there is redundant information in the result. I only ran 14 cases, all passed in one execution without any retries. However, there are 20 results recorded in run in the actual report.
2. I want to get the AZURE_PW_TEST_RUN_ID in the Azure pipeline, but I didn't get the value of AZURE_PW_TEST_RUN_ID with following settings.
   
   Thanks

[alexneo2003]

1. can you share pipeline log? I'm interested only with log which started with azure:pw prefix
2. did you read the docs carefully?

[AmyAn666]

From pipeline log, I can get run id as below.

Also paste playwright.config.ts as below

outputDir: path.resolve(__dirname, './test-results/results'),
    forbidOnly: !!process.env.CI,

    reporter: (!Pat)?[
        ['list'],
        ['junit', { outputFile: './test-results/xml-report/test-report.xml' }],
        ['html', { open: 'never', outputFolder: './test-results/html-report' }],]
        :
        [
        ['list'],
        ['junit', { outputFile: './test-results/xml-report/test-report.xml' }],
        ['html', { open: 'never', outputFolder: './test-results/html-report' }],
        [
          '@alex_neo/playwright-azure-reporter',
          {
            orgUrl: '***',            
            token: Pat,            
            planId: Number((azureReport as any).planId),
            projectName: (azureReport as any).adoProjectName,
            environment: `${env}`,
            logging: true,
            testRunTitle: (azureReport as any).testRunTitle,
            publishTestResultsMode: (azureReport as any).publishTestResultsMode,
            uploadAttachments: true,
            attachmentsType: ['screenshot', 'video', 'trace'],
            testRunConfig: {     
              owner: {
                displayName: (azureReport as any).displayName,
              },
              comment: (azureReport as any).comment,                    
            },            
            testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => {   
              return testPoints.filter((testPoint) => testPoint.configuration.id === globalParameters.testPlanConfigurationEnvironmentId);                         
            },
          } as AzureReporterOptions,
        ],    
      ],

Also paste my yml:

and related log, the system cannot print out AZURE_PW_TEST_RUN_ID

[alexneo2003]

@estruyf
greetings
Sorry to bother you, but can you please help us with AZURE_PW_TEST_RUN_ID in ADO pipeline as I'm not good at working with it at all
I will be very grateful to you for this

[alexneo2003]

@AmyAn666
can you remove publishTestResultsMode: (azureReport as any).publishTestResultsMode from config, run tests again and check results?
It will publish tests results one by one

[estruyf]

@estruyf greetings Sorry to bother you, but can you please help us with AZURE_PW_TEST_RUN_ID in ADO pipeline as I'm not good at working with it at all I will be very grateful to you for this

@alexneo2003 as you implemented the AZURE_PW_TEST_RUN_ID with the setVariable method, it should automatically become available to be used in the next steps.

[alexneo2003]

@alexneo2003 as you implemented the AZURE_PW_TEST_RUN_ID with the setVariable method, it should automatically become available to be used in the next steps.

Can you look at the example and tell us what it's wrong with it?

[AmyAn666]

Thanks @alexneo2003 I removed publishTestResultsMode: (azureReport as any).publishTestResultsMode from config, run tests again, still got 20 results for 14 test cases.

After investigation, I finally found that if the case is under the same testPlan but 2 different test suites, so the result would log twice, one with correct tester the other one with Unassigned Tester. Should log once, delete the once with Unassigned Tester

@AmyAn666 can you remove publishTestResultsMode: (azureReport as any).publishTestResultsMode from config, run tests again and check results? It will publish tests results one by one

[alexneo2003]

Should log once, delete the once with Unassigned Tester

It's in your current situation Tester unassigned, but how to handle it for other users?))

Each TestCase (whether it will be in multiple suites or test planes) has unique identifier - it's TestPoint
Look, the TestPoins object has next structure

{
    "points": [
        {
            "id": 4834315,
            "url": "https://dev.azure.com/apps-apps/some-project/_apis/test/Plans/3663833/Suites/5133915/Points/4834315",
            "assignedTo": {
                "displayName": "User Name <36da45f1-dd2c-4d1f-af13-5abe46b99921\\[email protected]>",
                "id": "964caade-7fd5-69d1-8a42-f56aac6e0dc8"
            },
            "automated": false,
            "configuration": {
                "id": "991",
                "name": "Windows 10"
            },
            "lastTestRun": {
                "id": "0"
            },
            "lastResult": {
                "id": "0"
            },
            "outcome": "Unspecified",
            "state": "Ready",
            "lastResultState": "Unspecified",
            "suite": {
                "id": "5133915"
            },
            "testCase": {
                "id": "5133919"
            },
            "testPlan": {
                "id": "3663833"
            },
            "workItemProperties": [
                {
                    "workItem": {
                        "key": "Microsoft.VSTS.TCM.AutomationStatus",
                        "value": "Not Automated"
                    }
                }
            ]
        }
    ]
}

It contains suite id relation
How are you thinking - if we implement some feature like [TestCaseID] and will add [SuiteID] - it will be helpful for you or another users who using this reporter?

[AmyAn666]

@alexneo2003 Actually in our condition, I run the test cases with multiple suites once. In my mind, one test case with one environment = one test point, I expect the same test case under two different suites record one result. Could you please give us a parameter in playwright.config.ts to set whether record once or twice?

It contains suite id relation How are you thinking - if we implement some feature like [TestCaseID] and will add [SuiteID] - it will be helpful for you or another users who using this reporter?

[alexneo2003]

to set whether record once or twice?

or three or four times?))
it depends on how many suites related to test case

[AmyAn666]

Can it be a boolen, once true log once, once false log as many times as suites number? If user choose log multiple times, the Tester should be configured displayName not Unassigned

to set whether record once or twice?

or three or four times?)) it depends on how many suites related to test case

[alexneo2003]

Can it be a boolen, once true log once, once false log as many times as suites number? If user choose log multiple times, the Tester should be configured displayName not Unassigned

If I have 4 cases in different suites and this 4 cases has only one difference - suite?
If I will add config like submitOnlyOneOutcome: true how to determine which one of test cases it true?

[AmyAn666]

Thanks for your reply. In my case, the test case only executed once no matter how many suites the same case is in. As below images shows, 14 cases have 14 results, that means each case give out one result, although there are 6 cases belonging to two different suites.

If I have 4 cases in different suites and this 4 cases has only one difference - suite?
If I will add config like submitOnlyOneOutcome: true how to determine which one of test cases it true?

[AmyAn666]

@alexneo2003 If above confused, any way to avoid multiple record?

[alexneo2003]

I understand you but, maybe, you don't understand me - at this moment, without any huge refactoring, we can't publish results for specific test case (especially if it placed in different suites within one test plan)
I believe that this is for the best, because it does not matter in which suite the test case is located, it is the same and works the same, accordingly, the result is the same.

[AmyAn666]

I understand you but, maybe, you don't understand me - at this moment, without any huge refactoring, we can't publish results for specific test case (especially if it placed in different suites within one test plan) I believe that this is for the best, because it does not matter in which suite the test case is located, it is the same and works the same, accordingly, the result is the same.

Okay, thanks anyway.

[alexneo2003]

@AmyAn666
I've published beta package with feature to handle root suite

@alex_neo/[email protected]

pls, install it, read the docs about rootSuiteId config and try it.

Pls, come back with feedback

[AmyAn666]

I got an error Cannot read properties of null (reading 'continuationToken') when set rootSuiteId for a root testplanID, it has son folder and child folders.
Sometimes I got warning Check, maybe testPlanId or assigned configurations per test case, what you specified, is incorrect when set rootSuiteId for a rootsuit with requirement based suite sons

[alexneo2003]

@AmyAn666

new package, check it pls

@alex_neo/[email protected]

[AmyAn666]

@alexneo2003 Got TypeError: Cannot read properties of null (reading 'length')

[alexneo2003]

@AmyAn666
can you try new version of package

@alex_neo/[email protected]

[AmyAn666]

@alexneo2003 I still got the same error, anything to config from my side?

[alexneo2003]

@AmyAn666 can you share with me your playwright-azure-reporter config block from playwright.config.ts file
p.s. previously remove all sensitive info!!!

[alexneo2003]

@AmyAn666 and can you run your tests with new version of package @alex_neo/[email protected]
I've changed log to understand in which place it happened

[alexneo2003]

@AmyAn666 any updates?