Help running Playwright TypeScript testcase in Azure Devops Pipeline with Artillery

[pditmar]

I am trying to use some of our functional testscripts and run them with Artillery in an Azure DevOps pipeline. I am using the example Github pipeline yaml with some small modifications:

#trigger:
#  - main

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'
    displayName: 'Install Node.js v20.x'

  - script: npm install -g artillery@latest
    displayName: 'Install Artillery'
 
  - script: mkdir $(System.DefaultWorkingDirectory)/reports
    displayName: 'Make reports directory'
 
  - script: artillery run --output reports/report.json apps/e2e/online.yml
    displayName: 'Execute load tests'
 
  - script: artillery report --output reports/report.html reports/report.json
    displayName: 'Generate HTML report'
 
  - publish: $(System.DefaultWorkingDirectory)/reports
    artifact: artillery-test-report

The test fails to start with the following error :

Script contents:
artillery run --output reports/report.json apps/e2e/online.yml
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/dd785957-3e87-4393-975a-01758b9791e6.sh
Test run id: txnhc_xe3ntbrpz74588nzqaafzq67jdnne_mhye
Bundled Typescript file into JS. New processor path: /home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/dist/FindLocation.spec.js
worker error, id: 2 Error: Cannot find module '@playwright/test'
Require stack:
- /home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/dist/FindLocation.spec.js
- /opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/node_modules/@artilleryio/int-core/lib/runner.js
- /opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/node_modules/@artilleryio/int-core/index.js
- /opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/lib/artillery-global.js
- /opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/lib/platform/local/worker.js
    at Function._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Function._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/FindLocation.spec.ts:1:30)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Object..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Function._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/dist/FindLocation.spec.js',
    '/opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/node_modules/@artilleryio/int-core/lib/runner.js',
    '/opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/node_modules/@artilleryio/int-core/index.js',
    '/opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/lib/artillery-global.js',
    '/opt/hostedtoolcache/node/20.15.1/x64/lib/node_modules/artillery/lib/platform/local/worker.js'
  ]
}

I've been looking at the docs and here in the knowledge base but I cannot seem to figure out what I am doing wrong. What am I missing here?

[bernardobridge]

Hi @pditmar 👋 ,

This might be related to #3263 . It's something I'll be looking at ASAP to try to reproduce it, as we weren't able to reproduce the issue on our local setup at the time. I will try to do it against a fresh pipeline.

In the meantime, can you confirm if the issue gets resolved if you install the @playwright/test package yourself?

[pditmar]

Hi!

I tried that also but got another error. I added this to the pipeline :

  - script: npm i -D @playwright/test
    displayName: 'Install Playwright module'

The error was :

Bundled Typescript file into JS. New processor path: /home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/dist/FindLocation.spec.js
worker error, id: 2 Error: Playwright Test did not expect test.describe() to be called here.
Most common reasons include:
- You are calling test.describe() in a configuration file.
- You are calling test.describe() in a file that is imported by the configuration file.
- You have two different versions of @playwright/test. This usually happens
  when one of the dependencies in your package.json depends on @playwright/test.
    at TestTypeImpl._currentSuite (/home/vsts/work/1/s/node_modules/playwright/lib/common/testType.js:71:13)
    at TestTypeImpl._describe (/home/vsts/work/1/s/node_modules/playwright/lib/common/testType.js:104:24)
    at Function.describe (/home/vsts/work/1/s/node_modules/playwright/lib/transform/transform.js:256:12)
    at Object.<anonymous> (/home/vsts/work/1/s/apps/e2e/src/e2e/post-deploy/serviceContact/FindLocation.spec.ts:6:6)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Object..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Function._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)

If you need help reproducing I am available to show the error on our environment. We can setup a screen share.

Thanks for the quick response!

[bernardobridge]

Hi @pditmar ,

Two things:

• Can you please make sure you're installing the @playwright/test version that matches the current Artillery version in our compatibility sheet?
• Are you using test.describe() anywhere in your test script? If you can share the contents of your typescript test file that would help. Thanks!

[pditmar]

I have tried setting versions, also went back 2 versions. That didn't help. I am using test.describe, I disabled that part and then it fell over the next line of code "test". This is the content of the spec file :

import { expect, test } from '@playwright/test'

// eslint-disable-next-line import/namespace
import { nlFindLocationPage } from '../../../pages/nlFindLocationPage'

test.describe('Find location @acc', () => {
  test('Find location', async ({ page }) => {
    const searchTerm = 'Somecity'
    const locationName = 'Name'

    // find  location and go to locationPage
    await nlFindLocationPage(page).visit()
    await nlFindLocationPage(page).verifyVisit()
    await nlFindLocationPage(page).performSearch(searchTerm)
    await expect(
      page.getByRole('heading', { name: `${locationName} - in ${searchTerm}` })
    ).toBeVisible()
    await page.getByRole('link', { name: 'Ga naar website' }).nth(1).click()
    await nlFindLocationPage(page).verifyVisitLocation(searchTerm, locationName)
  })
})

The included page (nlFindLocationPage) is as follows :

/* eslint-disable playwright/no-force-option */
import { visitAndCheck } from '@test/playwright-helpers'
import { expect, Page } from '@playwright/test'

import { accHomePage } from './accHomePage'
import { cookiePage } from './cookiePage'

export const nlFindLocationPage = (page: Page) => {
  const accUrl = accHomePage(page).accUrl
  const url = '/pathhere'
  const urlLocation = '/anotherpathhere'

  const visit = async () => {
    await visitAndCheck(accUrl, page, url)
  }

  const verifyVisit = async () => {
    await expect(
      page.getByText('text here')
    ).toBeVisible()
    await expect(page.getByText('Hoofdkantoor').first()).toBeVisible()
  }

  const performSearch = async (SearchTerm: string) => {
    await cookiePage(page).handleCookies()
    await page.locator('#SearchTerm').fill(SearchTerm)
    await page.getByRole('button', { name: 'Zoek' }).click()
  }

  const verifyVisitLocation = async (searchTerm: string, locationName: string) => {
    await expect(page).toHaveURL(`${accUrl}${urlLocation}`)
    await expect(page.getByRole('heading', { name: `Tekst here ${searchTerm}` })).toBeVisible()
    await expect(page.getByText(locationName).nth(1)).toBeVisible()
  }

  const verifyMobileVisitLocation = async (searchTerm: string) => {
    await expect(page).toHaveURL(`${accUrl}${urlLocation}`)
    await expect(page.getByRole('heading', { name: `Tekst here ${searchTerm}` })).toBeVisible()
  }

  return {
    visit,
    verifyVisit,
    verifyVisitLocation,
    verifyMobileVisitLocation,
    performSearch
  }
}

Let me know if you need anything else.

[bernardobridge]

Hi @pditmar ,

You cannot use test.describe or any Playwright Test Runner features, only the Playwright library directly. I suggest you look at our documentation for Playwright as well as this example on reusing existing Playwright tests as Artillery tests.

While you can reuse most of your code, you will have to adapt the code to use Artillery's function signatures.

[pditmar]

Hi @pditmar ,

You cannot use test.describe or any Playwright Test Runner features, only the Playwright library directly. I suggest you look at our documentation for Playwright as well as this example on reusing existing Playwright tests as Artillery tests.

While you can reuse most of your code, you will have to adapt the code to use Artillery's function signatures.

Thanks, I have been able to run my scripts with the instructions and examples provided.