[Bug]: Error Encountered When Generating HTML Accessibility Reports in Storybook: page.evaluate: ReferenceError: __test is not defined error

[aimad-majdou]

Describe the bug

When attempting to automate accessibility tests with the provided code snippet in Storybook's documentation : https://storybook.js.org/docs/writing-tests/accessibility-testing#setup, the test runner encounters errors related to page.evaluate: ReferenceError: __test is not defined. This occurs when attempting to generate an HTML report using axe-playwright or axe-html-reporter.

To Reproduce

1. Use the provided code snippet in Storybook's documentation for automating accessibility tests.
2. Attempt to generate an HTML report using axe-playwright or axe-html-reporter:

await checkA11y(
      page,
      '#storybook-root',
      {
        detailedReport: true,
        detailedReportOptions: {
          html: true,
        },
        verbose: false,
        axeOptions: {
          ...(storyContext.parameters?.a11y?.config?.axeOptions ?? {}),
        },
      },
      false,
      'html',
    );

4. Observe the page.evaluate: ReferenceError: __test is not defined error in the console.

reproduction: https://stackblitz.com/edit/github-dwueer?file=package.json,.storybook%2Fmain.ts&preset=node

System

Storybook Environment Info:
(node:125404) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)

  System:
    OS: Windows 10 10.0.19045
    CPU: (8) x64 Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz      
  Binaries:
    Node: 21.3.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.2.4 - C:\Program Files\nodejs\npm.CMD <----- active
  Browsers:
    Edge: Chromium (122.0.2365.80)
  npmPackages:
    @storybook/addon-a11y: ^8.0.5 => 8.0.5 
    @storybook/addon-coverage: 1.0.1 => 1.0.1 
    @storybook/addon-essentials: 8.0.5 => 8.0.5 
    @storybook/addon-interactions: 8.0.5 => 8.0.5 
    @storybook/addon-links: 8.0.5 => 8.0.5 
    @storybook/addon-themes: 8.0.5 => 8.0.5 
    @storybook/blocks: 8.0.5 => 8.0.5 
    @storybook/react: 8.0.5 => 8.0.5 
    @storybook/react-vite: 8.0.5 => 8.0.5
    @storybook/test: 8.0.5 => 8.0.5
    @storybook/test-runner: 0.17.0 => 0.17.0
    eslint-plugin-storybook: 0.8.0 => 0.8.0
    storybook: 8.0.5 => 8.0.5

Additional context

No response

[unional]

dup of storybookjs/test-runner#68

[shilman]

Thanks @unional. There is already a lot of discussion on storybookjs/test-runner#68 and work in progress, so I'm closing this as a duplicate.

[toBeOfUse]

if this issue is about errors that happen when using axe-playwright or axe-html-reporter, then it is not a duplicate of the linked issue in the test runner repository. the linked issue has the same error message as a symptom, but it occurs in the absence of those accessibility tools, so it likely has a different cause

[aliaks-ei]

Hello @aimad-majdou! 👋🏻
I ran into the same issue when trying to generate an HTML report for my components in Storybook. After some debugging, I discovered that the problem occurs with axe-html-reporter when attempting to create a file with the .html extension.

I’m not entirely sure why this happens, but one workaround that worked for me was explicitly setting the report extension to .htm instead. Functionally, it’s the same as .html format, and after making this change, the issue never reappeared.

Here’s an example of what the updated code might look like (note that the HTML report options are passed as the last argument):

await checkA11y(
  page,
  '#storybook-root',
  {
    detailedReport: true,
    detailedReportOptions: {
      html: true,
    },
    verbose: false,
    axeOptions: {
      ...(storyContext.parameters?.a11y?.config?.axeOptions ?? {}),
    },
  },
  false,
  'html',
 {
   outputDirPath: "reports",
   outputDir:  "accessibility",
   reportFileName: "a11y-audit.htm"
 }
);

Hope this helps! 🙂

P.S.
@shilman and I agree with @toBeOfUse, this is definitely not the same issue as test-runner#68