Skip to content

Testing

Jump to bottom
Raine Revere edited this page Dec 28, 2024 · 21 revisions

Quick Start

yarn test            # unit and jsdom tests
yarn test:puppeteer  # puppeteer (docker required)
yarn test:ios        # webdriverio (browserstack account required)

Stack

  • Vitest
  • JSDOM
  • React Testing Library
  • Puppeteer
  • Browserless
  • Docker
  • WebdriverIO
  • Github Actions

Reporting Bugs

When reporting a bug, use the following template to make it easy for others to reproduce the issue and understand what is not working. Describing something as "wrong", "not working", "broken", etc, is not sufficient. Broken behavior can only be understood in terms of the difference between current and expected behavior.

Steps to Reproduce

Describe the exact steps needed for someone else to trigger the unexpected behavior.

Current Behavior

The current (wrong) behavior that is observed when the steps are followed. Typically this refers to the main branch. When describing a regression in a PR, this refers to the PR branch.

Expected Behavior

The expected (intended) behavior that should occur when the steps are followed. Typically this refers to the behavior that has not yet been implemented. When describing a regression on a PR branch, this refers to the behavior on main.

Here's a real example from #2733:

Steps to Reproduce

- x
  - b
  - a
  - =sort
    - Alphabetical
      - Desc
  1. Set the cursor on x.
  2. Activate New Subthought Above (Meta + Shift + Enter).
  3. Move cursor up/down.

Current Behavior

  • Cursor up moves the cursor from the empty thought to a.
  • Cursor down: Nothing happens.

Expected Behavior

  • Cursor up should move the cursor from the empty thought to x.
  • Cursor down should move the cursor from the empty thought to b.

Test Levels

The project has multiple levels of automated testing, from single function unit tests up to realistic end-to-end (E2E) tests that run tests against an actual device or browser.

Use the lowest level that is sufficient for your test case. If your test case does not require a DOM, use a unit test. If it requires a DOM but is not browser or device-specific, use an RTL test. Higher level tests may provide a more realistic testing environment, but they are slower and, in the case of webdriverio on browserstack, cost per minute of usage.

You can find the test files spread throughout the project in __test__ directories.

[INSERT DIAGRAM]

1. Unit Tests

⚡️⚡️⚡️ 1–20ms

Basic unit tests are great for testing pure functions directly.

Related tests: actions, selectors, util

2. Store Tests

⚡️⚡️⚡️ 1–20ms

The shortcut tests require dispatching Redux actions but do not need a DOM. You can use the helpers createTestStore and executeShortcut to operate directly on a Redux store, then make assertions about store.getState(). This allows shortcuts to be tested independently of the user device.

Related tests: shortcuts

3. JSDOM Tests

⚡️⚡️ 1–1000ms

Anything that tests a rendered component requires a DOM. If there are no browser or device quirks, you can get away with testing against an emulated DOM (jsdom) which is cheaper and faster than a real browser.

Related tests: components

4. E2E Tests

⚡️ 1–2s

E2E, or End-to-End, tests involve running a real browser or device and controlling it with an automation driver. You can perform common user actions like touch, click, and type. These tests are the slowest and most expensive to run.

  • puppeteer (Chrome) - Requires docker
  • webdriverio (Mobile devices) - Requires a browserstack account

To run WebdriverIO tests, add BROWSERSTACK_USERNAME=your_username and BROWSERSTACK_ACCESS_KEY=your_access_key to .env.test.local in the project root and run npm run test:e2e:ios.

Related tests: e2e

5. Visual snapshot tests

⚡️ 1–2s

Snapshot tests are a specific type of puppeteer test used to prevent visual regressions. They automate taking a screenshot after your changes and comparing it to a reference screenshot. If the screenshot differs by a certain number of pixels, then it is considered a regression and the test will fail. In the case of a failed snapshot test, a visual diff will be generated that allows you to see why it failed.

In this example, the superscript position broke so the snapshot test failed. The expected snapshot is on the left; the current snapshot is on the right.

font-size-22-superscript-1-diff

When running the tests locally, a link to the visual diff will be output in your shell. When running the tests in GitHub Actions, the visual diff can be downloaded from the artifact link added to the test output under "Upload snapshot diff artifact":

Screenshot 2024-11-08 at 11 30 25 AM

If you are absolutely sure that the change is desired, and your PR was supposed to change the visual appearance of em, then run the snapshot test with -u to update the reference snapshot.

Manual Test Cases

Various test cases that may need to be tested manually.

Touch Events

  • Enter edit mode (#1208)
  • Preserve editing: true (#1209)
  • Preserve editing: false (#1210)
  • No uncle loop (#908)
  • Tap hidden root thought (#1029)
  • Tap hidden uncle (#1128-1)
  • Tap empty Content (#1128-2)
  • Scroll (#1054)
  • Swipe over cursor (#1029-1)
  • Swipe over hidden thought (#1147)
  • Preserve editing on switch app (#940)
  • Preserve editing clicking on child edge (#946)
  • Auto-Capitalization on Enter (#999)

Render

Test enter and leave on each of the following actions:

  1. New Thought

  2. New Subthought

  3. Move Thought Up/Down

  4. Indent/Outdent

  5. SubcategorizeOne/All

  6. Toggle Pin Children

  7. Basic Navigation

    - x
  - y
    - z
      - r
        - o
    - m
      - o
    - n

  8. Word Wrap

    - a
  - This is a long thought that after enough typing will break into multiple lines.
  - forcebreakkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
  - c

  9. Toggle Table View

    - a
  - =view
    - Table
  - b
    - b1
  - c
    - c1

  10. Table View - Column 2 Descendants

    - a
  - =view
    - Table
  - c
    - c1
      - c2
        - c3

  11. Table View - Vertical Alignment

    - a
  - =view
    - Table
  - b
    - b1
    - b2
    - b3
  - c
    - c1
    - c2
    - c3

    - a
  - =view
    - Table
  - b
    - This is a long thought that after enough typing will break into multiple lines.
  - c
    - c1

    - a
  - =view
    - Table
  - This is a long thought that after enough typing will break into multiple lines.
    - b1
    - b2
  - c
    - c1

    - a
  - =view
    - Table
  - This is a long thought that after enough typing will break into multiple lines.
    - b1
    - b2
  - c
    - c1

  12. Expand/collapse large number of thoughts at once

    - one
  - =pinChildren
    - true
  - a
    - =view
      - Table
    - c
      - c1
        - c2
          - c3
            - c4
    - This is a long thought that after enough typing will break into multiple lines.
      - b1
      - b2
    - oof
      - woof
  - x
    - =pinChildren
      - true
    - y
      - y1
    - z

  13. Nested Tables

    - a
  - =view
    - Table
  - b
    - =view
      - Table
    - b1
      - x
    - b2
      - y

Tips and Tricks

Database operations and fake timers

It looks like we must use fake timers if we want the store state to be updated based on database operations (e.g., if we use initialize() to reload the state). I think this is because the thoughtspace operations are asynchronous and don't call the store operations prior to the test ending. (I'm not sure why we didn't get other errors that made this clear.)

https://github.com/cybersemics/em/pull/2741

// Use fake timers here to ensure that the store operations run after loading into the db
vi.useFakeTimers()
await initialize()
await vi.runAllTimersAsync()
Clone this wiki locally