Could we avoid the flake + complexity of $nextTick by bundling a tiny retry helper?

[JessicaSachs]

👋🏻 A while ago I wrote a tiny JS helper to work around some Vue Test Utils flake. It's almost identical to the usage of waitFor by the Testing Library. I'd love to bundle this utility with VTU because it gives users the option to retry their assertions through multiple event loop updates.

Right now, there's a class of flaky race conditions that depend on when Vue's render loop is updated (and how often). When a component is refactored in a certain way, flake can arise -- even when $nextTick is used correctly. This happens most commonly with components that rely on watch internally. Refactors to components should (ideally) never break tests where the DOM contract hasn't changed. Currently, we must instruct users to use await wrapper.trigger // setValue, setData, etc in conjunction with flush-promises (components that use Axios, etc). Using a single retry helper would allow us to remove flush-promises from our documentation and make the nuances of awaiting the event loop less of a focal point in our guides.

I wrote about this kind of a retry helper in a blog post back in November (link). Here's the stand-alone Gist of it -- it's actually VTU agnostic, so I'm happy to publish it as a mini standalone library. Regardless, I think it would give our users an API that is resilient to the async nature of reactive components.

I'll inline the vanilla usage here. Ideally, we'd ship this on wrapper (??) or in an exported function from vtu.

import { retry } from './retry'

describe('A flaky test', () => {
  it('succeeds when the DOM is updated much later', async () => {
    setTimeout(() => {
      const h1 = document.createElement('h1')
      h1.innerText = 'Hello!'
      document.body.appendChild(h1)
    }, 500)
    
    retry(() => expect(document.body.innerHtml).toContain('Hello!'))
  })
})

Thoughts?

[cexbrayat]

IMHO I don't think we should package it in VTU. This kind of high level instruction would be more suited to a VTU plugin published a as standalone (as vue-router-mock for example).

We probably don't all have the same style of testing, but in my experience it is really important to keep control of the underlying event loop. VTU should offer all the low-level tools, and third-party plugins can add high-level features. Regarding your example, you probably don't want to really wait 500ms (or more!) in a unit test: you're going to use a fake timer to travel to the future and make the test instantaneous :)

[JessicaSachs]

As testing library authors, we understand quite a bit about testing. We know the intricacies of using fake timers and causing the event loop to tick multiple times. I agree that having this low level control can be useful for some types of testing. Many users do not get these concepts.

The reason this proposed helper function may make sense is that even for simple components, VTU is not deterministic. You can easily refactor a component to use different language features of Vue and cause the test to become flaky — even if you are using $nextTick correctly.

As a testing framework, our first job is to remove flake. If we require users to use libraries (testing library) that provide this helper function to prevent flake, then perhaps we should bundle the helper in our API. I don’t really want to make a “pretty” API — I want to be un-opinionated — but if the requirement for VTU to be stable is that users must bundle a third party framework to work around flake, then I think that’s a gap in the API.

I think higher-level frameworks should be a choice to make VTU prettier, but VTU needs to provide a standalone stable base.

I say

• keep the low-level $nextTick API, but add a utility to wrap it
• Change the timeout. Most tests will pass after the first setTimeout(cb, 1), but if not, waiting 500ms is a bit much

[lmiller1990]

I actually like the idea of including this. One thing someone said that stuck with me is "I can just install test utils and I have everything I need". While you technically do, you need to understand the event loop and the such, something which is not super obvious or accessible to newcomers.

One of the reasons testing library is popular is it has a great UX, partly due to things like this. I guess it would both be on wrapper and as a stand alone function.

[afontcu]

I'm a bit torn on this one, to be honest 😂:

1. If we understand VTU as a tool that embeds itself to Vue core to allow people to test components, then a retry helper is… not needed, and could be provided as a plugin (this is why we created the whole plugin interface, right? 😄). VTU already offers the required low-level tools to create plugins and other libraries on top of it (see Vue Testing Lib).

2. However, from a user point of view, being able to simply tell a test "wait until this happens" is quite useful and one of the Testing Lib features I love the most. That being said, users still need to know if something is not there because something's wrong, or rather if that thing is not there "yet" (and thus they need to await/retry).

---

So, given (1), what if… we suggested people to use Testing Library as a way to hide the complexity of $nextTick and flushPromises? Even Vue 3 docs recommend using VTL over VTU, so we could do the same thing here.

FWIW, VTL support for Vue 3 is almost done

[JessicaSachs]

1. The main reason I added the plugins interface to allow users to add their own syntactic sugar. (I didn’t want to implement getElementByWhatever in VTU, everyone has their own opinions, etc). I agree that this helper could be provided as a plugin, and I think that’s a great option.
2. We could recommend VTL next to the “$nextTick” + “flushPromises” section in the docs. We could also recommend this light weight plugin (or even Cypress component testing), which also builds on top of VTU and solves the same issues.
3. I wrote the docs for testing in Vue 3 with Ben in September and recommended VTL there. 😂

Whatever decision we make is fine by me. I just wanted to check if it made sense bundled in core, because it’s practically required by users to have flake-free, deterministic tests.

Also yay, VTL + Vue 3 🥳

[wobsoriano]

I am currently using https://github.com/sindresorhus/p-wait-for since @testing-library/vue doesn't include the vm in the returned object of a rendered component unlike @vue/test-utils. My use case is unit testing with https://vue-query.vercel.app.

An example:

import { mount } from '@vue/test-utils'
import { useQuery, VueQueryPlugin } from 'vue-query'
import { defineComponent } from 'vue'
import { test, expect } from 'vitest'
import waitFor from 'p-wait-for'

test('todos', async () => {
  const App = defineComponent({
    setup() {
      const { data, isSuccess } = useQuery('todo', () =>
        fetch('https://jsonplaceholder.typicode.com/todos/1').then(
          (res) => res.json
        )
      )

      return {
        data,
        isSuccess,
      }
    },
  })

  const wrapper = mount(App, {
    global: {
      plugins: [VueQueryPlugin],
    },
  })

  await waitFor(() => wrapper.vm.isSuccess)

  await expect(wrapper.vm.data).resolves.toMatchObject({
    userId: 1,
    id: 1,
    title: 'delectus aut autem',
    completed: false,
  })
})

Would be nice to have a function like waitFor :D

[lmiller1990]

I'm curious if this can be implemented using a plugin? https://test-utils.vuejs.org/guide/extending-vtu/plugins.html

[wobsoriano]

Oh I think there's a broken link on the plugins page. PR sent https://github.com/vuejs/test-utils/pull/1367/files