Lots of Random Failures on Find Operations when running on Build Server

[groogiam]

I'm running into an issue where we are getting a great deal of random assertion failures do to the waiting for elements on our builds servers. It seems to be dependent on load but our build server and typically happens when more than one build is running (even ones that do not use bunit). Typically this manifests as a Find throwing an ElementNotFoundException. Rerunning the build after the load has died down often resolves the issue but is resulting in very poor build analytics. Is this expected? Is there anything that can be done to help mitigate this? Thanks.

[linkdotnet]

Hey @groogiam. Of course builds should be as deterministic as possible. That said it is hard to help from here.
We would need a few basic information. For example which version of bUnit do you use? At best it would be the latest version 1.9.8, which fixed a range of errors in conjunction with race conditions.
If you are on the latest version of bUnit, maybe it would be possible for you to provide a minimal and reproducible example (your component under test and the test itself).

What keeps me puzzled is that other unrelated tests are failing which seems to indicate some pressure on the build agent itself. That would explain also the timeout as it just takes longer to render components. Do you have any monitoring on your CI pipeline?Even if one bUnit test would go rogue I wouldn't expect other failing tests.

[egil]

I think we might be misunderstanding each other. The WaitForX methods all appear have a default timeout of 1 second before they throw. I was thinking having some configuration around this default timeout may help in situations like this. This might be cleaner in situation were we know we have cpu pressure than overriding that time manually. But I suppose that I could just add a constant for the timeout into the tests and set that manually everywhere as well. Just seems cleaner to have some control over it at the driver level.

I see. Perhaps that something we can look into.

However, I would like to understand your tests better first. What happens in your component during a test, that can take more than one second to complete, even on a slow a machine?

[groogiam]

That is the part I can't figure out. All the async operations are just using the MockHttp library referenced in the documentation. There is nothing that should take more than maybe a few hundred milliseconds to serialize. Most of the mocks are using the method below. But just last night I had a failure of a WaitForElement that had the timespan set to 3 seconds on a server with nothing else running other than the xunit. There are some other pretty heavy tests for the server side and EF core components components that spin up docker containers so I am wondering if maybe that is causing the thread starvation. I'm currently doing some experimentation trying to limit the total threads for xunit to see if that helps.

public static MockedRequest RespondOData<T>(this MockedRequest request, T content)
        {
            const string contextFieldName = "@odata.context";

            //ensure proper serialization settings. JObject makes it much harder to apply serialization settings.
            var contentJson = JsonSerializer.Serialize(content, ODataJsonSerializerOptions);

            request.Respond(req =>
            {
                var response = new HttpResponseMessage(HttpStatusCode.OK);

                JObject returnJson;
                string entityName;
                string odataContext;
                if (typeof(T).IsAssignableTo(typeof(IEnumerable)))
                {
                    entityName = typeof(T).GetGenericArguments().First().Name;
                    odataContext =
                        $"{req.RequestUri?.GetLeftPart(UriPartial.Authority)}/odata/$metadata#{entityName}";
                    returnJson = new JObject
                    {
                        ["value"] = JArray.Parse(contentJson)
                    };
                }
                else
                {
                    entityName = typeof(T).Name;
                    odataContext =
                        $"{req.RequestUri?.GetLeftPart(UriPartial.Authority)}/odata/$metadata#{entityName}/$entity";
                    returnJson = JObject.Parse(contentJson);
                }

                returnJson[contextFieldName] = odataContext;

                response.Content = new StringContent(returnJson.ToString());
                response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
                return response;
            });
            return request;
        }

[egil]

If there is no real async operation happening because the response is just mocked, them you might have a deadlock that only show up in special conditioner on CI, perhaps due to slower hardware or different OS.

What you can do is run your test with dotnet test --blame and see if you get a dump file you can download and see what threads are doing what. You can take a look at our verification.yaml workflow to see how we do that.

Aside: I also generally advice against using HttpClient directly in components and wrap it in a service that you mock instead. Then you avoid the heavy mocking/emulation needed with HttpClient. It also allows you to more easily add a caching layer and similar at the service level and not having to change the component.

[linkdotnet]

On a site note: I made a small poc how to have a global timeout. Where this is trivial to implement, it might be too small of a barrier for folks just to increase the default timeout instead of fixing the root issue. So having "the pain" of going through all the code to increase might be more cumbersome than just fixing ;)

[egil]

On a site note: I made a small poc how to have a global timeout. Where this is trivial to implement, it might be too small of a barrier for folks just to increase the default timeout instead of fixing the root issue. So having "the pain" of going through all the code to increase might be more cumbersome than just fixing ;)

Yeah, unless you are writing end-2-end tests using bunit and calling real services then there should never be a need to extend the default, so that's why I am hesitant to add that feature.

We could increase it to 10 seconds and it should not change test runtime, since that's the max wait time, not the time that is actually waited. I'm also very sure that the problem would not be fixed for the Greg if he increased the timeout everywhere.

[egil]

If you are getting ElementNotFoundException on tests that pass on your local machine but fail in CI, it's very likely you have asynchronous code in your components that execute slower in CI, and causes the timings of your test to be different.

bUnit has a bunch of methods to get around that:

• WaitForAssertion
• WaitForState
• WaitForElement/WaitForElements (these are probably the ones you need instead of Find/FindAll)

The docs for WaitForAssertion explain the scenario's where you need those and the logic behind them: https://bunit.dev/docs/verification/async-assertion.

We dont have docs for WaitForElement/WaitForElements yet, bit the basic logic behind them is the same, it's just the return value that differs.

Hope this helps.

[groogiam]

Ok we have a good mix of Finds and Waits. I'll go through and try changing the finds to Waits and see if that helps with the transient failures. I'm pretty sure there are instances where the waits have failed perhaps due to load on the build server discussed above. But if that is the case then I think I can just increase the wait timeouts.

Any thoughts on adding a feature to allow setting of the default wait timeouts globally?

Both your and @linkdotnet responses have really helped to clarify this issue for me. Thanks for your help.

[groogiam]

I recently found a bug in our test logic that did not account for id attribute generation that relied on some static state. Essentially the id's of the components would change based on how many were instantiated and not disposed which would cause the selectors to fail under load. That being said we did implement many of the suggestions in this discussion and those have helped greatly in our unit testing in terms of maintainability, failure rates and performance.

A key take away from his is be very diligent about static state when unit testing your components, especially if you target web assembly where the static context being scoped to a browser windows does not always align with the expected static context of a unit test process.