Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor entity cleanup test to make it less flakey #2871

Merged
merged 5 commits into from
Jul 9, 2024
Merged

Refactor entity cleanup test to make it less flakey #2871

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f4692f7
unflake test
davidmrdavid Jul 8, 2024
b5b4720
reorder steps
davidmrdavid Jul 8, 2024
e7e833f
add delay
davidmrdavid Jul 8, 2024
b6c17b8
make test slightly longer
davidmrdavid Jul 8, 2024
9166608
incorporate feedback
davidmrdavid Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions test/Common/DurableTaskEndToEndTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5065,6 +5065,8 @@ public async Task DurableEntity_CleanEntityStorage(string storageProvider)
var orchestrationA = $"{prefix}-A";
var orchestrationB = $"{prefix}-B";

// PART 1: Test removal of empty entities

// create an empty entity
var client = await host.StartOrchestratorAsync(nameof(TestOrchestrations.CreateEmptyEntities), new EntityId[] { emptyEntityId }, this.output);
var status = await client.WaitForCompletionAsync(this.output);
Expand All @@ -5084,28 +5086,38 @@ public async Task DurableEntity_CleanEntityStorage(string storageProvider)
var result = await client.InnerClient.ListEntitiesAsync(query, CancellationToken.None);
Assert.Contains(result.Entities, s => s.EntityId.Equals(emptyEntityId));

// test removal of empty entity
var response = await client.InnerClient.CleanEntityStorageAsync(removeEmptyEntities: true, releaseOrphanedLocks: false, CancellationToken.None);
Assert.Equal(1, response.NumberOfEmptyEntitiesRemoved);
Assert.Equal(0, response.NumberOfOrphanedLocksRemoved);

// check that the empty entity record has been removed from storage
result = await client.InnerClient.ListEntitiesAsync(query, CancellationToken.None);
Assert.DoesNotContain(result.Entities, s => s.EntityId.Equals(emptyEntityId));
Comment on lines +5090 to +5096
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this used to occur later in the test, but there was no clear reason to do that. So I've moved this earlier, which separates the tests into 2 distinct parts: (1) testing the deletion or empty entities, and (2) testing the releasing of orphaned locks. This should make the test easier to read, I hope


// PART 2: Test recovery from orphaned locks

// run an orchestration A that leaves an orphaned lock
TestDurableClient clientA = await host.StartOrchestratorAsync(nameof(TestOrchestrations.LockThenFailReplay), (orphanedEntityId, true), this.output, orchestrationA);
status = await clientA.WaitForCompletionAsync(this.output);

// run an orchestration B that queues behind A for the lock (and thus gets stuck)
TestDurableClient clientB = await host.StartOrchestratorAsync(nameof(TestOrchestrations.LockThenFailReplay), (orphanedEntityId, false), this.output, orchestrationB);

// remove empty entity and release orphaned lock
var response = await client.InnerClient.CleanEntityStorageAsync(true, true, CancellationToken.None);
await Task.Delay(TimeSpan.FromMinutes(1)); // wait for a stable entity executionID, needed until https://github.com/Azure/durabletask/pull/1128 is merged

// remove release orphaned lock to unblock orchestration B
// Note: do NOT remove empty entities yet: we want to keep the empty entity so it can unblock orchestration B
response = await client.InnerClient.CleanEntityStorageAsync(removeEmptyEntities: false, releaseOrphanedLocks: true, CancellationToken.None);
Assert.Equal(0, response.NumberOfEmptyEntitiesRemoved);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the main thing I changed. Previously, we were deleting both empty entities and orphaned locks. We only want to remove orphaned locks because, if we remove empty entities, we could remove the entity that we need to unlock orchestration B, possibly causing this test to hang (as orchestration B will never complete)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move this line to after Assert.Equal(1, response.NumberOfOrphanedLocksRemoved); just for readability? It helps when comparing this to the lines later in this test (5121, 5122)

Assert.Equal(0, response.NumberOfOrphanedLocksRemoved);
Assert.Equal(1, response.NumberOfEmptyEntitiesRemoved);

Pretty much just think that we should check NumberOfOrphanedLocksRemoved and NumberOfEmptyEntitiesRemoved in the same order throughout this test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorporated! 9166608

Assert.Equal(1, response.NumberOfOrphanedLocksRemoved);
Assert.Equal(1, response.NumberOfEmptyEntitiesRemoved);

// wait for orchestration B to complete, now that the lock has been released
status = await clientB.WaitForCompletionAsync(this.output);
Assert.True(status.RuntimeStatus == OrchestrationRuntimeStatus.Completed);

// check that the empty entity record has been removed from storage
result = await client.InnerClient.ListEntitiesAsync(query, CancellationToken.None);
Assert.DoesNotContain(result.Entities, s => s.EntityId.Equals(emptyEntityId));

Comment on lines -5103 to -5106
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was moved to earlier in the test, lines ~5095 to 5097

// clean again to remove the orphaned entity which is now empty also
response = await client.InnerClient.CleanEntityStorageAsync(true, true, CancellationToken.None);
response = await client.InnerClient.CleanEntityStorageAsync(removeEmptyEntities: true, releaseOrphanedLocks: true, CancellationToken.None);
Assert.Equal(0, response.NumberOfOrphanedLocksRemoved);
Assert.Equal(1, response.NumberOfEmptyEntitiesRemoved);

Expand Down
4 changes: 2 additions & 2 deletions test/Common/HttpApiHandlerTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,14 +411,14 @@ public async Task WaitForCompletionOrCreateCheckStatusResponseAsync_Returns_HTTP
TaskHub = TestConstants.TaskHub,
ConnectionName = TestConstants.ConnectionName,
},
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(15),
TimeSpan.FromSeconds(3));
stopwatch.Stop();
Assert.Equal(HttpStatusCode.OK, httpResponseMessage.StatusCode);
var content = await httpResponseMessage.Content.ReadAsStringAsync();
var value = JsonConvert.DeserializeObject<string>(content);
Assert.Equal("Hello Tokyo!", value);
Assert.True(stopwatch.Elapsed < TimeSpan.FromSeconds(10));
Assert.True(stopwatch.Elapsed < TimeSpan.FromSeconds(15));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub actions sometimes run slightly slower, so needed this extra buffer to pass the test

}

[Fact]
Expand Down
Loading