[Spark][2.4] Fix flaky test suite DeltaSourceDeletionVectorsSuite #2001
Conversation
| new StreamManualClock(System.currentTimeMillis())), | ||
| AdvanceManualClock(10L * 1000L), | ||
| StartStream(Trigger.ProcessingTime(1000), clock), | ||
| AdvanceManualClock(1000L), |
There was a problem hiding this comment.
this part is basically no different from before .. isnt it?
There was a problem hiding this comment.
Yeah, just got rid of some unnecessary large numbers.
There was a problem hiding this comment.
Makes it clearer imo that all the units match up as well.
There was a problem hiding this comment.
But the numbers themselves dont matter for the runtime since its manual clock.. isnt it? just making sure.
Making consistent is definitely great.
There was a problem hiding this comment.
Yeah, the numbers are irrelevant. This purely a readability improvement.
| CheckAnswer((0 until 15): _*), | ||
| AssertOnQuery { q => | ||
| // Make sure we only processed a single batch since the initial data load. |
There was a problem hiding this comment.
The stream hadn't always stopped processing before we executed the 2 DML commands. So they'd sometimes end up together in the same batch and sometimes in two different batches, because they were processing even before the second increase of the clock. The thing that finally fixed it was adding the waiting for the stream to settle down before even starting on the DML commands.
Description
(Cherrypick of 67c4b98 to branch-2.4)
Fixes #1982
How was this patch tested?
Testing-only PR.
Does this PR introduce any user-facing changes?
No.