`#[Retry]` attribute to support retrying flaky test #6182

santysisi · 2025-04-13T20:32:04Z

✨ Add Retry Attribute for Retrying Flaky Tests

This PR introduces a new Retry attribute that can be applied to individual test methods in PHPUnit. Its goal is to help deal with flaky or unstable tests that occasionally fail due to non-deterministic reasons (e.g. network issues, race conditions, external dependencies).

🔧 How it works

The Retry attribute allows a test to be automatically retried a specified number of times before being marked as failed. You can also control the delay between retries, and optionally restrict retries to specific exception types.

🧪 Usage

Here’s how you can use it:

#[Retry(3)] // Retry up to 3 times on any exception
public function testFlakyApi(): void
{
    // test logic
}

You can also specify a delay (in seconds) between retries:

#[Retry(2, delay: 2)] // Retry twice, with 2 seconds delay between attempts

And restrict retries to certain exception types only:

#[Retry(4, retryOn: TimeoutException::class)]

All parameters:

maxRetries (int) — how many times the test can be retried.
delay (?int) — seconds to wait before retrying. Default: 0
retryOn (?string) — exception type to retry on. If omitted, retries on any throwable.

✅ Example

#[Retry(2)]
public function testRetriesUntilMaxAttemptsThenSucceeds(): void
{
    static $attempt = 0;

    if ($attempt++ < 2) {
        throw new RuntimeException('Transient failure');
    }

    $this->assertSame(3, $attempt);
}

📈 Benefits

Stabilize CI pipelines by automatically retrying flaky tests
Fine-grained control over retry behavior per test
Non-intrusive: no need to change existing test logic
Optional support for retrying only on specific exceptions

🔄 Prior art

Other popular testing frameworks already provide similar functionality:

santysisi · 2025-04-13T21:44:27Z

This PR could also offer a complementary approach to the discussion in #5718: Bring back --repeat CLI option.

While --repeat focuses on re-running tests via the CLI, the #[Retry] attribute is designed for more fine-grained control, enabling retries at the test method level, optionally with delay or exception-based filtering. It might be a useful alternative for handling flaky tests directly in code, without requiring changes to CI configs or CLI commands.

Thanks again, looking forward to your feedback! 😊

sebastianbergmann · 2025-04-14T05:28:49Z

src/Framework/TestCase.php

@@ -1276,6 +1278,12 @@ private function runTest(): mixed
            }

            if (!$this->shouldExceptionExpectationsBeVerified($exception)) {
+                $metadata = $this->getRetryMetadata($exception, $attempt);


Only had a very cursory look at the proposed changes yet, but this caught my eye: how are tests retried that use expectException(), for example?

Thanks for pointing that out!

Just to clarify how this works: in PHPUnit, if an exception is thrown after calling expectException(), it's treated as a failure, not an error and failures are not retried. On the other hand, if an exception is thrown before expectException() is set, PHPUnit treats it as an error, which will trigger a retry.

For example:

#[Retry(2)] public function testFoo(): void { throw new Exception; $this->expectException(Exception::class); }

This will be retried, since the exception occurs before the expected exception is declared.

Whereas:

#[Retry(2)] public function testFoo(): void { $this->expectException(Exception::class); throw new Exception; }

This will not be retried, because the exception is expected and treated as a failure if something goes wrong, not an unexpected error.

So i believe (maybe I'm wrong) that the the retry behavior aligns with how PHPUnit distinguishes between errors and failures.

Thanks again for your comment
I really appreciate the feedback 😄 .

That said, I'm not entirely sure if this approach fully aligns with your expectations or the overall goals of PHPUnit. If there's anything I'm not comfortable with or could be improved, I'd greatly appreciate your guidance. I'd be happy to review the logic and make adjustments to better fit PHPUnit's design and intent.

sebastianbergmann · 2025-04-14T05:30:11Z

src/Framework/TestCase.php

+            return null;
+        }
+
+        $metadatas = MetadataRegistry::parser()->forMethod($this::class, $this->name())->isRetry()->getIterator();


The manual call to getIterator() is superfluous as MetadataCollection implements IteratorAggregate.

Thank you for the comment 😄
good catch. I agree with you, the manual call to getIterator() is unnecessary given that MetadataCollection implements IteratorAggregate. I'll make that change.

Thanks again for your feedback! I think everything's in order now. If you need me to tweak anything else, please let me know.

nikophil · 2025-05-01T16:00:47Z

Hey @santysisi

Thanks for this PR! Any chance that you add the CLI option along with the attribute?

sebastianbergmann · 2025-05-01T17:33:25Z

Any chance that you add the CLI option along with the attribute?

I do not see how a CLI option would be relevant in the context of what this PR is about.

sebastianbergmann · 2025-05-01T17:38:43Z

I think that the functionality you propose here is useful.

However, the current implementation can "only" be considered an early proof of concept in my opinion. Retried tests are not run on a fresh instance of the test case object, the before-test, after-test, etc. template methods are not run for retried tests, etc. Furthermore, no events are emitted to record the fact that a test was retried. At least as far as I can see.

While I do think this would be a nice addition, I will not work on a "real" implementation myself. I am open to a more complete PR, of course.

santysisi · 2025-05-09T22:11:09Z

Hi @sebastianbergmann , thanks for your comment 😄. I really appreciate it!

I'd like to give it another try, now taking the new requirements into account. Please let me know if I’m on the right track:

Fresh Test Case Instances for Retries, allowing the proper execution of lifecycle/template methods (setUp, tearDown, etc.).
Emitting Events for Retried Tests, so retries can be logged or tracked appropriately.

Does that align with what you had in mind?

Also, just to confirm, do you agree with the behavior described here

mvorisek · 2025-06-06T20:19:01Z

src/Framework/Attributes/Retry.php

+final readonly class Retry
+{
+    private int $maxRetries;
+    private int $delay;


I would expect this to be of type float to support delays like 0.5 (500 ms).

And usually this is very bad retry strategy. Exponential delay is much better. Maybe there should be no native delayed retry support as it can slowdown testing massively.

You're right, but instead of passing a float, maybe it's a better idea to replace sleep with usleep. As for the performance issue, I think the solution is simply not to pass that parameter. Still, I believe the delay is a good idea since it's optional.

In SI units definitely, ie. seconds.

I believe using milliseconds instead of seconds is a better and more flexible option, especially when it comes to controlling timeouts and delays more precisely. This approach is also more commonly adopted, for example, both Symfony's HTTP client and Guzzle use milliseconds for time-related settings.

Additionally, we could consider adding a multiplier option (similar to Symfony) to allow increasing the delay between each retry. Personally, I prefer Guzzle's approach of using a callable for this logic. However, I understand that passing closures directly in PHP attributes isn't natively supported,

Happy to hear your thoughts on this!

Milliseconds are not base SI units.

Still, I am thinking if a delay is wanted to be supported natively.

I think there can be an elegant option - the "try number" can be accessible from setUp method and the use can wait/sleep the way he needs by any code.

santysisi · 2025-06-07T17:33:52Z

Hi @sebastianbergmann , thanks for your comment 😄. I really appreciate it!

I'd like to give it another try, now taking the new requirements into account. Please let me know if I’m on the right track:

Fresh Test Case Instances for Retries, allowing the proper execution of lifecycle/template methods (setUp, tearDown, etc.).

Emitting Events for Retried Tests, so retries can be logged or tracked appropriately.

Does that align with what you had in mind?

Also, just to confirm, do you agree with the behavior described here

Hi @sebastianbergmann, I hope you're doing well 🙂
Could you please confirm if this aligns with what you had in mind, or if I might be going in the wrong direction? 🙏
Thanks in advance! 😊

sebastianbergmann · 2025-06-08T03:10:29Z

Could you please confirm if this aligns with what you had in mind, or if I might be going in the wrong direction? 🙏

I am sorry that I have not gotten around to look into this yet.

mvorisek · 2025-06-08T06:54:50Z

tests/unit/Framework/Attributes/RetryTest.php

+use LogicException;
+use PHPUnit\Framework\TestCase;
+
+final class RetryTest extends TestCase


Currently, it seems pre/post test hooks are not called. But that can be an issue.

I would like to have this tested by E2E tests to show the behaviour and also combined with other features like process isolation.

sebastianbergmann added type/enhancement A new idea that should be implemented feature/test-runner CLI test runner feature/metadata/attributes labels Apr 14, 2025

sebastianbergmann changed the title ~~add Retry attribute to support retrying flaky test~~ #[Retry] attribute to support retrying flaky test Apr 14, 2025

sebastianbergmann reviewed Apr 14, 2025

View reviewed changes

santysisi force-pushed the feature/retry-attribute-for-flaky-tests branch from 2a68f67 to 3343e8e Compare April 14, 2025 13:05

add Retry attribute to support retrying flaky test

1ff940d

santysisi force-pushed the feature/retry-attribute-for-flaky-tests branch from 3343e8e to 1ff940d Compare April 14, 2025 13:09

santysisi requested a review from sebastianbergmann April 16, 2025 12:51

sebastianbergmann force-pushed the main branch 4 times, most recently from 1bb9097 to 5af6639 Compare April 20, 2025 07:09

sebastianbergmann mentioned this pull request May 18, 2025

Extension point for single test execution #6209

Open

mvorisek reviewed Jun 6, 2025

View reviewed changes

mvorisek reviewed Jun 8, 2025

View reviewed changes

Uh oh!

#[Retry] attribute to support retrying flaky test #6182

Are you sure you want to change the base?

#[Retry] attribute to support retrying flaky test #6182

Uh oh!

Conversation

santysisi commented Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ Add Retry Attribute for Retrying Flaky Tests

🔧 How it works

🧪 Usage

✅ Example

📈 Benefits

🔄 Prior art

Uh oh!

santysisi commented Apr 13, 2025

Uh oh!

sebastianbergmann Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

santysisi Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

santysisi Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

sebastianbergmann Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

santysisi Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

santysisi Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

nikophil commented May 1, 2025

Uh oh!

sebastianbergmann commented May 1, 2025

Uh oh!

sebastianbergmann commented May 1, 2025

Uh oh!

santysisi commented May 9, 2025

Uh oh!

mvorisek Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

santysisi Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

mvorisek Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

santysisi Jun 7, 2025

Choose a reason for hiding this comment

Uh oh!

mvorisek Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

santysisi commented Jun 7, 2025

Uh oh!

sebastianbergmann commented Jun 8, 2025

Uh oh!

mvorisek Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

`#[Retry]` attribute to support retrying flaky test #6182

`#[Retry]` attribute to support retrying flaky test #6182

santysisi commented Apr 13, 2025 •

edited

Loading

santysisi Apr 14, 2025 •

edited

Loading

santysisi Apr 14, 2025 •

edited

Loading

mvorisek Jun 6, 2025 •

edited

Loading

mvorisek Jun 8, 2025 •

edited

Loading

mvorisek Jun 8, 2025 •

edited

Loading