Flaky and Slow Tests

Some tests don't give the same result every time. They call a service that sometimes times out, test something that is non-deterministic by nature, or now and then take far too long. NUnit has attributes for each of these cases.

Tip

A flaky test is often a sign of a real problem, such as a race condition or state shared between tests. Use these attributes to keep your build reliable while you investigate, not to hide bugs.

Retrying a test that sometimes fails

[Retry] runs the test again when an assertion fails, up to the number of attempts you give. The test passes as soon as one attempt passes.

public class ExternalServiceTests
{
    [Test]
    [Retry(3)]
    public void Service_Responds()
    {
        // If the assertion fails, NUnit runs the test again, up to 3 attempts in total.
        Assert.That(CallService(), Is.EqualTo("OK"));
    }

    [Test]
    [Retry(3, RetryExceptions = [typeof(TimeoutException)])]
    public void Service_Responds_EvenAfterTimeouts()
    {
        // Also retried when the call throws a TimeoutException.
        Assert.That(CallService(), Is.EqualTo("OK"));
    }

    private static string CallService() => "OK";
}

The count is the total number of attempts, so [Retry(3)] means one run and up to two retries. By default, only assertion failures cause a retry. List the exceptions that should also cause a retry in RetryExceptions.

Allowing some failures

For systems that are non-deterministic by design, such as tests of AI-based features, you can accept that a test sometimes fails. [Repeat] with RequiredPassPercentage runs the test several times and passes when enough of the runs pass.

public class RecommendationTests
{
    [Test]
    [Repeat(20, RequiredPassPercentage = 90)]
    public void Recommendation_IsUsuallyRelevant()
    {
        // Passes when at least 18 of the 20 runs pass.
        Assert.That(GetRecommendation(), Is.Not.Empty);
    }

    private static string GetRecommendation() => "NUnit";
}

Set StopWhenOverallResultDetermined = true to stop repeating as soon as the outcome is certain. [Repeat] without a percentage is also a good way to find a flaky test: repeat it a hundred times and see whether it ever fails.

Note

RequiredPassPercentage and StopWhenOverallResultDetermined were added in NUnit 5.0.

Tests that take too long

[MaxTime] fails a test that takes longer than the time you give, in milliseconds. With WarningTime, a test that is slower than expected, but still within the limit, gives a warning instead. The test is never interrupted: NUnit measures the time when it finishes.

public class PerformanceTests
{
    [Test]
    [MaxTime(2000, WarningTime = 500)]
    public void Search_IsFastEnough()
    {
        // A warning above 500 ms, a failure above 2 seconds. The test is never interrupted.
        var result = Enumerable.Range(1, 1000).Where(n => n % 7 == 0).ToList();
        Assert.That(result, Is.Not.Empty);
    }
}

Tests that hang

To stop a test that runs too long, use [CancelAfter]. NUnit passes a CancellationToken to the test and cancels it when the time is up. The test must pass the token on to the code it calls, so that the work actually stops.

public class DownloadTests
{
    [Test]
    [CancelAfter(5000)]
    public async Task Download_Completes(CancellationToken cancellationToken)
    {
        // NUnit cancels the token after 5 seconds. Pass it on so the work actually stops.
        var content = await DownloadAsync(cancellationToken);
        Assert.That(content, Is.Not.Empty);
    }

    private static async Task<string> DownloadAsync(CancellationToken cancellationToken)
    {
        await Task.Delay(10, cancellationToken);
        return "content";
    }
}
Note

The older [Timeout] attribute only works on .NET Framework, and is reported as a test failure on .NET 5 and later. For code that can't be cancelled, dotnet test --blame-hang-timeout stops the whole test run when a test hangs.

See also