Email tests earn their reputation as flaky for three specific reasons, and none of them are mysterious. A test that polls for a message, mints its own addresses, and leaves them behind will fail intermittently on a fast machine and constantly on a loaded CI runner. Each cause has a fix that takes minutes.
Cause one: the poll races the message
The naive pattern is a sleep, then a check. It passes locally because production mail lands in under two seconds and the sleep is five. It fails in CI because CI is slower, the sender is queued, and five seconds is occasionally eight.
Two changes make this deterministic. Poll on an interval with a hard deadline instead of sleeping a fixed time, and assert on the deadline being reached rather than on a specific sleep duration. If the product under test exposes a since_id or cursor on its inbox endpoint, use it — then each poll is a cheap delta rather than a full re-read, and the test can tell "no new message yet" apart from "the wrong message is present."
Inboxmil's inbox endpoint accepts a since_id parameter for exactly this. Polling with a cursor also means the assertion can be precise: not "some message exists" but "a message newer than the one I saw at step three exists."
Cause two: the address quota
This is the failure that looks like a bug and is actually a design constraint. Mail providers cap address creation. Two independent ceilings typically apply:
- Network or account rate limits. Inboxmil's anonymous path defaults to 10 addresses per network per 24 hours, tracked per bucket and bucketed by /64 for IPv6. Signed-in users get their plan's quota instead.
- Provider key quota. The API key's own address ceiling, which falls back to a global default when unset.
A test suite that mints one address per assertion burns through both. The fix is to treat an address as a fixture, not a per-assertion resource: mint once for the run, poll it for every assertion in that run, then delete it in teardown. That changes consumption from O(assertions) to O(pipeline runs), which is one to three orders of magnitude fewer.
Cause three: leftover state
An address that is not deleted keeps counting against the quota until it expires, and it keeps appearing in the "recent" list of whatever session created it. Over a week of runs, a pipeline that never cleaned up will find itself quota-blocked with no obvious cause.
Teardown must be unconditional. Not "delete on success," not "delete if the assertion passed" — always, in a finally or its language equivalent, and it must tolerate a 404 because a previous partial run may have deleted it already. A teardown that throws on "already deleted" turns one flake into a failed pipeline.
The measurement budget
Assertions need numbers, otherwise "fast enough" is a feeling. Reasonable budgets for an email flow on a warm service:
- Address mint: under 300 ms. Inboxmil measured 143 to 237 ms across runs on production.
- Message poll after send: under 12 seconds to first sighting, which is a typical poll interval rather than a hard service limit.
- Page render with an inbox: under 800 ms authenticated, which is the budget the Inboxmil suite enforces.
Wire them as assertions, not logs. A budget that only prints is a budget that silently degrades.
What not to assert
Three things produce flake no matter how carefully they are written: exact timestamps, exact message ordering when two sends happen concurrently, and exact delivery latency. Assert on bounds and on identity — the message with this message_id arrived within the deadline — rather than on wall-clock equality.
The takeaway
Bound the wait, treat the address as a run-scoped fixture, and always clean up. Those three changes remove the flake that gets blamed on mail infrastructure, and they turn email testing from an unreliable extra into an ordinary part of the pipeline.