A common starting point for email testing is one shared account, something like qa-tests@ on a mail provider, and it holds up until two tests run at the same time. Then run A reads run B's code, the failure looks random, and someone adds a sleep. Giving every test its own inbox fixes the collision, but "its own inbox" only isolates a test if four separate things hold: the address is unique, the key running the tests cannot touch inboxes that are not test inboxes, each wait only considers the messages meant for it, and nothing is left behind to crowd out the next run. This post goes through each one as it works in CI with the Lumbox API, whatever test framework sits on top.
1. A unique address per test
curl -s https://api.lumbox.co/v1/inboxes \
-H "X-API-Key: $LUMBOX_API_KEY" -H "Content-Type: application/json" \
-d '{"name": "ci-8121-w2-k9x2qa"}'
{"id": "inb_...", "address": "ci-8121-w2-k9x2qa@trylumbox.com",
"name": "ci-8121-w2-k9x2qa", "status": "ACTIVE", "created_at": "..."}The name becomes the local part of the address, lowercased, with anything outside a-z, 0-9 and hyphen turned into a hyphen. Addresses are unique across the service, and asking for one that exists returns 409. Leave name out and Lumbox generates a random local part, which never collides but gives you nothing to search for later. A name built from the CI run number, the worker index and a few random characters avoids collisions between parallel workers, between retries of the same job, and between pipelines, and it leaves a prefix that a cleanup script can match on.
2. A CI key that cannot see anything else
An organization-level key can list, read and delete every inbox in the organization, including any that production agents depend on. A cleanup script with a bug and that key could delete one of them. Projects fence this off. Create a project for CI with an org key, then mint a key that belongs to the project:
curl -s https://api.lumbox.co/v1/projects \
-H "X-API-Key: $LUMBOX_ORG_KEY" -H "Content-Type: application/json" \
-d '{"name": "ci"}'
# {"id": "proj_...", "name": "ci", "slug": "ci", "inbox_count": 0, ...}
curl -s https://api.lumbox.co/v1/projects/proj_.../keys \
-H "X-API-Key: $LUMBOX_ORG_KEY" -H "Content-Type: application/json" \
-d '{"name": "github-actions"}'
# {"api_key": "ak_...", "project_id": "proj_...", ...} the key is shown onceInboxes created with a project key are pinned to that project, and every inbox lookup made with the key (list, wait, read, delete) only sees the project's inboxes. Project keys are also refused on organization-level routes such as domains, webhooks and the credential vault. The key has its own rate-limit bucket too, since the limit of 120 requests per minute is counted per key, so a heavy test run does not eat into the budget of the keys your agents use.
3. Each wait sees only its own messages
A unique inbox still holds more than one message over a test's life: a first code, a resent code, a welcome email your app sends after verification. Which one a wait returns depends on two rules in the long-poll endpoints.
GET /v1/inboxes/:id/waitconsiders messages received at or aftersince. Whensinceis omitted it means "from the moment this request arrived", so mail that landed between the click and the wait call is skipped.- Both
/waitand/otplook at the newest message matching the filters. Withhas_otp=true, if the newest message has no code, the wait keeps going rather than falling back to an older one.
So pass since explicitly. For the first message, the inbox's own created_at works well because the server stamped it, which rules out clock drift on the runner. For anything later, such as a resend, use the previous message's received_at plus one millisecond. If your framework makes client timestamps easier, take one before the action that sends the mail and subtract a few seconds. Add subject or from filters (both substring matches) when your app sends more than one kind of email in the same flow.
/otp has one more condition worth knowing for tests: it only returns codes from messages whose keyword-based category is verification. A login-code email without "verify" or "verification code" in it can have its code extracted and still be categorized general, so /wait?has_otp=true is the safer default.
4. The concurrency ceiling
Lumbox counts live inboxes per organization, and POST /v1/inboxes returns 402 with code: "PLAN_LIMIT_EXCEEDED", limit and current once you reach the plan's number.
| Plan | Live inboxes |
|---|---|
| Free | 3 |
| Starter | 10 |
| Pro | 50 |
| Scale | 250 |
The count is organization-wide, so shards on separate machines share it, and so do any inboxes outside the CI project. The arithmetic to check is shards times workers per shard, which has to stay at or below the limit minus inboxes used elsewhere. A fixture that retries on 402 with a short backoff turns the limit into a queue rather than a failure, at the cost of a slower run.
5. Nothing left behind
Inboxes have no expiry. There is no TTL field on create, and an inbox exists until something calls DELETE /v1/inboxes/:id, which removes it along with its emails. Teardown hooks handle the normal case. Runners that get killed mid-test skip their teardown, and each one leaks an inbox that keeps counting against the limit, so add a sweep as the last CI step:
# sweep_inboxes.py
import os
from datetime import datetime, timedelta, timezone
import httpx
PREFIX = "ci-"
cutoff = datetime.now(timezone.utc) - timedelta(minutes=30)
headers = {"X-API-Key": os.environ["LUMBOX_API_KEY"]}
with httpx.Client(base_url="https://api.lumbox.co", headers=headers) as api:
stale, cursor = [], None
while True:
params = {"limit": 200}
if cursor:
params["cursor"] = cursor
r = api.get("/v1/inboxes", params=params)
r.raise_for_status()
page = r.json()
for inbox in page["data"]:
created = datetime.fromisoformat(inbox["created_at"].replace("Z", "+00:00"))
if (inbox["name"] or "").startswith(PREFIX) and created < cutoff:
stale.append(inbox["id"])
if not page["has_more"]:
break
cursor = page["cursor"]
for inbox_id in stale:
api.delete(f"/v1/inboxes/{inbox_id}").raise_for_status()
print(f"deleted {len(stale)} stale inboxes")# .github/workflows/e2e.yml (excerpt)
- name: Sweep leftover test inboxes
if: always()
env:
LUMBOX_API_KEY: ${{ secrets.LUMBOX_CI_KEY }}
run: python sweep_inboxes.pyThe script collects IDs first and deletes afterwards because the list cursor is the ID of the last inbox on the page, and deleting that inbox mid-walk would leave the next request pointing at nothing. The 30-minute cutoff keeps it away from inboxes that a pipeline running in parallel is using right now. Run with the project key from step 2, it cannot reach anything outside CI.
What per-test inboxes do not isolate
New inboxes live on the shared trylumbox.com domain. If your signup form rejects domains it considers disposable, tests will fail on that check, which has nothing to do with isolation. Verify a domain you own and pass it as domain when creating inboxes. The request budget is shared as well: a test costs three calls (create, wait, delete) because a long-poll is one request however long it waits, and at 120 per minute that allows roughly 40 test starts a minute per key. A 429 response carries retry_after_seconds in its JSON body along with X-RateLimit-* headers.
Framework-specific versions of this setup are in the Playwright guide, the Cypress guide and the Selenium guide, and the inbox limit for each plan is on the pricing page.