Some data is only visible after logging in: a supplier portal's price list, the analytics dashboard of a tool your team pays for, the member area of an industry association. A scraper that only reads public pages cannot see it. A scraper that logs in can, and it then inherits every problem of a logged-in user: sessions expire, sites ask for an emailed code from a new device, and the account belongs to someone who is responsible for what it does.

This post covers building that kind of scraper around one account with a persistent Lumbox inbox: creating or connecting the account once, reusing the browser session, and logging back in by email when the session dies. It deliberately does not cover account pools or identity rotation. Spreading requests across many accounts to stay under a site's limits is how scrapers get sites to lock everyone out, and it is outside what Lumbox is for.

First question: should this be a scraper at all?

Before automating a login, check three things. Does the site offer an API or a data export for the same data? That is almost always more stable than scraping. Do the terms of service allow automated access for your account type? Many business tools permit it for your own data and forbid it otherwise. And is this your account, or one your organisation holds, so that the access is yours to automate? If the answers are no API, allowed, and yours, a logged-in scraper is a reasonable tool.

Why the account's email address matters

Scrapers usually break at login, not at extraction. Modern login flows send an email in several situations: the initial verification, "we noticed a sign-in from a new device" codes, passwordless magic links, and password resets. A headless browser on a server looks like a new device to most sites, so a scraper tied to a person's inbox ends up waiting for someone to forward a code.

Giving the scraper's account its own Lumbox inbox moves those emails somewhere the scraper can read them. Lumbox inboxes do not expire, which matters here in a way it does not for one-off verification: the login code that arrives eight months from now goes to the same address. If you are converting an existing account, change its email in the site's settings to the new address; if you are creating one, see the registration walkthrough for that step.

Reusing the session

Playwright can save a browser context's cookies and local storage to a file and load them into a new context later. Most runs should start from that file and never see a login page.

import { chromium, type Page } from "playwright";
import { existsSync } from "node:fs";

const STATE = "state/supplier-portal.json";

export async function scrape(urls: string[]) {
  const browser = await chromium.launch();
  const context = await browser.newContext(existsSync(STATE) ? { storageState: STATE } : {});
  const page = await context.newPage();
  const rows = [];
  try {
    for (const url of urls) {
      await page.goto(url);
      if (page.url().includes("/login")) {
        await logIn(page);
        await context.storageState({ path: STATE });
        await page.goto(url);
      }
      rows.push(await extract(page));
      await page.waitForTimeout(3_000);
    }
  } finally {
    await browser.close();
  }
  return rows;
}

extract is your page-specific parsing and logIn is shown below. Detecting the logged-out state by URL is the simplest check that works on most sites; use whatever the site actually does, such as a login heading or a 401 from its API. The fixed pause between pages is deliberate. One account reading at a human pace is unremarkable; one account reading hundreds of pages a minute looks like what it is. If the site publishes a rate limit or a crawl delay, use that number.

If the site keeps its session token in IndexedDB rather than cookies, pass indexedDB: true to storageState; Playwright added that option for exactly this case.

Logging in by email

When the session has expired, the scraper logs in with the stored password, and the site may answer with a code sent by email. Some sites skip the password and send a magic link instead. One helper handles both:

const API = "https://api.lumbox.co/v1";
const headers = { "X-API-Key": process.env.LUMBOX_API_KEY! };
const INBOX_ID = process.env.SCRAPER_INBOX_ID!;

async function logIn(page: Page) {
  const since = new Date(Date.now() - 60_000).toISOString();
  await page.getByLabel("Email").fill(process.env.SCRAPER_INBOX_ADDRESS!);
  await page.getByLabel("Password").fill(process.env.SCRAPER_PASSWORD!);
  await page.getByRole("button", { name: "Log in" }).click();

  if (!page.url().includes("/verify")) return;

  const q = new URLSearchParams({ from: "supplier-portal.com", since, timeout: "90" });
  const res = await fetch(`${API}/inboxes/${INBOX_ID}/wait?${q}`, { headers });
  if (res.status === 408) throw new Error("login email did not arrive; stopping");
  const { parsed } = await res.json();

  if (parsed.otp_codes.length > 0) {
    await page.getByLabel("Code").fill(parsed.otp_codes[0]);
    await page.getByRole("button", { name: "Continue" }).click();
  } else if (parsed.magic_links.length > 0) {
    await page.goto(parsed.magic_links[0]);
  } else {
    throw new Error("login email had no code or link");
  }
}

This uses /wait rather than Lumbox's shorter /otp endpoint on purpose. /otp only returns codes from emails its parser categorised as verification. New-device login emails tend to be worded as security notices ("a sign-in from a new device"), and the parser can file them under security instead, in which case /otp waits out its full timeout while the code sits in the inbox. /wait matches on sender and time regardless of category and returns the whole parsed email, with otp_codes and magic_links both extracted. Magic links are recognised by URL patterns such as magic-link, passwordless or a login?...token= query.

since is set a minute before the login attempt because this inbox is permanent and holds older login emails from the same site; without a lower bound, /wait would return the newest of those, which may be last week's code. The minute of slack covers clock differences between your machine and Lumbox's server.

When to stop instead of retry

The helper throws rather than looping, and the scraper should stop, not try harder, in these cases:

  • The login email does not arrive. Retrying the login sends more emails and can trip the site's own abuse checks.
  • The site shows a CAPTCHA or asks for a phone number. It has decided it wants a person. Alert one.
  • Responses turn into 429s or block pages. Slow down or stop for the day. Creating another account to continue is how accounts get banned.

Lumbox has its own limits in the same spirit. The API allows 120 requests per minute per key, and a scraper like this one makes a request only when it has to log in.

Keeping the credentials out of the code

The password, the Lumbox API key and the inbox ID all come from the environment in the examples above. In production, put them in a secret manager and scope the Lumbox key as narrowly as the setup allows: a project key sees only its project's inboxes, so a leaked scraper key cannot read the rest of your organisation's mail. The saved session file is a credential too, since anyone holding it is logged in as the account, so keep it out of version control.

For the lighter case of content behind a one-time "enter your email" form rather than a full account, see email-gated content for agents. How codes and login links reach an agent more generally is covered in the 2FA post, and the inboxes docs list every /wait filter.