Give Your LiveKit Voice Agent an Email Inbox with Lumbox

Voice agents make promises they cannot keep in audio. "I'll email you a summary of what we agreed." "Send me a photo of the meter and I'll check it while we're on the line." The caller cannot replay the call, so anything they need to keep has to arrive as text, and email is where they will look for it.

LiveKit Agents handles the audio loop: speech-to-text, the model, text-to-speech and turn-taking. Email is a tool call the model makes from inside that loop. This post wires two tools into a LiveKit agent in Python, one that emails the caller a summary and one that waits for a file the caller sends mid-call, and then covers the failure modes that are specific to doing this by voice.

Setup

The agent gets one persistent inbox, created once outside the agent:

curl -X POST https://api.lumbox.co/v1/inboxes \
  -H "X-API-Key: $LUMBOX_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "front-desk"}'

{ "id": "inb_...", "address": "front-desk@trylumbox.com", ... }

Put the key, the inbox id and the address in the agent's environment as LUMBOX_API_KEY, AGENT_INBOX_ID and AGENT_INBOX_ADDRESS.

Why an async HTTP client

LiveKit function tools are async methods that run on the same event loop as the audio pipeline. A blocking network call inside a tool stalls everything else the agent is doing until it returns. The lumbox Python SDK uses a synchronous httpx client, and it has no send method, so the tools below call the REST API directly with httpx.AsyncClient.

The agent

import os
import uuid
from datetime import datetime, timezone

import httpx
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, RunContext, function_tool
from livekit.agents.llm import ToolError

LUMBOX = httpx.AsyncClient(
    base_url="https://api.lumbox.co/v1",
    headers={"X-API-Key": os.environ["LUMBOX_API_KEY"]},
    timeout=httpx.Timeout(10, read=40),
)
INBOX_ID = os.environ["AGENT_INBOX_ID"]
INBOX_ADDRESS = os.environ["AGENT_INBOX_ADDRESS"]


class FrontDesk(Agent):
    def __init__(self) -> None:
        super().__init__(instructions=(
            "You book site visits by phone. Before emailing anyone, spell their address "
            "back letter by letter and get a clear yes. If the caller wants to send a "
            f"photo or document, give them this address: {INBOX_ADDRESS}."
        ))
        self.call_id = uuid.uuid4().hex
        self.started_at = datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")

    @function_tool()
    async def email_summary(self, run_ctx: RunContext, to: str, summary: str, confirmed: bool) -> str:
        """Email the caller a written summary of what was agreed on this call.

        Args:
            to: The caller's email address, exactly as they confirmed it.
            summary: Plain-text summary of the booking and next steps.
            confirmed: True only after reading the address back and hearing yes.
        """
        if not confirmed:
            return "Read the address back to the caller and get a yes first."
        r = await LUMBOX.post(
            f"/inboxes/{INBOX_ID}/send",
            headers={"Idempotency-Key": f"{self.call_id}-summary-{to.lower()}"},
            json={"to": to, "subject": "Your site visit booking", "text": summary},
        )
        body = r.json()
        if r.status_code == 409 and body.get("error") == "idempotency_key_reuse":
            return f"A summary was already emailed to {to} on this call."
        if r.status_code == 403 and body.get("code") == "INVALID_RECIPIENT":
            reason = body["invalid_recipients"][0]["reason"]
            raise ToolError(f"Address rejected: {reason}. Ask the caller to spell it again.")
        if r.status_code == 429:
            raise ToolError("Email is busy right now. Tell the caller the summary will follow shortly.")
        if r.status_code >= 400:
            raise ToolError("The email could not be sent. Offer to read the summary out instead.")
        return f"Summary emailed to {to}."

    @function_tool()
    async def wait_for_caller_email(self, run_ctx: RunContext, sender: str) -> str:
        """Wait up to 30 seconds for an email the caller is sending during this call.

        Args:
            sender: The caller's email address or its domain.
        """
        run_ctx.session.say("Okay, I'm watching for it now.")
        r = await LUMBOX.get(
            f"/inboxes/{INBOX_ID}/wait",
            params={"from": sender.lower(), "since": self.started_at, "timeout": 30},
        )
        if r.status_code == 408:
            return "Nothing has arrived yet. Ask whether they sent it, and to which address."
        r.raise_for_status()
        email = r.json()
        files = "with an attachment" if email["has_attachments"] else "with no attachment"
        return f"Received '{email['subject']}' from {email['from']['address']} {files}."


server = AgentServer()


@server.rtc_session(agent_name="front-desk")
async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        stt="deepgram/nova-3:en",
        llm="google/gemma-4-31b-it",
        tts="inworld/inworld-tts-2:Ashley",
    )
    await session.start(room=ctx.room, agent=FrontDesk())


if __name__ == "__main__":
    agents.cli.run_app(server)

The LiveKit parts follow LiveKit's current docs: @function_tool on an async method of an Agent subclass, the docstring's Args: section describing each parameter to the model, ToolError for failures the model should explain to the caller, and run_ctx.session.say() to speak while a slow tool runs. The model descriptors in AgentSession are the examples from LiveKit's session docs; swap in whatever STT, LLM and TTS you use. While testing, adding "dry_run": true to the send body makes the API return the message it would send (sender, recipients, subject and the first 500 characters of the body) without sending anything.

Misheard addresses

Speech recognition turns "j dot smith at gmail" into several plausible strings, and an email to the wrong one is worse than none because the caller thinks it was sent. The confirmed argument forces a read-back before the tool will send. LiveKit's own tool design guide uses the same pattern with a read_back flag on a booking tool.

Lumbox catches part of what gets past that. On the Free plan, each recipient's domain is checked before sending (MX records, common typos, disposable domains), and a bad one comes back as 403 with code: "INVALID_RECIPIENT" and a list of addresses with reasons. The tool turns that into a ToolError, so the model asks the caller to spell the address again instead of reporting success. Paid plans, and organizations with a verified domain, skip that check, so there the read-back is the main defense.

Sending twice

Models sometimes call the same tool twice, for example after the caller says "yes, send it" a second time. The Idempotency-Key is built from a per-call id and the recipient, so a repeat within 24 hours replays the first response instead of sending again. If the model rewrote the summary in between, the body no longer matches the key, and Lumbox answers 409 with idempotency_key_reuse. The tool reads that as "already sent", which is the correct outcome for the caller.

Rate limits on a busy line

Sending has its own limit per organization per minute: 3 on Free, 30 on Starter, 60 on Pro and 120 on Scale. A new Free account without a verified domain is also held to 5 sends in its first hour. A handful of concurrent calls that all end with a summary will hit the Free limit, and the API answers 429 with retry_after_seconds in the body. The tool tells the model to promise the summary shortly; a production agent would put the send on a queue and retry after that many seconds.

Dead air

/wait holds the request open until a matching email arrives or the timeout passes, then returns 408. Thirty seconds of silence on a phone call feels broken, which is why the tool speaks before it waits and keeps the timeout short. since is set to when the call started, so a file the caller sent before asking ("I just emailed it") is still found. from is a substring match on the sender address, so a domain works when the caller is not sure which of their addresses they used.

Replies after the call

A caller may reply to the summary an hour later. That mail lands in the agent's inbox, and POST /v1/webhooks with an email.received subscription pushes it to your backend, signed with HMAC-SHA256 in X-Lumbox-Signature. Lumbox does not retry a failed delivery, so keep a periodic list call as a backstop. From there you might create a CRM note, schedule a callback, or answer with POST /v1/inboxes/:id/reply so the response stays in the caller's thread.

One inbox for the agent is the right default. Per-caller inboxes add nothing for sending, and each one counts against the plan (3 inboxes on Free). To send as bookings@yourcompany.com instead of trylumbox.com, add a custom domain first. The request fields used above are in the sending docs, webhooks for agent email covers the inbound side, and the quickstart gets you a key and an inbox.