Building Mailbay in Rust: an architecture tour

Mailbay is a Cargo workspace of ten crates and a Next.js app. It receives mail on port 25, sends it through KumoMTA, serves a REST API, an MCP server, SMTP submission and IMAP, and runs a job worker. One rule holds all of that together, and one testing habit keeps it honest.

The map#

• core: every business rule, as methods on one Core type.
• api (axum): REST under /v1, the WebSocket event stream and the long poll wait.
• mcp (rmcp): the MCP server over streamable HTTP, plus a stdio proxy.
• inbound: our ESMTP server for mail from the internet. submission: SMTP for apps and mail clients. imap: IMAP for Apple Mail, Outlook and Thunderbird.
• outbound: injection into KumoMTA and the receiver for its delivery logs. worker: jobs and webhook delivery. dns: record plans, live checks and DNS host writers. cli: the mailbay command.

One core, four doors#

Creating an inbox is POST /v1/inboxes, mailbay inbox create, the MCP tool create_inbox and the “New inbox” button. All four end up in the same method in core. The API handler, the tool and the CLI command parse input, call core, and format the answer. They hold no rules of their own, so the four can never disagree about limits, permissions or error messages.

Every core method takes an Actor, which is a session or an API key, and checks it before touching data:

actor.require(Permission::InboxRead, &Resource::inbox(project_id, inbox_id))?;

// sessions: workspace role, project role, optional inbox scope
// keys:     full, sending (optionally one domain) or one inbox
// invisible resources answer 404, visible but not allowed 403

Returning 404 for something you cannot see, rather than 403, means a scoped key cannot even learn that another project's inbox exists. Tests assert this for every endpoint and every role.

Data access is sqlx with plain SQL against Postgres. There is no ORM; queries map to structs with FromRow, and every query runs in some test against a real database.

The inbound server#

The SMTP server is a tokio task per connection with rustls for STARTTLS. It advertises PIPELINING, 8BITMIME, SMTPUTF8, ENHANCEDSTATUSCODES and a 25 MB SIZE. The interesting decisions are about when to say what:

• Unknown recipients are refused at RCPT TO so a message for a typo is never accepted and bounced later.
• After DATA, the mail-auth crate checks SPF, DKIM and DMARC, and rspamd gives a verdict over HTTP. A reject verdict is a 550 in the same session.
• The message is parsed with mail-parser, the raw bytes go to object storage, and the rows, attachments and thread link are written in one transaction. Only then does the client get 250.

Sending through KumoMTA#

We did not write a mail transfer agent. Queues, retries, per destination throttles and connection reuse are years of work, and KumoMTA, open source and itself written in Rust, does them well. We configure it with a Lua policy and inject messages over its HTTP API with three metadata fields: the workspace, our message id and the IP pool.

• DKIM keys stay with us. The policy asks an internal endpoint for the key of the From domain, memoizes it for five minutes, and signs with rsa-sha256. Keys are encrypted at rest with ChaCha20-Poly1305, bound to their row so a ciphertext copied elsewhere will not open.
• Each delivery, deferral, bounce and complaint comes back through a log hook to the outbound crate, which updates per recipient status, emits message.* events and adds suppressions.
• Pools travel as KumoMTA tenants, so free, paid and system mail leave from different IPs. More on that in why we run our own mail servers.

Postgres as the queue#

There is no Redis and no message broker. Background work is rows in a jobs table, claimed by any number of workers without stepping on each other:

WITH due AS (
  SELECT id FROM jobs
  WHERE run_at <= now() AND (locked_until IS NULL OR locked_until < now())
  ORDER BY run_at LIMIT $1 FOR UPDATE SKIP LOCKED)
UPDATE jobs j SET locked_until = now() + make_interval(secs => $2), attempts = j.attempts + 1
FROM due WHERE j.id = due.id RETURNING j.id, j.kind, j.payload

SKIP LOCKED lets each worker take rows nobody else holds, and the lease means a crashed worker only delays a job. Webhook deliveries are jobs too: signed with the Standard Webhooks scheme and retried after 5 seconds, 5 minutes, 30 minutes, 2, 5 and 10 hours, seven attempts over about 18 hours.

Latency comes from LISTEN/NOTIFY. Triggers notify inside the writing transaction, so a notice arrives only after commit; each process keeps one listening connection and fans notices out to waiting requests, WebSocket clients and IMAP IDLE sessions. A slow safety poll catches anything a dropped connection missed.

Errors with a fix#

Every error in core is the same struct: a kind (which becomes the HTTP status), a stable code, a message, and a fix the user can act on:

Error::new(Kind::Forbidden, "sender_not_verified",
    format!("{domain} is not a domain of this project, so Mailbay cannot send as {from}"))
  .fix(format!("Add the domain with `mailbay domain add {domain}`, then verify it with `mailbay domain verify {domain}`"))

The API returns it as JSON, the MCP server as a tool result, the CLI as text, and the SMTP submission server puts it after 550 5.7.1. Writing the fix at the place the error is raised is a small habit that makes the whole product easier to use, for people and for agents.

Tests send real mail#

Our one firm testing rule: nothing in the mail path is mocked. Database tests run against real Postgres with #[sqlx::test], which gives each test a fresh database with every migration applied. SMTP tests open real TCP connections, with TLS where the protocol needs it.

• The inbound tests speak SMTP to the server and then read the message back through core.
• The submission tests log in over TLS with a real client and check the email reaches the send log.
• The IMAP tests use a real IMAP client: log in, fetch bodies, round trip flags, and receive IDLE pushes.
• The MCP tests drive the server with a real MCP client.
• Webhook signatures are checked against the Standard Webhooks reference test vector.
• The dashboard has Playwright tests that run against a real local API and mail sink.

It makes the suite slower than one full of mocks. In exchange it exercises what mocks skip: TLS handshakes, line endings, header folding and the order in which transactions commit.