Runbooks

When something breaks. Pages mirror the failure modes in Observability, but with concrete commands and decision trees.

Severity legend. P1 = customer-visible outage. P2 = degraded experience. P3 = internal tooling broken but public site healthy.

1. Public site returns 500 across all pages — P1#

1. Check Vercel → matters-web → Deployments. Roll back to the last green deployment via "Promote to Production".
2. While the rollback propagates, check Sentry for the first error timestamp. If it's a Mongo connection failure, check Atlas dashboard for cluster health + IP allowlist.
3. If it's a WordPress 5xx fanning out, check WordPress host. Public-site fetches have a short cache; a full WordPress outage will appear as 502s on /blogs/* first.
4. Communicate via the engineering Slack channel within 5 minutes of detection.

2. Workspace login loop (Clerk session not sticking) — P2#

Symptom: User authenticates → bounced back to /auth/login.

1. Check Clerk dashboard → Production → Session → Cookie configuration. The cookie Domain must be exactly .matters.ai (note the leading dot).
2. Check the browser's devtools cookie tab: is the Clerk session cookie set with that domain?
3. If yes and still bouncing, check apps/workspace/proxy.ts is matching the path. The isPublicRoute matcher should be ["/auth(.*)", "/api/webhooks/clerk(.*)", "/api/health"].
4. Last resort: redeploy matters-workspace. The middleware bundle sometimes ships stale.

3. WordPress content stale after a publish — P3#

1. Check Vercel → matters-web → Logs → filter for /api/revalidate. Did the webhook arrive?
   • No → check the WordPress plugin status. Trigger a manual revalidation:

     curl -X POST https://matters.ai/api/revalidate \
       -H "Content-Type: application/json" \
       -H "x-webhook-secret: $WORDPRESS_WEBHOOK_SECRET" \
       -d '{"contentType":"post","slug":"<the-stale-slug>"}'
     

   • Yes but 401 → secret mismatch. Compare WORDPRESS_WEBHOOK_SECRET in Vercel against the WordPress plugin settings.
   • Yes and 200 → revalidation fired but content still stale. CDN cache. Open https://matters.ai/<path>?_=$(date +%s) to bust the URL cache; if still stale, file Vercel support.

4. Bulk-invite stuck / partial — P2#

Symptom: /api/workspace/webinars/email/bulk-invite reports failed > 0.

1. Get the audit log entry:

   getAuditLogs({ eventType: "webinars.bulk-invite", limit: 1 });
   

   The metadata field has { test_mode, sent, failed, total }.
2. Cross-reference Sentry for [bulk-invite] errors in the same time window. Most likely: AWS SES bounce / sandbox-mode limit.
3. Check SES console → Sending statistics. If the bounce rate spiked, pause the campaign and investigate.
4. To resume from where it stopped: webinars currently retry by re-running the bulk-invite — SES dedupes on Message-ID, so duplicates won't double-send.

5. Salesforce sync failures — P3#

1. Audit log: getAuditLogs({ eventType: "webinars.sync-salesforce", limit: 1 }).
2. Look at metadata.failed count and individual failures[] in the response.
3. Common causes:
   • 403 from Salesforce → CRM credentials rotated. Update SALESFORCE_* env vars in Vercel.
   • Field validation failure → dial-code field ID changed. Update SALESFORCE_DIAL_CODE_FIELD_ID.
   • Web-to-Lead rate limit → reduce concurrency in the next sync call's body (concurrency: 1 or 2).

6. Rate-limit blocking a legitimate user — P3#

Symptom: A super-admin gets a 429 on a route they should be able to call freely.

1. Audit log: filter eventType: rate_limit_exceeded for the user.
2. The bucket key is <action>:<userId> — they'll need to wait for reset (an epoch ms in the response).
3. Don't add a bypass for super-admins blindly. Rate limits are quota, not security gates — if the workflow legitimately needs more throughput, raise the limit in the route handler (default cap stays low for non-admins).
4. If a single user is repeatedly tripping the same limit, it's usually a stuck client retry loop. Have them clear browser state.

7. Suspicious permission escalation — P1 — escalate to security#

Symptom: audit log shows eventType: permission_escalation_attempt or user_role_changed where the actor is not a super-admin.

1. Lock the actor immediately:
   • Clerk dashboard → user → Block sign-in.
2. Read the surrounding audit entries (5 minutes before and after) to map the attack window.
3. Check audit_logs for any other actions by the same userId in the last 24h. Cross- reference with Sentry for related errors.
4. File an incident: #security-incidents, severity P1. Notify legal + super-admin within 1 hour. Send a written timeline within 24h.

8. Disk-full on Vercel build — P2 (deploy blocked)#

Symptom: build fails with ENOSPC: no space left on device.

This is a local-development trap more than a Vercel issue — Vercel's build agents are clean per run. But locally:

rm -rf .next apps/workspace/.next apps/webdocs/.next labs/.next
rm -rf node_modules/.cache labs/node_modules/.cache

The labs/ sibling app's .next is the usual culprit (~700MB).

9. CI pipeline failure#

Open .github/workflows/ci.yml. The pipeline runs:

1. Install (frozen lockfile — if it fails here, someone broke pnpm-lock.yaml).
2. Typecheck root + workspace + webdocs.
3. Lint (pnpm run lint:error, currently continue-on-error: true due to pre-existing debt).
4. Build workspace + webdocs (root build skipped — webpack disk-hungry on CI).

If the workspace build fails on Clerk prerender, the dummy pk_test_* key in .github/workflows/ci.yml isn't passing Clerk's format validator. Update to a longer realistic-looking dummy.

10. Transactional email not arriving (datasheets, webinar confirmations, subscriptions) — P1#

Symptom. Visitors submit a gated form and see "Check your inbox", but nothing arrives; or, after 2026-09-03, the datasheet form shows "We couldn't email your datasheet right now" (HTTP 502 from /api/datasheets/submit). Sentry fills with feature:email warnings ("Email primary provider failed — using fallback") or feature:datasheet-download errors ("failed on every provider").

Severity. P1 — every lead-capture flow on the marketing site delivers by email. Revenue-facing.

Background. lib/email/index.ts sends through the primary (EMAIL_PROVIDER: ses / smtp / resend) and, on any failure, retries once through the backup (EMAIL_FALLBACK_PROVIDER, default Resend from resources.matters.ai). Policy: Resend is always the backup. On 2026-09-03 SES stopped delivering and there was no second path — every mail on the site went dark.

Decision tree.

1. Sentry → filter feature:email. tags.primary tells you which engine is failing; extra.error carries the provider's message (SES: MessageRejected, Throttling, AccountSuspended; SMTP: EAUTH, ECONNECTION).
   • Warnings only, no datasheet-download errors ⇒ the backup is carrying traffic. Mail is flowing. Fix the primary at leisure (step 3), but do not leave it: Resend quotas are sized as a backup, not the primary.
   • failed on every provider ⇒ both engines down. Go to step 2 immediately.
2. Check the backup is actually armed in Vercel → Project → Environment Variables (Production): RESEND_API_KEY set, EMAIL_FALLBACK_PROVIDER unset or resend. If the key is missing, add it and redeploy — that alone restores delivery. Confirm resources.matters.ai shows Verified in the Resend dashboard (Domains).
3. Diagnose the primary:
   • SES / SMTP: AWS console → SES → Account dashboard. Look for account under review / sending paused (bounce or complaint rate breach — see SES_SETUP.md → "Deliverability thresholds"), quota exhaustion, a rotated IAM key (AWS_SES_ACCESS_KEY_ID / SMTP_PASS), or the sending identity no longer verified (AWS_SES_FROM_EMAIL).
   • If SES will be down for a while, flip the primary rather than lean on failover latency: set EMAIL_PROVIDER=resend in Vercel and redeploy. Revert once SES is healthy.
4. Re-test: submit a datasheet form with a work email you control. The response is 200 with delivery: "email" on success and 502 with the visitor-facing message on total failure. The route logs provider and fellBack for the send.

Escalation. Both providers failing for more than 15 minutes, or SES showing account under review: page the on-call owner of the AWS account. Reputation suspensions need a human reply to AWS and cannot be fixed by redeploying.

Adding a new runbook#

When you debug a production incident, take 30 minutes after the fix to write the runbook here. Format:

1. Symptom — what the on-call sees first.
2. Severity — P1 / P2 / P3 with the rule of thumb.
3. Decision tree — numbered steps, with command-line invocations where applicable.
4. Escalation criteria — when to wake someone up.

Future-you will thank present-you.