Runbooks

When something breaks. Pages mirror the failure modes in Observability, but with concrete commands and decision trees.

Severity legend. P1 = customer-visible outage. P2 = degraded experience. P3 = internal tooling broken but public site healthy.

1. Public site returns 500 across all pages — P1#

1. Check Vercel → matters-web → Deployments. Roll back to the last green deployment via "Promote to Production".
2. While the rollback propagates, check Sentry for the first error timestamp. If it's a Mongo connection failure, check Atlas dashboard for cluster health + IP allowlist.
3. If it's a WordPress 5xx fanning out, check WordPress host. Public-site fetches have a short cache; a full WordPress outage will appear as 502s on /blogs/* first.
4. Communicate via the engineering Slack channel within 5 minutes of detection.

2. Workspace login loop (Clerk session not sticking) — P2#

Symptom: User authenticates → bounced back to /auth/login.

1. Check Clerk dashboard → Production → Session → Cookie configuration. The cookie Domain must be exactly .matters.ai (note the leading dot).
2. Check the browser's devtools cookie tab: is the Clerk session cookie set with that domain?
3. If yes and still bouncing, check apps/workspace/proxy.ts is matching the path. The isPublicRoute matcher should be ["/auth(.*)", "/api/webhooks/clerk(.*)", "/api/health"].
4. Last resort: redeploy matters-workspace. The middleware bundle sometimes ships stale.

3. WordPress content stale after a publish — P3#

1. Check Vercel → matters-web → Logs → filter for /api/revalidate. Did the webhook arrive?
   • No → check the WordPress plugin status. Trigger a manual revalidation:

     curl -X POST https://matters.ai/api/revalidate \
       -H "Content-Type: application/json" \
       -H "x-webhook-secret: $WORDPRESS_WEBHOOK_SECRET" \
       -d '{"contentType":"post","slug":"<the-stale-slug>"}'
     

   • Yes but 401 → secret mismatch. Compare WORDPRESS_WEBHOOK_SECRET in Vercel against the WordPress plugin settings.
   • Yes and 200 → revalidation fired but content still stale. CDN cache. Open https://matters.ai/<path>?_=$(date +%s) to bust the URL cache; if still stale, file Vercel support.

4. Bulk-invite stuck / partial — P2#

Symptom: /api/workspace/webinars/email/bulk-invite reports failed > 0.

1. Get the audit log entry:

   getAuditLogs({ eventType: "webinars.bulk-invite", limit: 1 })
   

   The metadata field has { test_mode, sent, failed, total }.
2. Cross-reference Sentry for [bulk-invite] errors in the same time window. Most likely: AWS SES bounce / sandbox-mode limit.
3. Check SES console → Sending statistics. If the bounce rate spiked, pause the campaign and investigate.
4. To resume from where it stopped: webinars currently retry by re-running the bulk-invite — SES dedupes on Message-ID, so duplicates won't double-send.

5. Salesforce sync failures — P3#

1. Audit log: getAuditLogs({ eventType: "webinars.sync-salesforce", limit: 1 }).
2. Look at metadata.failed count and individual failures[] in the response.
3. Common causes:
   • 403 from Salesforce → CRM credentials rotated. Update SALESFORCE_* env vars in Vercel.
   • Field validation failure → dial-code field ID changed. Update SALESFORCE_DIAL_CODE_FIELD_ID.
   • Web-to-Lead rate limit → reduce concurrency in the next sync call's body (concurrency: 1 or 2).

6. Rate-limit blocking a legitimate user — P3#

Symptom: A super-admin gets a 429 on a route they should be able to call freely.

1. Audit log: filter eventType: rate_limit_exceeded for the user.
2. The bucket key is <action>:<userId> — they'll need to wait for reset (an epoch ms in the response).
3. Don't add a bypass for super-admins blindly. Rate limits are quota, not security gates — if the workflow legitimately needs more throughput, raise the limit in the route handler (default cap stays low for non-admins).
4. If a single user is repeatedly tripping the same limit, it's usually a stuck client retry loop. Have them clear browser state.

7. Suspicious permission escalation — P1 — escalate to security#

Symptom: audit log shows eventType: permission_escalation_attempt or user_role_changed where the actor is not a super-admin.

1. Lock the actor immediately:
   • Clerk dashboard → user → Block sign-in.
2. Read the surrounding audit entries (5 minutes before and after) to map the attack window.
3. Check audit_logs for any other actions by the same userId in the last 24h. Cross- reference with Sentry for related errors.
4. File an incident: #security-incidents, severity P1. Notify legal + super-admin within 1 hour. Send a written timeline within 24h.

8. Disk-full on Vercel build — P2 (deploy blocked)#

Symptom: build fails with ENOSPC: no space left on device.

This is a local-development trap more than a Vercel issue — Vercel's build agents are clean per run. But locally:

rm -rf .next apps/workspace/.next apps/webdocs/.next labs/.next
rm -rf node_modules/.cache labs/node_modules/.cache

The labs/ sibling app's .next is the usual culprit (~700MB).

9. CI pipeline failure#

Open .github/workflows/ci.yml. The pipeline runs:

1. Install (frozen lockfile — if it fails here, someone broke pnpm-lock.yaml).
2. Typecheck root + workspace + webdocs.
3. Lint (pnpm run lint:error, currently continue-on-error: true due to pre-existing debt).
4. Build workspace + webdocs (root build skipped — webpack disk-hungry on CI).

If the workspace build fails on Clerk prerender, the dummy pk_test_* key in .github/workflows/ci.yml isn't passing Clerk's format validator. Update to a longer realistic-looking dummy.

Adding a new runbook#

When you debug a production incident, take 30 minutes after the fix to write the runbook here. Format:

1. Symptom — what the on-call sees first.
2. Severity — P1 / P2 / P3 with the rule of thumb.
3. Decision tree — numbered steps, with command-line invocations where applicable.
4. Escalation criteria — when to wake someone up.

Future-you will thank present-you.