Runbook

Step-by-step procedures for the common operational tasks and incidents on the Partners Portal. Each is grounded in the real code under apps/partners/.

Conventions used below

• All pnpm commands assume you are in the repo root unless a cd is shown. The portal app is @matters/partners (filterable as pnpm --filter @matters/partners …).
• The dev server runs on port 4005 (pnpm --filter @matters/partners dev).
• The Neon connection string lives in PARTNERS_NEON_DATABASE_URL. Every script and drizzle.config.ts loads env in this order (first hit wins): ~/.envs/.matters-ai/.env.partners.prod → apps/partners/.env.local → apps/partners/.env → apps/partners/.env.prod.
• lib/db/schema.ts is the single source of truth for the database; the Partners Neon DB is manually managed (no auto-migrate on deploy).

1. Add a new Super Admin / internal teammate#

Internal access is the 7-type account model (pending · rejected · prospect · partner · internal_team_member · admin · super_admin). New teammates start at pending and a Super Admin promotes them. There is also an env fail-safe and a one-off bootstrap script for the very first Super Admin.

1a. Normal path — promote an existing @matters.ai teammate#

1. They sign in. Have the teammate sign in to the portal with their @matters.ai email. On first login the resolver (resolveAccountType in lib/auth/account-type-store.ts) auto-derives their type. Because @matters.ai users are never auto-granted anything above pending (deriveDefaultType in lib/auth/account-type.ts), they land as pending with no admin access yet.

2. A Super Admin opens /admin/accounts. Only super_admin can see this page (gated by requireSuperAdminSession). Find the teammate under their org group.

3. Set their type via the dropdown:

   • internal_team_member — read-only admin.
   • admin — full admin except role assignment and access requests.
   • super_admin — everything, including this page and /admin/access-requests.

   The dropdown only offers internal types for @matters.ai emails — canAssignType rejects assigning an internal type to an external domain server-side, so a doctored request still fails.

4. It takes effect immediately. The assignment writes the partner_sessions row and the activity_log row in one transaction (setAccountType), then mirrors accountType to Clerk privateMetadata after commit (best-effort, self-healing via account_type_synced_at). The hot path trusts the Clerk mirror with a 15-minute TTL; on expiry or mismatch it reconciles against the DB (source of truth). So a freshly-promoted user is live on their next request, and within 15 minutes at the latest if the mirror write lagged.

Guard rail: the last remaining super_admin cannot be demoted in the UI. There is always at least one.

1b. Pre-assign someone who has not logged in yet#

You can assign a type by email before the user exists in Clerk. setAccountType keys the row by a pending:<lowercased-email> placeholder (pendingKey). On that user's first login, resolveAccountType adopts the placeholder row (rewrites clerk_user_id to the real id) and applies the pre-assigned type. No script needed — just assign from /admin/accounts.

1c. Env fail-safe — SUPER_ADMIN_EMAILS#

The bootstrap floor means the portal can never lock itself out of approvals. resolveAccountType returns super_admin unconditionally when:

• the email is in the SUPER_ADMIN_EMAILS allowlist, or
• the user is an admin of the Matters.AI Clerk org (MATTERS_CLERK_ORG_ID).

To add an emergency Super Admin without touching the DB, append their email to SUPER_ADMIN_EMAILS (comma-separated) in the prod Vercel env and redeploy:

# Vercel env (Production) — comma-separated, lowercased
SUPER_ADMIN_EMAILS=eshank@matters.ai,newadmin@matters.ai

This is a floor, not a replacement for /admin/accounts — it grants super_admin regardless of DB/Clerk state. Use it to recover access, then assign the DB type properly.

1d. Bootstrap the very first Super Admin (fresh Clerk app)#

On a brand-new Partners Clerk app, run the idempotent bootstrap script. It finds-or-creates the Matters.AI Clerk org, then either adds the user as org:admin (if they exist) or sends an org:admin invitation, and prints the org id.

# default: email = first of SUPER_ADMIN_EMAILS (fallback eshank@matters.ai), org = "Matters.AI"
pnpm --filter @matters/partners bootstrap:super-admin

# explicit:
pnpm --filter @matters/partners bootstrap:super-admin -- --email eshank@matters.ai --org "Matters.AI"

Requirements: CLERK_SECRET_KEY (Partners app) set, and Organizations enabled in the Partners Clerk app. The script prints the values to set in your env afterward:

MATTERS_CLERK_ORG_ID=org_xxxxxxxxxxxxxxxxxxxxxxxxx
SUPER_ADMIN_EMAILS=eshank@matters.ai

If the user did not exist, they must accept the emailed invitation, then sign in.

2. Approve / reject a partner#

External users whose email domain matches a seeded partner_accounts row auto-become partner on first login. Unknown domains land as pending and surface a self-service access request (see /partners/access-requests). Super Admins action those at /admin/access-requests (gated by requireSuperAdminSession).

2a. Approve#

1. Open /admin/access-requests → Pending Review section.
2. Click ✓ Approve on the row. The approveAccessRequest server action (lib/actions/access-request-admin.ts) runs this sequence:
   1. Provision the partner account — upserts a partner_accounts row for the request's email domain (so the login gate admits the domain on first sign-in).
   2. Reversible external step first — creates the Clerk org (named from the request's company) and sends an org:admin invite to the requester. This is done before any DB commit so that, if Clerk fails, nothing is committed (ACID revert — no partial approval).
   3. Commit — marks the request approved and writes a pre-login partner_sessions row keyed by pending:<email> with account_type = partner.
   4. Email — sends the approval email (sendAccessApprovedEmail).
3. On their first login the pending:<email> row is adopted (see §1b) and they are a live partner.

2b. Reject#

1. On the row, expand Confirm Reject, optionally enter a reason.
2. The rejectAccessRequest action marks the request rejected, writes a partner_sessions row with account_type = rejected, and sends the rejection email (sendAccessRejectedEmail).
3. Rejected users appear in a separate list on /admin/accounts with a "Move to Pending" action if you want to reconsider.

The list page caps at the 200 most-recent requests, newest first. Pending and actioned are shown in separate sections.

3. Apply a schema change#

The Partners Neon DB is manually managed. After editing lib/db/schema.ts, apply the diff to Neon by hand. Always keep schema.ts and the live DB in lockstep — an un-applied change 500s pages (see §5).

3a. Preferred — drizzle-kit push#

push diffs schema.ts against the live DB and applies exactly the missing changes (it ignores the drizzle/ migration files, so it works regardless of what the DB has applied).

cd apps/partners
pnpm db:push          # = drizzle-kit push

Notes:

• It needs a TTY — it prompts before writing. Run it in an interactive terminal, not a non-interactive CI step.
• It writes immediately on confirm — review the printed statements before saying yes.
• It is fixed for our stack (drizzle-orm 0.45 + drizzle-kit 0.31, which handle Postgres 18). On a clean DB it emits ~7 cosmetic array-default no-ops (benign ALTER … SET DEFAULT '{}' on jsonb/array columns) — those are expected and safe to confirm.
• A second run on an in-sync DB should report "No changes."

3b. Fallback — direct additive DDL via a one-off node script#

If push can't run (no TTY, or it misbehaves), apply additive DDL directly with a throwaway script that reuses the script env-load order. Mirror the dotenv block from scripts/seed-from-csv.ts:

// apps/partners/scripts/_oneoff-ddl.ts  (delete after running)
/* eslint-disable import/order */
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";

import { neon } from "@neondatabase/serverless";
import dotenv from "dotenv";

const dirname =
  typeof __dirname !== "undefined" ? __dirname : path.dirname(fileURLToPath(import.meta.url));

dotenv.config({ path: path.join(os.homedir(), ".envs", ".matters-ai", ".env.partners.prod") });
dotenv.config({ path: path.resolve(dirname, "..", ".env.local") });
dotenv.config({ path: path.resolve(dirname, "..", ".env") });
dotenv.config({ path: path.resolve(dirname, "..", ".env.prod") });

async function main() {
  const url = process.env.PARTNERS_NEON_DATABASE_URL;
  if (!url) throw new Error("PARTNERS_NEON_DATABASE_URL not set");
  const sql = neon(url);

  // Additive only — example: a new nullable column.
  await sql`ALTER TABLE partner_sessions ADD COLUMN IF NOT EXISTS new_col text;`;

  console.log("DDL applied");
}

main().catch((e) => {
  console.error(e);
  process.exit(1);
});

Run it once, then delete it:

cd apps/partners
pnpm exec tsx scripts/_oneoff-ddl.ts
rm scripts/_oneoff-ddl.ts

Keep lib/db/schema.ts updated to match by hand — the resolver and queries import from it, and a second db:push should then report "No changes."

Example — the multi-account composite key. The unique that prevents one org's tier/SF-ids bleeding into another is partner_sessions_user_org_unique on (clerk_user_id, clerk_org_id). It is already in schema.ts (replacing the old single-column unique). If a DB predates it, the direct-DDL equivalent is:

ALTER TABLE partner_sessions DROP CONSTRAINT IF EXISTS partner_sessions_clerk_user_id_unique;
CREATE UNIQUE INDEX IF NOT EXISTS partner_sessions_user_org_unique
  ON partner_sessions (clerk_user_id, clerk_org_id);

3c. Inspect after applying#

cd apps/partners
pnpm db:studio        # drizzle-kit studio — browse tables + indexes

4. Seed data#

Two idempotent seed scripts. Both require PARTNERS_NEON_DATABASE_URL and use the script env-load order. They use @neondatabase/serverless's HTTP driver (neon(...) + drizzle-orm/neon-http).

4a. Partner accounts from the Salesforce CSV#

seed:csv reads apps/partners/.sf_partners_data.csv (resolved from process.cwd(), so run it from apps/partners/) and upserts partner_accounts rows. It upserts on sf_account_id; the website is parsed down to a bare domain (extractDomain) which is what the login gate matches against for auto-partner derivation.

cd apps/partners
pnpm seed:csv         # = tsx scripts/seed-from-csv.ts

Expected CSV columns: Account Name, Account ID, Website, Partner Account Name, Partner Account ID, Partner Account: Type, Partner Account: Industry, Partner Account: Billing Country, Partner Account: Billing City, Partner Account Owner. Output: Parsed N rows then Upserted N partner accounts.

4b. Partner tiers#

seed:tiers upserts exactly three PRD tiers by slug — Registered, Silver, Gold (no "Platinum"). Margin/MDF are per-deal, not per-tier, so the ARR/MDF/deal-protection columns are intentionally left null. The /dashboard reads partner_tiers, so this must be seeded for the hub to render (see §5).

cd apps/partners
pnpm seed:tiers       # = tsx scripts/seed-tiers.ts

Output: Upserted 3 partner tiers (registered, silver, gold).

5. Diagnose a portal-wide 500 / "Error ID" page#

Symptom: Multiple portal pages render the (portal)/error.tsx boundary — a "Something went wrong" card with an Error ID (the Next.js digest). The digest is opaque: the real error message is masked by the boundary and only the digest is shown to the user.

Most likely cause: a schema change in lib/db/schema.ts was not applied to Neon — a missing table or column. Because:

• The (portal) layout fetches notifications on every page (getNotifications / getUnreadCount → notifications table).
• /dashboard queries partner_tiers.

…a single missing table takes down the whole hub, not just one route. A query against a non-existent table/column throws, the layout/page server component fails, and the boundary swallows it behind the digest.

Diagnose:

1. Get the real error, not the digest. The digest is a hash; look it up in Sentry (the boundary calls captureException(error) in a useEffect). The Sentry event has the underlying Postgres error (e.g. relation "notifications" does not exist or column "…" does not exist).

2. Compare information_schema against schema.ts. Connect to Neon (db:studio, the Neon SQL console, or psql "$PARTNERS_NEON_DATABASE_URL") and check the tables exist:

   SELECT table_name
   FROM information_schema.tables
   WHERE table_schema = 'public'
   ORDER BY table_name;
   

   Cross-check against the tables defined in lib/db/schema.ts: partner_sessions, partner_invitations, opportunities, opportunity_notes, callout_events, partner_profiles, content_documents, partner_agreements, mdf_requests, partner_access_requests, partner_accounts, partner_tiers, notifications, activity_log.

   For a suspect table, check its columns:

   SELECT column_name, data_type
   FROM information_schema.columns
   WHERE table_name = 'partner_sessions'
   ORDER BY ordinal_position;
   

3. Fix: apply the missing table/column via §3 (db:push or direct DDL), then reload. If partner_tiers is empty (not missing), run pnpm seed:tiers (§4b).

Design note: decorative reads should degrade, not crash. A failed notifications fetch should render an empty bell, not 500 the whole shell. If you find a decorative query taking down the hub, that read should be wrapped to fall back to empty on error — the only hard dependency a page should crash on is its own primary data.

6. Enable multi-account login#

Behind the master flag NEXT_PUBLIC_MULTI_ACCOUNT_ENABLED. Default is OFF, and with it off there is exactly one partner_sessions row per user and the RBAC is correct — this is not a current bug. See /partners/multi-account and /partners/rbac.

Enabling it lets one person hold multiple Clerk sessions (e.g. a partner in org A and an internal @matters.ai account). Do these in order:

1. Reconcile the account-type resolver to be org-scoped FIRST. With multi-session on, a user has multiple partner_sessions rows (one per org), but resolveAccountType (lib/auth/account-type-store.ts) currently resolves by clerk_user_id. The recommended model: per-(user, org) for external tiers, global for internal — a person can be partner in org A and prospect in org B, while internal tiers (internal_team_member / admin / super_admin) describe the person, not an org, and stay global via the SUPER_ADMIN_EMAILS / Matters-org floor. The composite unique partner_sessions_user_org_unique on (clerk_user_id, clerk_org_id) is already in schema.ts to make this safe (apply it via §3 if your DB predates it). Ship this resolver change before flipping the flag, or org data can cross-contaminate.

2. Set the flag in apps/partners/.env (local) and prod Vercel env:

   NEXT_PUBLIC_MULTI_ACCOUNT_ENABLED=true
   

3. Enable Clerk multi-session per instance. In the Clerk dashboard, pick the Partners application → the correct instance (Production = the one whose publishable key matches NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY) → turn on multi-session. This is a per-instance setting; do it for every instance you run the flag on.

4. Verify the isolation invariant (the whole point):

   • Sign in as the partner → land on /dashboard, see only that partner's deals/MDF/notifications.
   • Navbar account chip → Add account → sign in as the internal @matters.ai account → lands on /admin (must never flash the partner dashboard).
   • Switch back to the partner → /admin is gone; only partner data shows. Switch to internal → partner data is gone.

7. Rotate / set third-party keys#

The portal depends on Clerk, Neon, Resend, Upstash (Redis + ratelimit), AWS S3 / R2, Sentry, and Svix. Exhaustive per-service env reference and rotation steps live on /partners/operations.

Quick pointers:

• PARTNERS_NEON_DATABASE_URL — the Neon connection string. Rotating it requires updating the prod Vercel env and the local env files (~/.envs/.matters-ai/.env.partners.prod, apps/partners/.env*); scripts read it directly.
• CLERK_SECRET_KEY / NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY — Partners Clerk app keys. The bootstrap and approval flows call the Clerk API with CLERK_SECRET_KEY; the publishable key must match the instance whose multi-session you enabled (§6).
• SUPER_ADMIN_EMAILS / MATTERS_CLERK_ORG_ID — the bootstrap floor (§1c). Changing these changes who can never be locked out.
• Resend / Upstash / S3 / Sentry / Svix — see /partners/operations for each key's purpose and rotation procedure.

After rotating any prod key, redeploy so the new value is picked up (Vercel injects env at build/runtime per the platform's rules).