Runbook

Symptom-driven. Each section is "you see X → check Y → fix Z."

"Visitors can't get past the email gate"#

Symptoms: OTP emails not arriving. Visitors stuck on /<slug>/otp.

Check:

1. GET https://api.resend.com/emails/<recent_id> — did Resend accept the send?
2. audit_logs for the visit — nda.sign won't be there, but the gate progression doesn't write per-step audit rows; check Sentry for handler errors.
3. Redis: GET memo:otp:<slug>:<lowercased-email> — is the code present?

Fix:

• If Resend rejects: re-verify the sending domain memo.matters.ai in Resend's dashboard.
• If the OTP key is missing: confirm Upstash Redis credentials are correct in Vercel env, and that the email gate handler isn't 429'd (check memo:rl:visitor-email:*).
• If both are fine but the visitor still doesn't receive: spam filter; have them check, or fall back to /<slug>/password if the link also has a password.

"Owner can't see Activity tab data"#

Symptoms: Activity table empty or missing recent visits.

Check:

1. SELECT count(*) FROM memo.visits WHERE link_id IN (SELECT id FROM memo.links WHERE project_id = '<projectId>') — is the data in Postgres?
2. If yes, the page query is fine — look for an island-rendering error in DevTools.
3. If no, the visit-row insert from view.astro is failing. Check Sentry for view.astro errors.

Fix:

• The visit-row insert runs ON CONFLICT DO NOTHING keyed on id — so if the same JWT sub is reused, no row is inserted. This is correct behavior.
• If Postgres connection is failing: confirm NEON_DATABASE_URL is set and Neon's compute is awake.

"LogRocket Replay column always blank"#

Check:

1. LOGROCKET_APP_ID in Vercel env?
2. CSP allows https://cdn.lr-ingest.io in script-src? (Yes, default policy includes it.)
3. Did the browser actually call getSessionURL and POST to /api/recording/logrocket?

Fix:

• If LogRocket dashboard shows the session but visits.logrocket_session_url is null, the round-trip POST failed. Check Sentry on /api/recording/logrocket; common cause is a CSP block on cdn.lr-ingest.io (default CSP allows it; an ALLOWED_HOSTS override that omits LogRocket would block it).
• The endpoint takes { visit, slug, url } in the JSON body and re-verifies the visitor JWT against the slug; if either is wrong, the update is dropped silently.

"Signed-NDA PDF never appears"#

Symptoms: signatures.signed_pdf_r2_key is null long after a successful sign.

Check:

1. QSTASH_TOKEN set?
2. QStash dashboard — was the job published? Did it fail?
3. If the job failed: MEMO_CHROMIUM_PACK_URL correct? Without it, @sparticuz/chromium-min falls back to a local binary which won't exist in Vercel's serverless environment.

Fix:

• Re-trigger manually: curl -X POST -H "Upstash-Signature: …" https://memo.matters.ai/api/cron/render-signed-pdf -d '{"signatureId":"…"}'.
• Or use QStash's "Retry" button on the failed message.

"Cron jobs aren't running"#

Check:

1. Vercel dashboard → Cron Jobs — are they enabled and showing recent executions?
2. CRON_SECRET set in Vercel env?
3. The cron handler returns 401 if the Authorization header doesn't match. Vercel sets the header automatically; if it's mis-cased or the value is wrong, every cron fires but does nothing.

Fix:

• Re-deploy with a fresh CRON_SECRET if you suspect it was rotated incorrectly.
• For one-off triggers: curl -H "Authorization: Bearer $CRON_SECRET" https://memo.matters.ai/api/cron/expire-links.

"Activity tab shows visits but no scroll / click / dwell data"#

Symptoms: memo.visits rows are landing fine; db.events.find({ link_id: ... }) in Mongo returns nothing for recent visits; Activity tab dwell-time/scroll-depth numbers are blank.

Background: public/beacon.js posts events to /api/beacon with the slug embedded in the body. The slug used to be parsed from location.pathname.split("/")[2] (the position the slug occupied under the old /v/<slug>/view URL scheme). When visitor pages moved to the root and later /view was collapsed into /<slug>/ (canonical trailing slash), URL-position parsing breaks silently — /api/beacon 204s because loadLink("view") or loadLink("") finds nothing.

Check:

1. View source on a /<slug>/ page. Both meta tags must be present:

   <meta name="memo-visit" content="<visit-uuid>">
   <meta name="memo-slug"  content="<slug>">
   

   If memo-slug is missing, injectVisitorChrome in src/lib/visitor/watermark.ts regressed.
2. In DevTools, hit /api/beacon with a manual sendBeacon and confirm it does NOT 204:

   navigator.sendBeacon('/api/beacon', JSON.stringify({
     slug: 'YOUR-SLUG', visit: 'YOUR-VISIT-UUID',
     events: [{ type: 'page_view', payload: {}, t: Date.now() }],
   }));
   

3. Confirm public/beacon.js reads slug from the meta tag, not from location.pathname:

   var slugMeta = document.querySelector('meta[name="memo-slug"]');
   var slug = slugMeta ? slugMeta.getAttribute("content") : "";
   

Fix: restore the meta-tag handoff in both src/lib/visitor/watermark.ts (server injection) and public/beacon.js (client read). The URL-position parser is a footgun — never bring it back. Same lesson applies to /api/recording/logrocket: it takes slug in the body rather than parsing Referer.

"Section heatmap is empty / says no section data yet"#

Symptoms: Performance tab → "Section engagement" shows the placeholder text "No section data yet. Tag sections in your HTML with <section data-memo-section="hero">" even after visitors have viewed the deck.

Check:

1. Is the published HTML actually tagged? View source on a /<slug>/ page (signed in as the visitor or yourself) and grep for data-memo-section. If zero matches, the deck's HTML lacks the attribute and the IntersectionObserver has nothing to instrument.
2. Is the beacon firing? Open DevTools → Network → filter beacon. You should see POST /api/beacon requests every ~5s as the visitor scrolls. If no requests, the beacon isn't loaded — see the "Activity tab shows visits but no scroll / click / dwell data" entry above.
3. Are section_exit events landing in Mongo?

   db.events.find({ project_id: "<projectId>", type: "section_exit" }).limit(5)
   

   If empty, no section ever crossed the IntersectionObserver's 50% visibility threshold during a recent visit. Common causes:
   • Sections are too short (less than half the viewport height) so 50% intersection never triggers.
   • Sections are inside a position: fixed ancestor — IntersectionObserver against the viewport gets confused.
   • The element has display: none or visibility: hidden at observe time.

Fix:

• Wrap content in <section data-memo-section="…"> with stable kebab-case ids: hero, problem, financials, team, cta. Avoid slide-1 / slide-2 (rewrites change indices and break historical aggregation).
• For very-short sections, raise their min-height or merge into a parent section.
• The aggregation endpoint caches Cache-Control: private, max-age=30 — refresh after 30s or hard-reload to bypass.

"Behavioural signals not appearing on activity rows"#

Symptoms: Activity tab shows visits with engagement scores but the Signals column is empty (no skim / deep_read / return / share_detected chips).

Background: Two paths produce signals:

• Per-visit (skim / deep_read) — emitted inline by /api/beacon when the visit's page_leave event arrives. If the visit never page_leaves (visitor closed the tab via task manager, network died mid-flight), no per-visit signal fires.
• Cross-visit (return / share_detected) — emitted by the derive-cross-visit-signals cron every 10 minutes.

Check for per-visit signals:

1. Did the visit finalise? SELECT id, ended_at, duration_ms, completion_pct FROM memo.visits WHERE id = '<uuid>'. If ended_at IS NULL, no page_leave arrived — no skim / deep_read will fire. Expected for tabs that visitors actually closed cleanly; problem for tabs that crashed.
2. Did it meet the thresholds? Check src/lib/analytics/derive-signals.ts:
   • skim: completion ≥ 80 AND duration < 30s AND sections ≤ 2. ALL three required.
   • deep_read: sections ≥ 5 AND duration ≥ 3min AND avg section dwell ≥ 45s.
3. Are the signal rows in Postgres?

   SELECT * FROM memo.visit_signals WHERE visit_id = '<uuid>';
   

Check for cross-visit signals:

1. Has the cron run recently?

   SELECT * FROM memo.audit_logs
   WHERE action = 'visit_signals.derive'
   ORDER BY ts DESC LIMIT 5;
   

   You should see entries every 10 minutes with meta containing returnCount and shareCount. If 0/0 consistently, no thresholds are being met.
2. Are visits within the 72h window? The cron only looks at the trailing 72 hours. Older visits are ignored.
3. Are visitor_persistent_id values populated? SELECT COUNT(*) FROM memo.visits WHERE visitor_persistent_id IS NULL — if non-zero for recent visits, the memo_vp cookie path isn't running (e.g., migration 0004 wasn't applied).

Fix:

• Verify migration 0004 is applied: \d memo.visits | grep visitor_persistent_id.
• Verify migration 0005 is applied: \d memo.visit_signals should show the table.
• Confirm the cron is in vercel.json and the latest deploy includes the new cron path.
• Manually trigger it: curl -X GET -H "Authorization: Bearer $CRON_SECRET" https://memo.matters.ai/api/cron/derive-cross-visit-signals.

"Live activity feed isn't updating"#

Symptoms: Activity tab open, a new visitor arrives, but no row slides in at the top with the yellow-fade animation. Page refresh shows the new row.

Background: The Activity tab polls /api/owner/projects/<id>/activity/since?ts=<iso> every 8 seconds and prepends new rows. Polling only runs in "Recent" sort mode (when ?sort=score is on the URL, the page would have to re-sort on every poll which would be visually thrashy).

Check:

1. Sort mode: is the URL …/activity (Recent) or …/activity?sort=score (Engagement)? In Engagement mode, polling is intentionally disabled.
2. DevTools → Network: filter since. You should see one request every 8s. If you don't, the inline <script> in activity.astro didn't run — check the page console for errors.
3. Server time drift: the client uses serverTime from the response as the next ?ts=. If the server clock is way off (it shouldn't be on Vercel), the gt(started_at, ts) query returns empty even for fresh visits.

Fix:

• Switch to Recent sort if you accidentally landed on Engagement.
• If polling fires but visits don't appear, query the endpoint manually:

  curl -b "<your-cookie-jar>" "https://memo.matters.ai/api/owner/projects/<id>/activity/since?ts=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)"
  

  If this returns visits, the client polling is broken (browser cache, ad blocker stripping fetch, etc.). If it returns empty, the visits genuinely aren't landing — see "Activity tab shows visits but no scroll / click / dwell data".

"The Changes panel says 'no changes' but I just edited a file"#

Symptoms: Edited and saved a file in the editor; the Changes panel (activity bar's third icon) shows "No changes vs v3" instead of listing the file.

Background: The Changes panel queries /api/owner/projects/<id>/changes, which compares staged files (everything in R2 under staging/<projectId>/) against the current published version's memo.files rows. The comparison is by sha256 of file bytes.

Check:

1. Did the save round-trip succeed? DevTools → Network: look for the PUT /api/owner/projects/<id>/files/<path> after your save. Status should be 204 or 200.
2. Is the file in R2 staging? The publish endpoint reads from staging/<projectId>/ — if the save somehow wrote elsewhere, the diff sees nothing.
3. Refresh key: the Changes panel only re-fetches when changesRefreshKey bumps. The shell bumps it on save and on publish. If you're looking at a stale render, force a panel re-mount (collapse activity bar then open Changes again).
4. 413 too_many_*_files: the endpoint caps at 1000 files per side. Decks shouldn't approach this; if you're seeing 413, you're hitting the cap.

"Visitor sees scrapers blocked but I want to allow my own monitoring agent"#

Background: scraperGate blocks any UA matching the blocklist regex on visitor pages (/<slug>/* — anything that isn't an app path like /u/, /api/, /oauth/, /.well-known/, etc.).

Fix:

• Set your monitoring agent's UA to include a non-blocked substring. The regex is in src/middleware.ts.
• Or, if monitoring needs to live at the owner surface, point it at /health (the visitor surface is intentionally hostile).

"Links won't expire on time"#

Check:

1. Cron expire-links running every 15 min? Vercel dashboard.
2. audit_logs for action='link.expire' — recent entries?
3. links where expires_at < now() AND active = true — should be empty after the cron runs.

Fix:

• If the cron is running but no links expire, check that expires_at is stored as timestamptz (the cron compares against now()).
• If you need to force-expire: UPDATE memo.links SET active = false WHERE expires_at < now() - interval '1 minute';

"JWT key rotation drift between regions"#

Symptoms: Some visitors get logged out unexpectedly while others are fine.

Background: The rotate cron runs once per day at 03:17 UTC; Upstash propagates within seconds globally. The previous key with 48 h TTL covers any rare propagation delay.

Fix:

• Confirm Upstash Redis is healthy and the daily cron actually executed (Vercel dashboard).
• If the cron is broken and the MEMO_JWT_KEY_FALLBACK is your last line of defense, visitors with cookies signed by the old :current key will silently fail. Re-running the cron manually re-establishes the chain.

"I need to invalidate every visitor session right now"#

upstash-cli redis del memo:jwt:key:current memo:jwt:key:previous
curl -H "Authorization: Bearer $CRON_SECRET" https://memo.matters.ai/api/cron/rotate-jwt-key

This wipes both keys and rotates a fresh one. Every visitor cookie is now invalid; they re-enter the gate flow on their next request.

"I deleted a project but the published version files are still in R2"#

R2 retains files keyed by <userId>/<projectId>/<versionId>/. Postgres-side cascading deletes drop the versions + files rows, but R2 is not auto-pruned.

Fix:

• Manual cleanup script (not in repo yet):

  aws s3 rm s3://memo/<userId>/<projectId>/ --recursive \
    --endpoint-url https://<account>.r2.cloudflarestorage.com
  

• Roadmap: a cleanup-orphans QStash job that finds R2 prefixes without matching versions rows and deletes them after a 30-day grace.

"I need to audit who saw an investor link"#

SELECT v.started_at, v.visitor_email, v.ip, v.duration_ms, v.logrocket_session_url
FROM memo.visits v
JOIN memo.links  l ON l.id = v.link_id
WHERE l.slug = '<slug>'
ORDER BY v.started_at DESC;

For per-event detail:

db.events.find({ link_id: "<uuid>" }).sort({ ts: 1 })

For signed NDAs:

SELECT s.signed_at, s.visitor_email, s.ip, s.agreement_version_hash, s.signed_pdf_r2_key
FROM memo.signatures s
JOIN memo.links l ON l.id = s.link_id
WHERE l.slug = '<slug>';

"How do I run the same flow locally?"#

See Operations § Local development. Cron endpoints can be triggered manually with the CRON_SECRET Bearer header.

To trigger the visitor flow end-to-end locally, set ALLOWED_HOSTS=memo.matters.ai,localhost,127.0.0.1 (loopback is already accepted automatically) and use a real Resend send to a real inbox, or temporarily stub sendOtpEmail to log the code to stdout.

Database migrations#

schema.ts is the single source of truth. The drizzle/ chain is a single clean baseline (0000_init.sql + meta/0000_snapshot.json) that exactly reflects it — drizzle-kit generate reports "No schema changes." The pre-Phase-1 → current incremental ALTERs are archived in apps/memo/drizzle-legacy/ (with a README) for upgrading an existing database by hand.

Apply to a live DB — recommended (push): from apps/memo/

pnpm exec dotenv -e <path>/.env.memo.prod -- pnpm exec drizzle-kit push

push diffs schema.ts against the live DB and applies exactly the missing changes — it ignores the migration files, so it works regardless of what the DB has applied. ⚠️ It writes immediately — review the printed statements before confirming. A second run should report "No changes."

Fresh database: drizzle/0000_init.sql (or drizzle-kit migrate/push) creates the entire current schema. Never apply 0000_init.sql to an existing DB — it's a full CREATE-everything baseline and will conflict. Use push (or the drizzle-legacy/ ALTERs in order) instead.

Required env beyond the base set: