Architecture

Request lifecycle#

Every request to memo.matters.ai passes through three middleware stages, then either an owner page (Clerk-gated SSR) or a visitor route (JWT-gated).

┌─────────────────────────────────────────────────────────────┐
│  1. hostGate                                                │
│     • Compare Host header against ALLOWED_HOSTS             │
│     • Allow loopback in dev                                 │
│     • Set HSTS, X-Content-Type-Options, Referrer-Policy,    │
│       Permissions-Policy on every response                  │
│     • Apply X-Frame-Options: SAMEORIGIN for visitor pages   │
│       (/<slug>/*), DENY everywhere else                     │
│     • Apply visitor-tuned CSP for visitor pages (allows     │
│       LogRocket + Clarity origins, denies everything else)  │
├─────────────────────────────────────────────────────────────┤
│  2. scraperGate (only on visitor pages, /<slug>/*)          │
│     • Reject empty UA                                       │
│     • Block known scraper UAs (curl, wget, httpie, scrapy,  │
│       bot, spider, crawler, headless chrome, phantomjs, …)  │
├─────────────────────────────────────────────────────────────┤
│  3. clerkMiddleware (if Clerk keys are set)                 │
│     • Resolves Clerk session on every request               │
│     • Owner pages call requireOwner() which joins Clerk →   │
│       memo.users on clerk_user_id                           │
└─────────────────────────────────────────────────────────────┘

The middleware chain is composed in src/middleware.ts. Both hostGate and scraperGate delegate the "is this a visitor or app surface?" question to a single source of truth — src/lib/routing.ts — which exports isVisitorPath(pathname: string): boolean. The function returns false for /, /u/*, /api/*, /oauth/*, /.well-known/*, /_*, and the small named pages (/sign-in, /sign-up, /health, /404, /500, /favicon.*, /robots.txt, /sitemap.xml) and true for everything else (i.e. visitor slugs at the root). Extracting this to its own module lets the vitest suite cover middleware-shape parity with SLUG_RESERVED without booting the Astro/Clerk runtime.

Owner path (/u/<username>/*)#

1. requireOwner(context) reads Clerk's session via context.locals.auth(), then looks up the matching row in memo.users keyed by clerk_user_id.
2. The page checks session.user.username == params.username; mismatches return 404.
3. Owner pages query Postgres directly through Drizzle. Heavy or interactive surfaces (Monaco, drawers, charts, Tiptap) are React islands with client:only="react".
4. Mutations go through /api/owner/* routes, each of which calls requireOwner and then validates input via zod.

Audit logging is in scope for every state-changing owner action: project create + publish, link create/update/delete, file write/delete, project upload, agreement create. See Security · Audit Log.

Visitor path (/<slug>/*)#

The visitor flow is a small state machine. loadLink(slug) runs on every visitor request and returns one of { ok, not_found, inactive, expired }. From there, nextStep(link, claims) picks the next required gate:

email   if link.requireEmail && !claims.email
otp     if link.verifyEmail   && !claims.email_verified
password if link.passwordHash && !claims.password_passed
nda     if link.requireAgreementId && !claims.nda_signed
view    otherwise

The visitor JWT is HS256-signed with a daily-rotated key (current+previous kept in Upstash) and dropped as an HttpOnly+SameSite=Lax+Secure cookie named memo_v_<slug>. Claims accumulate as the visitor passes gates; the cookie is re-signed each time.

view.astro (now [slug]/index.astro since /view was collapsed into the canonical trailing-slash root) runs the gate check one more time, idempotently inserts a visits row keyed on claims.sub, mints / refreshes the persistent-visitor cookie, captures country/city from Vercel geo headers, streams the HTML entry file from R2, and injects:

• An SVG diagonal-tile watermark (visitor email + ISO date), mix-blend-mode: difference for adaptive contrast.
• Two meta tags consumed by the beacon: <meta name="memo-visit" content="<visitId>"> and <meta name="memo-slug" content="<slug>">.
• The vanilla telemetry beacon (<script src="/beacon.js" async>).
• LogRocket + Clarity + rrweb snippets (only if env vars / per-link toggles are set).
• Optional print/contextmenu disable when allow_downloads: false.

beacon.js reads the slug from the meta tag (NOT from location.pathname). This is a load-bearing design choice: visitor URLs are at /<slug>/ (canonical trailing-slash root, formerly /<slug>/view), so any URL-position-based parser is one off-by-one away from silently dropping every event. The meta-tag handoff makes the beacon URL-layout-independent. The same pattern is used by the recording snippets — slug is passed via the request body, not derived from Referer.

Per-asset requests go through /<slug>/raw/<path> (and the catch-all /<slug>/<...path>), which re-validate the gate before streaming from R2. There is no asset CDN cache — visitor content is private by design.

Visitor identity — two cookies#

Memo issues two cookies during the visitor flow, with distinct lifecycles:

The split lets the cross-visit signal cron (derive-cross-visit-signals) say "this is the 3rd visit by the same browser" without depending on email equality (which fails when two people share an inbox). See Observability → Visitor identity model for the full state-transition table.

Observability flow (read → derive → surface)#

Beacon events land in Mongo time-series (10 event types — see Data Model → MongoDB). The handler ALSO finalises the Postgres visits row at page_leave time, computing duration_ms, completion_pct, and engagement_score in a single UPDATE. Per-visit signals (skim, deep_read) are upserted into memo.visit_signals from the same handler.

Cross-visit signals (return, share_detected) need context from multiple visit rows. A Vercel cron at */10 * * * * (/api/cron/derive-cross-visit-signals) sweeps the last 72h of visits, classifies them through the pure helpers in src/lib/analytics/derive-signals.ts, and upserts into memo.visit_signals with ON CONFLICT (visit_id, kind) DO UPDATE.

Owner surfaces (Activity / Performance / People timeline) read from Postgres + Mongo. The Activity tab polls /api/owner/projects/<id>/activity/since?ts=… every 8s for a live feed; the Performance tab's "Section engagement" island calls /api/owner/projects/<id>/analytics/sections and renders a CSS-only heat bar. The people page at /u/<u>/people/<emailHash> joins everything by email across all the owner's projects.

The whole observability stack is documented in detail at Observability.

Module layout#

apps/memo/
├── astro.config.mjs                # adapter + integrations (react, clerk?, sentry?)
├── drizzle.config.ts               # schema=memo, dialect=postgres
├── drizzle/0000_*.sql              # generated migration (citext + pgcrypto prepended)
├── public/beacon.js                # ~3 KB vanilla telemetry script
├── sentry.client.config.ts
├── sentry.server.config.ts
├── vercel.json                     # native crons
└── src/
    ├── middleware.ts               # hostGate + scraperGate + (optional) Clerk
    ├── layouts/BaseLayout.astro    # noindex meta, design-system imports
    ├── styles/                     # globals.css + memo-design-system.css
    ├── lib/
    │   ├── auth/{owner,password,visitor-jwt,bearer,pat}.ts
    │   ├── db/{postgres,mongo,schema}.ts
    │   ├── storage/{r2,staging}.ts
    │   ├── security/{audit,ratelimit,otp,qstash,qstash-publish,cron-auth}.ts
    │   ├── visitor/{gate,watermark}.ts
    │   ├── recording/{provider,rrweb-storage}.ts
    │   ├── publish/compile.ts
    │   ├── pdf/generate.ts
    │   ├── validation/{link,agreement,profile}.ts
    │   ├── webhooks/store.ts            # firing + delivery + signing
    │   ├── oauth/server.ts
    │   ├── mcp/{server,tools,tools-v3,tools-v3-registry}.ts
    │   ├── ui/dialogs.ts                # window.memo.{confirm,prompt,alert,toast}
    │   ├── email/resend.ts
    │   ├── urls.ts                      # central URL builders (ownerHref / visitorHref / …)
    │   ├── routing.ts                   # isVisitorPath — single source of truth used by middleware + tests
    │   ├── redis.ts
    │   ├── path-safety.ts
    │   └── mime.ts
    ├── components/
    │   ├── editor/EditorShell.tsx
    │   ├── agreements/AgreementEditor.tsx
    │   └── analytics/VisitorMap.tsx
    └── pages/
        ├── index.astro
        ├── health.astro
        ├── sign-in.astro / sign-up.astro
        ├── u/[username]/…                 # owner UI shell
        ├── [slug]/…                       # visitor link surface
        ├── oauth/…                        # OAuth 2.0 endpoints
        ├── .well-known/…                  # OAuth discovery
        └── api/{owner,v,cron,recording,beacon,mcp}/…

Data-room subsystems (Phase 1)#

Layered on the link core, sharing its primitives rather than duplicating them:

• Spaces (docs) — multi-project data rooms at /s/<spaceSlug>/. The spaces table mirrors links' access columns so the gate factories (gate-handler-factories.ts) and chrome serve both. Content streams from a native viewer (/s/<slug>/p/<id>/) gated on space claims — never via link redirect. Views record scope-aware visits (space_id/project_id, nullable link_id) so the same beacon + engagement engine applies; session replay reuses the same rrweb pipeline (visit_recordings.space_id).
• Q&A (docs) — a vanilla-JS widget injected into the gated view posts to /api/<slug>/qa; owners answer from a React island. qa_items is visit-scoped.
• Teams & Diligence (docs) — teams.member_emails backs the team-visibility link gate (checkTeamAccess); diligence_items/diligence_comments track per-space requests.
• Integrations (docs) — cloud-storage OAuth connect (Drive/Dropbox/Box) with PKCE + server-trusted state + AES-256-GCM token encryption (lib/crypto/secret-box.ts, shared with webhooks).

Owner-facing dashboards live under /u/<username>/spaces; every owner API is withOwner + per-row ownership re-checked.

Collaboration & realtime#

Two additive layers turn a single-owner project into a co-owned, live-edited one. Neither changes the URL shape — each collaborator still opens a project under their own namespace (/u/<you>/projects/<id>), and assertOwner is unchanged. Full feature docs at Collaboration.

• Shared ownership. Two new tables — project_members + project_invites — let multiple users co-own a project with full authority (no roles below owner). The old single-owner check is replaced across ~30 routes/pages by requireProjectAccess(projectId, userId) (owner OR member). An accessibleProjectIds helper powers the dashboard's "shared with me", and a Collaborators tab lives at /u/[username]/projects/[projectId]/collaborators.

• Live co-editing. A transport-agnostic Yjs (CRDT) layer in src/lib/collab/ exposes a CollabSession interface and a createCollabSession factory that lazy-imports one of two adapters, selected at runtime by env MEMO_REALTIME_PROVIDER (off | liveblocks | durable):

  • Liveblocks — @liveblocks/yjs getYjsProviderForRoom.
  • Cloudflare Durable Objects — y-partyserver YProvider against a standalone Worker in apps/memo/workers/realtime/ (a YServer DO using WebSocket Hibernation, free Workers plan, no new R2).

  Auth for both flows is gated by requireProjectAccess via /api/owner/realtime/liveblocks-auth and /api/owner/realtime/durable-token. Persistence is unchanged — Yjs is the live concurrency authority, the existing publish/save remains the durable snapshot. The editor binding (useCollab + MonacoBinding) is documented in Editor → Live co-editing. The whole layer is gated off by default, mirroring the existing RecordingProvider pattern, so it is exact single-user behaviour with zero regression until switched on.

Why Astro instead of Next.js#

The Matters monorepo standardizes on Next.js for everything else (workspace, weblabs, the public website). Memo is the exception:

• Visitor pages must ship near-zero JS. Astro's island model gives us a server-rendered shell with React only where it earns its weight (Monaco, Tiptap, the world map). Next's app router still streams a React runtime to every page.
• Watermark injection is a string transform on a streamed Response. Astro's set:html + raw Response ergonomics make this trivial; Next's app router would push us toward next/server middleware rewrites for what is fundamentally an HTML transformation.
• The owner area is read-heavy and mostly server-rendered. Per-island data fetching wins over a full RSC tree for the small admin surface area memo has.

Why Drizzle + Neon#

• Edge-friendly: Neon's HTTP driver + Drizzle's neon-serverless adapter (which we use because we need real transactions during publish) both work in any Vercel runtime.
• One source of truth: schema lives in src/lib/db/schema.ts, the migration is generated from it, types are inferred from it.
• No ORM dance: Drizzle gives us typed SQL without query-builder ceremony.