Content Engagement Telemetry

1. Why it exists#

Before this system, a page's read depth and engaged time were computed in the browser and kept in localStorage only — they never left the visitor's machine, so nobody on the team could ever see them. Shares were tracked nowhere at all: a "share this post" click had no server-side record, so there was no way to know which posts got shared, or whether a shared link actually brought anyone back. This spine gives every page a first-party, server-recorded engagement stream — views, read depth, active time, shares, downloads, CTA clicks — queryable from the workspace, independent of GA4.

It sits beside the existing Lead Attribution system rather than replacing it. JourneyRecorder still owns the localStorage journey attached to lead forms at submit; TelemetryRecorder (components/analytics/TelemetryRecorder.tsx) adds the server-side stream. The two share the mai_vid visitor cookie and the mai_sid session id, which is what lets a journey and a telemetry stream be joined later if that is ever needed.

2. Event taxonomy#

types/telemetry.ts defines exactly 14 TelemetryEventType values. TelemetryRecorder emits only pageview, read_depth and engaged_time today — the rest are accepted by the Zod schema and counted by the rollup, but nothing calls them yet. Wiring a remaining type is a small per-surface change once the spine is proven, not a schema change.

Every event also carries path and sessionId (required), and the stored row (TelemetryEvent) adds server-derived context that the client never sends: contentType and contentSlug (from classifyPath, the same classifier the attribution system uses), referrerHost (host only — a full referrer url can carry tokens), country (from x-vercel-ip-country / cf-ipcountry — the IP itself is never stored), device and isBot / botKind (from lib/telemetry/bots.ts), and clockSkewed (see §4).

share, share_visit and parentShareToken are in the schema and rollup from day one even though nothing emits them yet — the deliberate reason is that adding those columns later would force a schema-version bump and a backfill, whereas shipping them unused now costs nothing. Likewise personId / accountId exist on TelemetryEvent but are stripped at tier 1 and nothing sets them yet (a later phase).

3. The consent boundary#

ConsentTier is 1 | 2. Tier 1 is anonymous-essential: everything keyed to the first-party mai_vid cookie, with no identity join. Tier 2 adds the identity-joinable fields (personId, accountId).

The tier is decided client-side by readConsentTier() in TelemetryRecorder.tsx, evaluated fresh at every flush (passed to the client as a getter, not a value) so a consent change applies to the very next beacon with no listener and no rebuilt client. The check:

• Reads localStorage[CONSENT_STORAGE_KEY] (matters_user_consent, from components/privacy/ConsentManager.tsx).
• Gates on the marketing category — documented in ConsentManager as "RB2B, visitor identification, personalization" and defaulting to false. It deliberately does not gate on analytics, which defaults to true — gating on that would have made tier 2 universal and inverted the fail-closed requirement.
• Reads the nested settings object ({ version, settings: ConsentSettings }), not a flattened record.
• Checks record.version === CONSENT_VERSION (ConsentManager's current version). ConsentManager itself treats a version mismatch as void consent and re-prompts; without this check telemetry would keep honouring a grant the consent system had already discarded after a policy bump.

Fails closed. An absent key, unreadable JSON, a malformed shape, or a stale version all resolve to tier 1 — never tier 2. applyConsentTier() (lib/telemetry/consent.ts) enforces the same boundary again, server-side, at write time: it strips personId / accountId for any non-tier-2 event before the row is built, so a caller that forgets to check the tier still cannot leak identity. redactFreeText() additionally strips anything that looks like an email out of operator-visible free text (title, target) before storage, since a search box or form field is exactly where a visitor's own email arrives by accident.

4. Deterministic event ids#

deriveEventId() (lib/telemetry/event-id.ts) computes the stored row's _id as a SHA-256 hash of:

visitorId + "\0" + sessionId + "\0" + type + "\0" + path + "\0" + tsMs + "\0" + ordinal

ordinal is the event's index within its batch — two genuinely distinct events can share a millisecond, and without it they would derive the same id and one would silently vanish.

The id is computed server-side only, in buildEvent() (lib/telemetry/build-event.ts), never in the browser, for two reasons. First, a client-computed id would be forgeable — a hostile client could mint an id that collides with (and displaces) someone else's row. Second, crypto.subtle.digest is asynchronous, and hashing on the client would push that latency into the sendBeacon path, where nothing is allowed to block or fail visibly.

Because the id depends only on the event's own fields, deriving it twice for the same logical event produces the same id both times. That is what makes a retried batch a no-op rather than a duplicate: Mongo's _id uniqueness turns a retry into a duplicate-key error that MongoEngagementStore.writeEvents() treats as success (see §5), and the same property is what lets the reconciler's set-difference approach (§5, §7) be run repeatedly without ever creating a duplicate.

5. The Zod trust boundary and control characters#

lib/telemetry/schema.ts is the only place client-supplied data is trusted from. Beyond the usual type/length/enum checks (batch capped at MAX_EVENTS_PER_BATCH = 64, schemaVersion a literal — not a range, so a client holding a stale buffer shape is rejected outright), it rejects any C0 control character (Unicode U+0000-U+001F, plus U+007F) in visitorId, sessionId and path via NO_CONTROL_CHARS / safeText().

This is security-load-bearing, not cosmetic. deriveEventId joins those exact fields with a NUL (U+0000) separator, and nothing in the hashing code itself enforces that NUL cannot appear inside a field — that enforcement lives here, at the boundary. Without it, a client could embed a NUL inside visitorId (or sessionId / path) to make its supplied fields split differently than intended, deriving an id that belongs to a different, legitimate field combination. That would pre-write a forged row under that id, so the victim's later genuine event — hashing to the same id — would be dropped as a duplicate. Rejecting control characters at the trust boundary closes that off entirely.

6. Dual-store contract#

EngagementStore (lib/telemetry/store/types.ts) is the seam between the ingest/rollup pipeline and whatever stores the data: writeEvents, contentPerformance, rawEventsForRollup, writeContentDaily, bucketCounts, eventIdsInBucket, eventsByIds, deleteByVisitor. Phase 1 ships only MongoEngagementStore; a columnar implementation is a later phase behind the same interface, and resolveEngagementStore() / resolveWriteTargets() (lib/telemetry/store/resolve.ts) are the only two functions that change when it lands.

Write order is Mongo first, deliberately. resolveWriteTargets() returns Mongo as the first (and today, only) entry because it is the durable landing zone and idempotent by construction (see §4) — an ingest request cannot lose data even if every other configured store is down. In app/api/telemetry/collect/route.ts, the write loop is sequential and stops treating the batch as stored if the first store fails; a later store failing is logged and the touched hour buckets are marked dirty (markBucketDirty) for the reconciler to pick up, but the request is not failed for it.

Silent read fallback. resolveEngagementStore() returns { store, degraded }. Today, with only Mongo configured, it is trivially Mongo and degraded is always false. Once a second store exists, this function is where a short-timeout attempt against the primary read store falls back to Mongo on error, surfacing degraded: true rather than a 500 — a slow analytics engine must never turn a workspace dashboard into an outage. GET /api/workspace/intelligence/content threads degraded straight through to the response, and the Content tab (§8, §9) shows a banner when it is true.

Three reconcile passes, run in app/api/cron/telemetry/reconcile/route.ts, in priority order:

1. Dirty buckets — every hour bucket a failed write touched, taken from the Redis-backed dirty-bucket queue (takeDirtyBuckets, capped at MAX_DIRTY_PER_RUN = 200 per run).
2. Count comparison — diffBucketCounts() (lib/telemetry/reconcile.ts) compares per-hour row counts between the primary and each mirror across the whole retention window, and any mismatched bucket is added to the repair set.
3. Rolling deep sweep — regardless of what the counts say, the oldest DEEP_SWEEP_BUCKETS (24) buckets from the primary's count list are always added to the repair set, and for every bucket in the repair set diffIdSets() compares the actual id sets (not just counts) between the two stores, backfilling whatever is missing on either side.

Counts, not an XOR checksum, and deliberately so. An XOR-of-hashes per bucket would be more elegant — it catches content divergence even when counts already agree — but it does not survive contact with Mongo: there is no portable equivalent to maintain per bucket ($function is tier-gated on Atlas), and computing one at write time would require knowing which documents in a bulkWrite were genuinely new inserts versus duplicate-key no-ops, which the Mongo driver's result does not report. Counts are portable to any engine. The deep sweep is what makes the overall scheme exact rather than merely cheap: two stores can agree on a bucket's count while disagreeing on its actual contents (say, one dropped event N and gained a different event M), and only an id-set diff catches that.

Because event ids are content-derived (§4), backfilling in both directions converges to the union of what both stores hold, and running the same reconcile pass again finds nothing left to repair. That idempotence is what makes it safe to run the cron on a fixed schedule rather than only on demand.

7. Retention and erasure#

Retention is 90 days (RAW_RETENTION_DAYS), and it must be identical on every store. Mongo enforces it with a TTL index (ts_ttl, expireAfterSeconds: RAW_RETENTION_DAYS * 86400), created idempotently by MongoEngagementStore.ensureIndexes() on every rollup cron run. A store with a different retention window would make the reconciler report a permanent, unfixable mismatch on every bucket that has aged out on one store but not the other — so any future store must use the same 90-day window, not its own policy.

Erasure is POST /api/telemetry/erase (app/api/telemetry/erase/route.ts). It is self-service and cookie-authenticated only: the visitor id comes from the caller's own mai_vid cookie, never from the request body — accepting an id from the body would let anyone erase anyone else's history. On a successful call it deletes every row for that visitor from every write target (Mongo today), then upserts a suppression record keyed by visitor id into telemetry_suppressed_visitors (TELEMETRY_SUPPRESSION_COLLECTION).

The suppression record exists because a beacon can be in flight — buffered in the browser — when the erasure request lands. POST /api/telemetry/collect checks the suppression collection for the batch's visitorId before writing anything, and fails closed: if the check itself fails (a database blip), the batch is dropped rather than written, and if the visitor is found suppressed, the batch is dropped. Either way the route still answers 200 with { success: true, accepted: 0 } — its contract constrains the response, not the write: a validation-clean batch must never surface a failure to the visitor or trigger a retry storm. Storing data for a visitor who exercised their right to erasure is a privacy-control failure; losing one beacon during a transient database error is not, and the asymmetry in how the two are handled is intentional.

8. Runbook: "the two screens disagree"#

This is the expected failure mode of any dual-store system, not a bug report on its own. Two dashboards backed by two different stores will eventually show different numbers for the same window — a write that landed on one side and not the other, a reconcile pass that hasn't reached that bucket yet, or a mirror that was down when the beacon arrived. Work through it in this order:

1. Check degraded in the API response. GET /api/workspace/intelligence/content returns { rows, degraded, days }. If degraded: true, the Content tab is already reading from the mirror, not the primary — the numbers you're seeing are honest for the mirror, and the disagreement is expected until the primary recovers. (Also visible in the UI: the Content tab renders a "Served from the mirror store." banner exactly when this is true.) If degraded is false, the primary answered the read and something else is wrong — continue.

2. Run the reconcile cron manually with the secret. It is not scheduled to run on demand from the UI, so trigger it directly:

   curl -s -X POST https://<host>/api/cron/telemetry/reconcile \
     -H "Authorization: Bearer $TELEMETRY_CRON_SECRET"
   

   TELEMETRY_CRON_SECRET is the same shared-secret pattern the rollup cron uses (authorized() in app/api/cron/telemetry/reconcile/route.ts accepts either an Authorization: Bearer <secret> header or x-cron-secret). The route is guarded by a Redis lock (withLock, 900s TTL) — a concurrent run returns { success: true, skipped: "another run in progress" } rather than racing itself; if you see that, wait and retry rather than firing again immediately.

3. Read repaired in its response. A successful run returns { success: true, stores, buckets, repaired }. repaired is the count of individual events that were backfilled in either direction during this run. 0 with buckets > 0 means every bucket it checked was already consistent — the disagreement you're chasing is either outside the buckets this run covered (the deep sweep only advances 24 buckets per run — see §6 — so a 90-day discrepancy can take several runs to fully cover), or it is a rollup-layer problem, not a raw-event problem (see step 4). With only one store configured (stores: 1), the route always reports { skipped: "single store — nothing to reconcile" } — there is nothing to reconcile until a second store exists.

4. Confirm both stores report identical bucketCounts for the disputed window. Once repaired shows work was done, re-run the bucket-count comparison for the specific hours in question — EngagementStore.bucketCounts(fromBucket, toBucket) on each store, using hourBucket() (lib/telemetry/buckets.ts, always UTC — never construct a bucket from host-local time here) to compute the bucket strings for the disputed window. If the counts now match, the raw-event layer is reconciled and any remaining dashboard disagreement is in the rollup, not the events: check whether GET /api/cron/telemetry/rollup has run since the repair (rollups are computed from raw events as of when the rollup cron last ran, so a repair that landed after the last rollup run won't be reflected in telemetry_content_daily — the table the Content tab actually reads — until the next nightly run, or a manual trigger of that same cron with the same secret).

9. Bots and AI crawlers#

classifyUserAgent() (lib/telemetry/bots.ts) classifies every request's User-Agent server-side, before the row is stored. AI crawlers are matched first and specifically, ahead of the generic bot pattern — ClaudeBot and GPTBot would both also match the generic bot|crawler|... regex, so if the AI-crawler patterns were checked second every one of them would collapse into botKind: "generic" and the one signal that actually distinguishes "an LLM read this page" from "some scraper hit this page" would be lost. The four recognized BotKind values are gptbot, claudebot, perplexity and google-extended; anything else matching the generic bot pattern is generic. An empty User-Agent is classified unknown, not a bot — real browsers occasionally send none, and treating that as a bot would silently discard genuine human traffic.

Bot events are stored, not discarded. isBot: true rows are written to telemetry_events exactly like human rows. The exclusion happens one layer up, in rollupContentDaily() (lib/telemetry/rollup.ts): the very first line of its per-event loop is if (event.isBot) continue, so every human-facing metric in telemetry_content_daily and the Content tab (views, unique visitors, read rate, median engaged time, shares, downloads, CTA clicks) is bot-free by construction — a rollup, not a filter applied on the way out.

Keeping bot rows in telemetry_events (rather than dropping them at ingest) is what makes AI crawler reads queryable at all, as a GEO signal: which URLs LLM crawlers are actually fetching. There is no dedicated dashboard surface for this yet — query the raw collection directly, e.g. a Mongo aggregation grouping telemetry_events by path where isBot: true and botKind is one of the four AI-crawler kinds, over the desired date range (ts or dayKey). Because these rows carry the same hourBucket / dayKey fields as human rows, the same bucket/day utilities in lib/telemetry/buckets.ts apply to a bot-only query without any special-casing.

10. Reads50 / reads100 — distinct visitors, not beacons#

Worth calling out on its own because it is easy to get backwards: reads50 and reads100 in ContentDailyRow (lib/telemetry/rollup.ts) count distinct visitors who crossed each threshold, cumulatively — a visitor who reaches 100% also counts toward reads50. They never count beacons. One engaged reader scrolling to the bottom of a long page fires the read_depth event at 50%, 75% and 100% during a single read — counting beacons there would report three readers where there was genuinely one, and a read-rate computed as reads50 / views (which is exactly what the Content tab renders as a percentage) could then exceed 100%. The Content tab's readRate() divides by uniqueVisitors, not views, for the same reason.

11. Share tracking and referral chains#

Shares were previously tracked nowhere. Three hand-rolled widgets (glossary, blogs, videos) were raw <a href> / window.open / navigator.share calls that emitted nothing at all — not even a GA4 event. This section covers what replaced them.

Tokens are minted in the browser, synchronously#

lib/telemetry/share-token.ts mints a 128-bit base64url token (22 characters) using crypto.getRandomValues, with a Math.random fallback.

Client-side and synchronous is not a shortcut, it is a requirement. navigator.share() and navigator.clipboard.writeText() are gesture-gated: any await between the click and the call gets it blocked or silently dropped by the browser. Minting locally needs no round trip, so the gesture stays synchronous. At 128 bits, collision is not a practical concern, so the server never has to arbitrate uniqueness.

A share token is a correlation id, not a secret — holding one authorises nothing — which is why the weaker random fallback degrades attribution rather than security.

The shared URL#

lib/telemetry/share-links.ts stamps the token as ?s=<token>.

• buildShareUrl uses URLSearchParams.set, so re-sharing a link that already carries a token replaces it instead of accumulating a second one and leaving the parent ambiguous.
• stripShareParam removes only the token. UTM parameters are preserved deliberately — the attribution pipeline reads those from the landing URL, and stripping them would break campaign reporting.
• Both fail soft on unparseable input and return the URL untouched. A share that loses its tracking is a measurement gap; a share that produces a broken link is a broken product.

SEO is unaffected: /blogs/[...slug] and /glossary/[slug] both emit alternates.canonical, so the parameter cannot fragment indexing.

Both ends of a share#

On landing, TelemetryRecorder records the share_visit before the pageview so the batch reads in causal order, then history.replaceStates the token away so the URL the visitor sees — and might copy by hand — is clean.

share_visit is deliberately a separate event from pageview. One landing is one click, while a pageview repeats on refresh; conflating them would inflate every share's click count.

Reporting an event from anywhere#

lib/telemetry/record.ts holds a single module-level recorder slot. TelemetryRecorder registers into it on mount and clears it on unmount, and recordTelemetry(facts) fills in the timestamp, path and session so a share button never has to reach into localStorage or read location.

A single slot rather than a subscriber list, because there is exactly one buffer per page — fanning out to a stale closure from a previous mount would write events into a buffer nothing will flush. It is a no-op when nothing is registered, and swallows every failure: it runs inside click handlers, and a telemetry problem must never break the control the visitor pressed.

Referral chains#

lib/telemetry/share-graph.ts → buildShareTrees(events) assembles the chain: A shares, B arrives on A's link, B re-shares, C arrives and requests a demo — and the demo traces back to A.

A share nests under another when its author had already arrived through that link. Ordering is the discriminator: a share minted before the visit cannot have been caused by it. Where someone arrived through several links first, the most recent one before the share wins, since that is the one they were reading when they passed it on.

Two things would otherwise break the traversal, and both are guarded:

• a visitor clicking their own link would make a share its own ancestor;
• clock skew between two visitors' devices can produce a pair that each claim to precede the other.

Hence a visited set plus MAX_SHARE_DEPTH (10). reachOf(node) totals every landing in a subtree, and trees are ordered by reach so the widest chain reads first.

What this does not do: it never reveals a recipient's identity. No network exposes that. It shows that someone arrived and what they did next; the chain resolves to a real person only if and when they convert.

Dark social#

lib/telemetry/arrivals.ts classifies how each session arrived. Most real sharing is invisible: WhatsApp, Slack, iMessage, Signal and every email client strip the referrer, so those arrivals land in "direct" alongside people who typed the domain.

The discriminator is path depth. People type matters.ai or /pricing from memory; nobody types /blogs/dpdp-compliance-checklist-2026. A referrer-less landing on a path deeper than one segment was pasted from somewhere, and is counted as dark_social.

Precedence is deliberate, not fallthrough:

1. a real share token always wins — a tokenised share is measured exactly and must never also be counted as an inference;
2. a campaign outranks the referrer host, which for an ad click is usually just the network's domain;
3. our own domain is internal, not a referral;
4. AI assistants (ChatGPT, Perplexity, Claude, Gemini, Copilot) get their own channel rather than being buried in referrals — with no Search Console integration yet, this is the only visibility into them;
5. then search, then referral;
6. finally, no referrer at all → dark_social if the path is deep, direct if not.

Arrivals are counted once per session, using the session's earliest pageview chosen by timestamp rather than array order, so a visitor reading five articles counts once and the channel mix is not drowned in internal navigation.

The Shares tab#

/intelligence?tab=shares shows the totals, the arrival mix and the nested chains together — the arrival mix beside the trees because the two only make sense as a pair. Tracked shares are the part we see; dark social is the part we can only infer, and a large dark-social number is exactly the measure of how much sharing happens beyond instrumentation.

This tab reads raw events, not rollups, because both answers need per-event ordering that a daily rollup has already discarded. Its range is therefore capped at 30 days against the Content tab's 90, and the longer range buttons disable rather than silently returning a truncated window.

12. Content classification#

classifyPath in lib/attribution/parse.ts maps a pathname to a ContentType and, for slug-bearing routes, its slug — so a dashboard can name which blog or industry page produced a lead rather than only that one did.

It originally knew four prefixes, which meant glossary terms, case studies, industry pages, use-cases, videos, key-features, tools, campaigns and author pages all classified as other. That limited the Lead Attribution dashboard from the day it shipped, and it would have limited the Content tab the same way.

Two structures, deliberately separate:

• PREFIX_MAP for /<prefix>/<slug> routes. Every entry keeps its trailing slash so matching happens on the separator. A bare string prefix would classify /videos-archive as a video and hand it a nonsense slug.
• EXACT_MAP for index and standalone pages, which carry no slug. /industries is the index while /industries/banking is a page within it, and a prefix match would claim both. Exact matches are therefore tried first, and a single trailing slash is normalised so /videos/ and /videos agree.

/case-studies/ and /case-study/ are both mapped, because both spellings are live in the catch-all route.

Adding a type: extend the ContentType union, add the route to the right map, and add the type to CONTENT_PRIORITY in lib/services/marketing/attribution-aggregate.ts. That list is not an exhaustive switch, so TypeScript will not catch the omission — the type will simply fall through to other in the Sankey. Consider whether it also belongs in HIGH_INTENT_CONTENT.

Case studies and tools were added to HIGH_INTENT_CONTENT: case studies are gated exactly as datasheets are, and completing an interactive tool is a deliberate act rather than a scroll.

13. CTA, download and outbound clicks#

One delegated click listener in TelemetryRecorder covers all three.

Tagging is one attribute#

<a href="/demo" data-telemetry="cta:hero-demo">Request a demo</a>
<a href="/downloads/guide.pdf" data-telemetry="download:dpdp-guide">Download</a>

Declarative rather than a hook, because most CTAs on this site live in server components — a hook would force each of them to become a client component. There is nothing to import.

Accepted aliases: cta, download, outbound, copy, print, search, video, form, abandon. The map is an allowlist, not a passthrough: an unrecognised type would fail Zod validation at ingest and take every other event in the same batch down with it, so unknown values are dropped at the source. The value splits on the first colon only, so a target that is itself a URL survives intact.

Outbound clicks are free#

Any link to another origin reports an outbound_click with no tagging at all. Only http(s) counts — mailto: and tel: are not navigations to another site, and counting them would put contact links into a report about referral traffic.

Why capture phase, and why passive#

Capture, because a CTA that calls stopPropagation and navigates programmatically would never be seen during bubbling. Passive, because this listener must never call preventDefault: telemetry is not allowed to change what a click does.