Publish Pipeline

A memo project lives in two places: a writable staging area the owner edits, and an immutable version prefix that links serve from. Publish takes a snapshot of staging, runs the TypeScript compiler over it, and atomically swaps current_version_id.

Staging area#

While editing, files live at:

r2://<R2_BUCKET>/staging/<projectId>/<path>

Two endpoints write here:

• POST /api/owner/projects/<id>/upload — multipart form, accepts loose files and .zip archives. ZIPs are unzipped on the server (lazy unzipper import) and every entry is path-validated. Hard caps: 15 MB per file, 80 MB per upload. Audit-logged as project.upload.
• PUT /api/owner/projects/<id>/files/<path> — write or overwrite a single file. Used by the Monaco editor's Save button. Audit-logged as project.file.write.

DELETE on the same per-path route deletes from staging only. Versions already published are unaffected.

GET on the per-path route streams the staged contents back — Monaco uses this to populate the buffer when a file is selected.

GET /api/owner/projects/<id>/files lists every staged path (R2 ListObjectsV2 filtered by prefix).

All file endpoints validate ownership via Drizzle (projects.owner_id == session.user.id) before any storage read or write.

The compile step#

When the owner clicks Publish, POST /api/owner/projects/<id>/publish runs:

listStagedFiles(projectId)
  → [path₁, path₂, ...]

await Promise.all(paths.map(p => readStagedFile(projectId, p)))
  → CompileFile[]

compileStagedFiles(sources)
  → { status, artifacts, errors, warnings, durationMs }

compileStagedFiles (in src/lib/publish/compile.ts) walks every staged file:

• Non-TS files are passed through as-is.
• .ts / .tsx files are run through native esbuild's build() API (per-file, no bundling — stdin source, write: false):
  • format: 'esm'
  • target: 'es2020'
  • sourcemap: 'inline'
  • bundle: false
  • loader: 'tsx' for .tsx, 'ts' otherwise. Output is written as a sibling .js file. The original .ts source is dropped from the published version — clients only consume the compiled JS.

If any file fails to compile, the response is 422 Unprocessable Entity with the full esbuild error array. Nothing is written to R2 and no versions row is created.

The commit step#

On a clean compile:

1. Generate a new versionId (randomUUID).
2. Compute the next version_number (max + 1).
3. For each artifact:
   • Upload to r2://<bucket>/<userId>/<projectId>/<versionId>/<path> with the correct content-type.
   • Compute sha256 of the bytes for forensic comparison + later dedup.
4. Open a Drizzle transaction (real transactions, via neon-serverless Pool):
   • Insert one versions row with file count, total bytes, compile_status='ok', and the warnings + duration in compile_log.
   • Bulk-insert the files rows (path, mime, size, sha256, r2_key).
   • Update projects.current_version_id to the new version and bump updated_at.
5. Outside the transaction, write an audit_logs entry: action='project.publish', target_type='version', target_id=versionId, meta={ projectId, versionNumber, fileCount, totalBytes }.

Because the version prefix is content-addressed by versionId, publish is atomic for visitors: until current_version_id flips, every existing link continues to serve the old version. The moment it flips, new visits to that link serve the new one.

Note: links reference a specific version_id, not current_version_id — so existing links keep serving the version they were created against. To bump a link to a newer version, an owner explicitly creates a new link (or in v2, an "edit link" endpoint will support PATCH version_id).

TS pipeline tradeoff#

In v1 we deliberately compile only on publish, not on save. This is documented in the project plan as a UX/infra tradeoff:

• Pro: zero esbuild-wasm worker in the editor bundle (~2 MB lazy chunk + worker plumbing + error surface), simpler error model, no client divergence between save and publish.
• Con: the preview pane can't show TypeScript-rendered output until publish. The editor prompts "Publish to preview TypeScript."

A v2 plan adds compile-on-save via esbuild-wasm — additive, no schema change. The publish-time compile remains the canonical pipeline.

Performance#

Typical publish cycles on a 20-file project:

• esbuild build per file: ~5 ms each.
• R2 PUT per file: ~30 ms.
• Postgres transaction (insert version + files): ~50 ms.
• End-to-end p50: ~600 ms for 20 files / 2 MB total.

The R2 PUTs are serialized in the current implementation — Promise.all would be a one-line change but risks pushing into R2's per-account rate ceiling if owners batch-publish in parallel. Punt until measurements demand it.

Verification checklist#

When changing the pipeline, the smoke test is:

1. Drop two files into staging via the editor — index.html, app.ts.
2. POST /publish — assert 200, status: "ok", versionNumber: 1, fileCount: 2 (because app.ts becomes app.js).
3. Inspect R2 — the <userId>/<projectId>/<versionId>/ prefix has index.html + app.js. The .ts is not present.
4. SELECT * FROM memo.versions WHERE id = '<versionId>' — compile_status='ok', compile_log has warnings: [] and a numeric durationMs.
5. SELECT current_version_id FROM memo.projects WHERE id = '<projectId>' — matches.
6. Create a link bound to this version — visit it — view.astro streams index.html with app.js referenced as a sibling file at /<slug>/raw/app.js.