Serve the app shell gzip/brotli-compressed with ETags

2026-09-30 06:45:11 +00:00
parent 7229ca16b9
commit 794410916a
3 changed files with 51 additions and 6 deletions
--- a/plans/done/002-compress-and-etag-shell-bd459c.md
+++ b/plans/done/002-compress-and-etag-shell-bd459c.md
@@ -0,0 +1,163 @@
+---
+id: 002-compress-and-etag-shell-bd459c
+title: Serve the app shell gzip/brotli-compressed with ETags
+created: 2026-09-29
+depends_on: [001-perf-timing-marks-105acc]
+est_files: 2
+---
+
+# 002 — Serve the app shell gzip/brotli-compressed with ETags
+
+## Objective
+
+Measured on prod: `app.js` is sent as 426 244 bytes with no `content-encoding`
+and no `ETag` (2.8 s download at ~150 KB/s). gzip -9 makes it 117 527 bytes;
+the whole shell drops from 618 KB to ~158 KB. After this plan:
+- Every static text file under `./public` (`.js .css .html .json .webmanifest .svg`)
+  is served compressed (`br` preferred, else `gzip`) when the client accepts it,
+  with `Vary: Accept-Encoding`.
+- Every static file carries a strong `ETag` and `If-None-Match` returns `304`.
+- `index.html` (build-stamped at request time) and `/sw.js` (BUILD_TAG-injected) are
+  also compressed + ETagged using the text they actually send.
+- `Cache-Control` values stay EXACTLY as they are today (`no-cache` for the shell,
+  `no-store` for sw.js) — CLAUDE.md "Update-flow architecture" depends on that.
+
+## Context the executor must NOT rediscover
+
+`server/server.js:1895-1910` today:
+
+```js
+function indexHtml(c) {
+  if (_indexSource === null) {
+    try { _indexSource = readFileSync('./public/index.html', 'utf8'); }
+    catch { return c.text('index.html not found', 404); }
+  }
+  return c.html(_indexSource.replace('__BUILD_TAG__', BUILD_TAG), 200, { 'Cache-Control': 'no-cache' });
+}
+app.get('/', indexHtml);
+app.get('/index.html', indexHtml);
+...
+app.use('/*', serveStatic({ root: './public', onFound: (_path, c) => { c.header('Cache-Control', 'no-cache'); } }));
+// SPA fallback — return index.html for any unmatched path
+app.get('/*', indexHtml);
+```
+
+`/sw.js` handler at ~line 1860 ends with `return c.text(src, 200, { ...no-store headers })`.
+
+Bun provides `Bun.gzipSync(buf, { level: 9 })`. Brotli: `import { brotliCompressSync, constants } from 'node:zlib'`
+(works in Bun). `createHash` is already imported from `node:crypto` at the top.
+
+**Never compress** `/api/play`, `/api/media/*`, `/api/download/*` (Range/binary) —
+they are not served by `serveStatic`, so a static-only middleware cannot touch them.
+
+## Steps
+
+1. `server/server.js` — add near the top imports:
+   `import { brotliCompressSync, constants as zlibConstants } from 'node:zlib';`
+2. `server/server.js` — directly ABOVE `function indexHtml(c) {` add this helper block:
+   ```js
+   // Compressed + ETagged static text. The shell is ~620 KB raw / ~160 KB gzip
+   // and every byte crosses the slow VPS→homelab link, so compress once per
+   // file content and keep it in memory. ETag = sha256 of the RAW bytes, so a
+   // `no-cache` revalidation costs a 304 instead of the whole file.
+   const COMPRESSIBLE = /\.(js|css|html|json|webmanifest|svg|txt)$/i;
+   const compressedCache = new Map(); // key -> { etag, raw, gz, br, type }
+   function compressedEntry(key, raw, type) {
+     let e = compressedCache.get(key);
+     const etag = '"' + createHash('sha256').update(raw).digest('hex').slice(0, 32) + '"';
+     if (e && e.etag === etag) return e;
+     e = {
+       etag, raw, type,
+       gz: Bun.gzipSync(raw, { level: 9 }),
+       br: brotliCompressSync(raw, { params: { [zlibConstants.BROTLI_PARAM_QUALITY]: 11 } }),
+     };
+     compressedCache.set(key, e);
+     return e;
+   }
+   function sendCompressed(c, e, cacheControl) {
+     const headers = { 'Content-Type': e.type, 'Cache-Control': cacheControl, ETag: e.etag, Vary: 'Accept-Encoding' };
+     const inm = c.req.header('if-none-match') || '';
+     if (inm.split(',').map((s) => s.trim()).includes(e.etag)) return new Response(null, { status: 304, headers });
+     const ae = c.req.header('accept-encoding') || '';
+     if (/\bbr\b/.test(ae)) return new Response(e.br, { headers: { ...headers, 'Content-Encoding': 'br' } });
+     if (/\bgzip\b/.test(ae)) return new Response(e.gz, { headers: { ...headers, 'Content-Encoding': 'gzip' } });
+     return new Response(e.raw, { headers });
+   }
+   const MIME = { js: 'text/javascript; charset=utf-8', css: 'text/css; charset=utf-8', html: 'text/html; charset=utf-8',
+     json: 'application/json', webmanifest: 'application/manifest+json', svg: 'image/svg+xml', txt: 'text/plain; charset=utf-8' };
+   ```
+3. `server/server.js` — change `indexHtml` so its return line becomes:
+   ```js
+   const html = _indexSource.replace('__BUILD_TAG__', BUILD_TAG);
+   return sendCompressed(c, compressedEntry('index.html', Buffer.from(html), MIME.html), 'no-cache');
+   ```
+4. `server/server.js` — in the `/sw.js` handler, replace its final return, which is exactly:
+   ```js
+     return c.text(src, 200, {
+       'Content-Type': 'application/javascript; charset=utf-8',
+       'Cache-Control': 'no-store, no-cache, must-revalidate',
+     });
+   ```
+   with:
+   ```js
+     return sendCompressed(c, compressedEntry('sw.js', Buffer.from(src), MIME.js), 'no-store, no-cache, must-revalidate');
+   ```
+   (`text/javascript` is a valid service-worker MIME type.) Dry-run result: app.js 426 244 → 93 717 bytes (br).
+5. `server/server.js` — directly BEFORE the `app.use('/*', serveStatic(...))` line add:
+   ```js
+   app.get('/*', async (c, next) => {
+     const p = decodeURIComponent(new URL(c.req.url).pathname);
+     if (!COMPRESSIBLE.test(p) || p.includes('..') || p.startsWith('/api/')) return next();
+     const file = Bun.file('./public' + p);
+     if (!(await file.exists())) return next();
+     const raw = Buffer.from(await file.arrayBuffer());
+     const ext = p.slice(p.lastIndexOf('.') + 1).toLowerCase();
+     return sendCompressed(c, compressedEntry(p, raw, MIME[ext] || 'application/octet-stream'), 'no-cache');
+   });
+   ```
+   (`/sw.js`, `/` and `/index.html` are registered earlier and win; this only covers the rest.)
+
+## Out of scope / do NOT touch
+
+- `frontend/sw.js`, `frontend/sw-update.js`: no changes. The SW fetches shell files with
+  `cache: 'reload'` and `?__ytpfresh=` — query strings don't affect the pathname match, fine.
+- Do not change any `Cache-Control` value. Do not add `hono/compress` globally (it would
+  hit Range media responses).
+- Do not touch `computeBuildTag`.
+
+## Verification
+
+```bash
+cd /home/user/ytplayer/server && [ -e public ] || ln -s ../frontend public
+PORT=3999 bun server.js > /tmp/ytp002.log 2>&1 & SRV=$!; sleep 4
+curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip, br' http://localhost:3999/app.js | grep -iE 'content-encoding|etag|cache-control|vary'
+ET=$(curl -s -D - -o /dev/null http://localhost:3999/app.js | grep -i '^etag' | cut -d' ' -f2 | tr -d '\r')
+curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" http://localhost:3999/app.js
+curl -s -H 'Accept-Encoding: gzip' --compressed http://localhost:3999/ | grep -c 'ytp-build'
+curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip' http://localhost:3999/sw.js | grep -iE 'content-encoding|cache-control'
+curl -s --compressed http://localhost:3999/sw.js | grep -c "__BUILD_TAG__ !== 'undefined'" ; true
+kill $SRV; true
+cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
+```
+
+Expected: `content-encoding: br`, an `etag`, `cache-control: no-cache`, `vary: Accept-Encoding`;
+the If-None-Match request prints `304`; index grep prints `1`; sw.js shows `content-encoding: gzip`
+and its original no-store cache-control; the last grep prints `0` (tag was injected); tests `fail 0`.
+
+## Report format (executor: follow exactly)
+
+Output ONLY the following, no other prose:
+
+1. `git diff` (unified) of all changes.
+2. Raw output of the Verification commands.
+3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
+   skipped and why.
+
+Do not commit. Do not push. Do not touch files outside the Steps.
+
+## Execution log
+
+- Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 0.
+- Orchestrator re-ran Verification on a fresh port: `Content-Encoding: br`, ETag, `Vary`, 304 on If-None-Match, index stamped (`ytp-build` 1), sw.js gzip + original no-store `Cache-Control`, BUILD_TAG injected (0 placeholders), SPA fallback 200, 52 frontend tests pass. app.js 426 244 -> 93 759 bytes.
+- Executor Findings (verbatim): All 5 verification tests pass. All unit tests pass. Changes follow the plan exactly: added brotli/zlib imports, implemented compressedEntry/sendCompressed helpers with ETag and dual-compression support (br preferred, fallback to gzip), updated indexHtml and sw.js handlers to use sendCompressed, and added middleware for compressing all text files under ./public. Cache-Control headers preserved exactly as specified. No changes made to sw.js, sw-update.js, or computeBuildTag.
+- Orchestrator note: the executor left a background server on port 3999; killed. Cosmetic: the helper block sits between the old indexHtml comment and `let _indexSource`.