215 lines
11 KiB
Markdown
215 lines
11 KiB
Markdown
# Performance Baseline Harness (`perf/`)
|
||
|
||
Automated performance baseline and regression harness for the **ytplayer PWA**.
|
||
|
||
The harness drives the real Bun backend server (`server/server.js`) serving static Brotli-compressed assets, combined with Playwright browser automation on Chromium and WebKit.
|
||
|
||
---
|
||
|
||
## 1. Architecture
|
||
|
||
### Backend Server (`server/server.js`)
|
||
- Runs as an isolated child process using **Bun** on an ephemeral port.
|
||
- Configured with dedicated scratch directories under `perf/.tmp/` (`DATA_DIR`, `DB_PATH`, `MEDIA_DIR`) so no production or developer data is ever touched.
|
||
- Employs the real static asset delivery stack (`Brotli` encoding, `ETag` generation, `Cache-Control` immutable caching for version-stamped assets, and `computeBuildTag()`).
|
||
|
||
### Browser Automation & Network Throttling
|
||
- **Chromium**:
|
||
- Network emulation via Chrome DevTools Protocol (`Network.emulateNetworkConditions`).
|
||
- CPU throttling at **4× slowdown** (`Emulation.setCPUThrottlingRate`).
|
||
- Unrestricted audio/video playback via `--autoplay-policy=no-user-gesture-required`.
|
||
- **WebKit**:
|
||
- WebKit on Linux lacks CDP network emulation capabilities.
|
||
- Throttling and stalling are handled by a dedicated local proxy (`perf/proxy.mjs`) sitting between WebKit and the Bun server.
|
||
- The proxy shapes traffic by pacing chunk delivery through backpressured streams and injecting round-trip latency and stalls.
|
||
- **Network Profiles**:
|
||
- **`lte`**: 1.6 Mbps (200 KB/s) download, 750 kbps upload, 150 ms round-trip latency.
|
||
- **`lossy`**: 1.0 Mbps (125 KB/s) download, 250 ms latency, with intermittent connection stalls (every 7th request delayed by 1,500 ms).
|
||
|
||
### Shell Fixture Extractor (`perf/make-shell-fixture.sh`)
|
||
- Archives the frontend from git commits (`git archive <commit> frontend`) into `perf/fixtures/shell-<tag>/`.
|
||
- Computes the build tag using the exact sha256 walk algorithm from `server/server.js computeBuildTag()`.
|
||
- Validates the computed tag against live production (`https://worship.hesed.sbs/api/version`).
|
||
|
||
---
|
||
|
||
## 2. Test Scenarios
|
||
|
||
1. **`cold` (Cold First Load)**:
|
||
- Fresh browser context with empty cache, no service worker, and clean storage.
|
||
- Measures initial page fetch, FCP, LCP, DOMContentLoaded, boot milestone (`#cards` render), media playback readiness, service worker precaching duration, and origin storage quota.
|
||
2. **`warm` (Warm Reload under SW)**:
|
||
- Re-navigates the page with active Service Worker and primed HTTP cache.
|
||
- Measures cached asset retrieval efficiency and second-load boot performance.
|
||
3. **`offline` (Offline Reload)**:
|
||
- Sets offline state and reloads the application.
|
||
- Verifies the app boots entirely from `CacheStorage` precached shell assets without network connectivity.
|
||
4. **`update-js` (Update N → N+1 with 1-line JS comment)**:
|
||
- Boots client against Tree N (base frontend), establishing an active Service Worker.
|
||
- Swaps server to Tree N+1 (one comment line appended to `app.js`, altering `BUILD_TAG`).
|
||
- Triggers `registration.update()` and measures total wire bytes and time until the new Service Worker reaches `waiting` state (`installed`).
|
||
5. **`update-css` (Update N → N+1 with 1-line CSS rule)**:
|
||
- Identical to `update-js`, but Tree N+1 has one CSS rule appended to `theme-glass.css`.
|
||
- Measures the full shell re-download penalty caused by any stylesheet modification.
|
||
|
||
---
|
||
|
||
## 3. Metrics Collected
|
||
|
||
Each scenario records the following metrics across **5 independent runs** (reporting min, max, and median):
|
||
- **Request Count**: Total HTTP requests made across the wire.
|
||
- **Wire Bytes**: Total compressed bytes transferred over the network.
|
||
- **Wire Bytes by Type**: Categorized byte breakdown (`html`, `js`, `css`, `fonts`, `images`, `media`, `api`, `other`).
|
||
- **FCP (First Contentful Paint)**: Time from navigation start to first rendered content (ms).
|
||
- **LCP (Largest Contentful Paint)**: Time to largest contentful paint via `PerformanceObserver` (ms).
|
||
- **DOMContentLoaded**: Time to DOMContentLoaded event end (ms).
|
||
- **Boot Done**: Time from navigation start until the first `#cards` view rendering is completed (ms).
|
||
- **Long Tasks Total**: Cumulative execution time of main thread tasks exceeding 50 ms (ms, Chromium).
|
||
- **SW Install Duration**: Time from Service Worker registration until precaching completes (ms).
|
||
- **SW Install Wire Bytes**: Network bytes transmitted during Service Worker installation.
|
||
- **Storage Estimate**: Quota and usage reported by `navigator.storage.estimate()` (bytes).
|
||
- **Media Ready**: Elapsed time for deterministic media fixture (`benchmedia01`) to reach `readyState >= 3` / playing state. Notes if autoplay policy blocked playback.
|
||
|
||
---
|
||
|
||
## 4. How to Run
|
||
|
||
### Prerequisites
|
||
Symlink dependencies and ensure server dependencies are installed:
|
||
```bash
|
||
# Worktree root
|
||
ln -s ~/development/personal/ytplayer/node_modules node_modules
|
||
cd server && bun install && cd ..
|
||
```
|
||
|
||
### Basic Commands
|
||
Run the complete baseline suite (5 runs, all browsers, LTE profile):
|
||
```bash
|
||
node perf/baseline.mjs
|
||
```
|
||
|
||
Run a specific browser or profile:
|
||
```bash
|
||
# Chromium on LTE profile
|
||
node perf/baseline.mjs --browser chromium --profile lte
|
||
|
||
# WebKit on LTE profile
|
||
node perf/baseline.mjs --browser webkit --profile lte
|
||
|
||
# Both browsers on lossy profile
|
||
node perf/baseline.mjs --browser all --profile lossy
|
||
```
|
||
|
||
Run a single scenario:
|
||
```bash
|
||
# Cold load only
|
||
node perf/baseline.mjs --scenario cold
|
||
|
||
# Update scenario with 1-line JS change
|
||
node perf/baseline.mjs --scenario update-js
|
||
|
||
# Update scenario with 1-line CSS change
|
||
node perf/baseline.mjs --scenario update-css
|
||
```
|
||
|
||
Configure run count:
|
||
```bash
|
||
# Quick sanity check (1 iteration)
|
||
node perf/baseline.mjs --runs 1
|
||
|
||
# High precision benchmark (7 iterations)
|
||
node perf/baseline.mjs --runs 7
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Phase 5 Comparison Mode (`--compare`)
|
||
|
||
To verify performance gains and detect regressions in later phases:
|
||
```bash
|
||
node perf/baseline.mjs --compare perf/results/baseline-2026-10-07.json
|
||
```
|
||
Prints an ASCII/Markdown delta comparison table highlighting percentage changes across wire bytes, boot duration, and update overhead.
|
||
|
||
---
|
||
|
||
## 6. Hidden Timers Verification (`perf/hidden-timers.mjs`)
|
||
|
||
Verifies that non-playback UI intervals (`chips`, `reanchor`, `rotateChips`, `flush`, `checkBuildTag`) suspend while the document is hidden and restore upon visibility, while essential background intervals (`maybeCheck`, 120s resume check) remain active:
|
||
```bash
|
||
node perf/hidden-timers.mjs
|
||
```
|
||
Exits with code 0 on success, or code 1 if any interval fails to pause, fails to resume, or if page errors occur.
|
||
|
||
## Phase 1 production-client migration
|
||
|
||
Build the production fixture locally without contacting the unavailable production
|
||
site (COMMON.md takes precedence over the older live-tag instructions above):
|
||
|
||
```bash
|
||
perf/make-shell-fixture.sh b77938a 1175f1a1d2c1
|
||
node perf/migration.mjs # Chromium and WebKit
|
||
node perf/migration.mjs --browser webkit # one browser
|
||
```
|
||
|
||
The harness archives the actual `b77938a` server and uses the reconstructed old
|
||
frontend, isolated database/media/upload paths, and an ephemeral origin behind the
|
||
`lossy` throttle/stall proxy. It restarts onto the current server and frontend at
|
||
that same origin, then invokes the OLD application's update check and clicks its
|
||
actual Refresh UI button. Assertions cover one banner, exactly one reload, cleared
|
||
update-attempt state, all new SHELL entries with correct hashes, no script errors,
|
||
and offline boot. Plain, stale `?v=<oldTag>` and `?__ytpfresh=` asset responses are
|
||
also compared byte-for-byte with the current files. Results are stored in
|
||
`perf/results/phase1-migration-2026-10-07.json`; scratch trees are removed on exit.
|
||
No service-worker source is patched or simulated. Offline checks stop Bun and
|
||
block the proxy; WebKit avoids its broken native setOffline API. Known WebKit
|
||
transport errors for disconnected version/recommendation requests are verified
|
||
separately from script errors.
|
||
|
||
The current server snapshots static bytes at boot; restart it after frontend
|
||
changes. `ASSET_HASHING=0` restores the previous single-tag stamp/cache behavior.
|
||
The manifest's tag hashes sorted canonical metadata: CSS/JS URLs are stamped
|
||
before hashing the index, with its derived build-meta placeholder intact. The
|
||
published index/SW hashes describe their final served bytes after tag injection;
|
||
these derived hashes are not fed recursively into their own build identity.
|
||
|
||
JS-created workers/importScripts/dynamic imports retain their existing plain URLs
|
||
in Phase 1. Adding a separate loader would require changing the intentionally
|
||
unchanged SHELL; worker import URL propagation belongs with the client migration.
|
||
WASM binaries are embedded in the existing `sha256-wasm.js`/`loudness-wasm.js`,
|
||
which receive hashed URLs as HTML scripts. Font preloads retain plain URLs to match
|
||
`fonts.css`; the webmanifest and its icon URLs also stay plain and precached.
|
||
`piano-engine.mjs` is manifested but is historically absent from SHELL; this
|
||
existing offline piano limitation is deferred to the loader phase.
|
||
|
||
## Phase 2 incremental-sync verification
|
||
|
||
The current migration harness verifies the production page's own ready fast path
|
||
into `ytplayer-assets`: no `__ytpfresh` downloads, no reload before Refresh UI,
|
||
one banner/reload, an open N-1 tab, playback guard, incremental refresh after
|
||
an eviction, and offline launch. It preserves the legacy user-triggered reload
|
||
exception. Async CacheStorage/registration conditions are polled with awaited
|
||
`page.evaluate`; Playwright's `waitForFunction` does not poll async predicates.
|
||
|
||
```bash
|
||
node perf/migration.mjs --browser all --profile lossy
|
||
node perf/migration.mjs --browser all --profile unthrottled
|
||
node perf/baseline.mjs --runs 1 --browser all --profile lte --scenario all \
|
||
--out perf/results/phase2-final-2026-10-07.json \
|
||
--compare perf/results/after-phase0-2026-10-07.json
|
||
ASSET_SYNC=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase2-rollback-2026-10-07.json
|
||
ASSET_HASHING=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase2-hashing-rollback-2026-10-07.json
|
||
```
|
||
|
||
Results now include response paths/statuses and compressed response-body bytes
|
||
for cold/update runs. The total includes service-worker script checks/imports;
|
||
inspect individual responses to distinguish those from app asset downloads.
|
||
Migration results are `phase2-migration-<profile>-2026-10-07.json`.
|
||
`ASSET_SYNC=0` selects the legacy path while preserving the persistent utility
|
||
cache. `ASSET_HASHING=0` also selects legacy sync because single-tag URLs cannot
|
||
pass per-file hash checks. Current assets.json groups all block; idle loading and
|
||
lazy parsing await Phase 3. Incremental mode now caches the manifested
|
||
`piano-engine.mjs` even though it remains absent from the legacy SHELL.
|