338 lines
18 KiB
Markdown
338 lines
18 KiB
Markdown
# Performance Baseline Harness (`perf/`)
|
||
|
||
Automated performance baseline and regression harness for the **ytplayer PWA**.
|
||
|
||
The harness drives the real Bun backend server (`server/server.js`) serving static Brotli-compressed assets, combined with Playwright browser automation on Chromium and WebKit.
|
||
|
||
---
|
||
|
||
## 1. Architecture
|
||
|
||
### Backend Server (`server/server.js`)
|
||
- Runs as an isolated child process using **Bun** on an ephemeral port.
|
||
- Configured with dedicated scratch directories under `perf/.tmp/` (`DATA_DIR`, `DB_PATH`, `MEDIA_DIR`) so no production or developer data is ever touched.
|
||
- Employs the real static asset delivery stack (`Brotli` encoding, `ETag` generation, `Cache-Control` immutable caching for version-stamped assets, and `computeBuildTag()`).
|
||
|
||
### Browser Automation & Network Throttling
|
||
- **Chromium**:
|
||
- Network emulation via Chrome DevTools Protocol (`Network.emulateNetworkConditions`).
|
||
- CPU throttling at **4× slowdown** (`Emulation.setCPUThrottlingRate`).
|
||
- Unrestricted audio/video playback via `--autoplay-policy=no-user-gesture-required`.
|
||
- **WebKit**:
|
||
- WebKit on Linux lacks CDP network emulation capabilities.
|
||
- Throttling and stalling are handled by a dedicated local proxy (`perf/proxy.mjs`) sitting between WebKit and the Bun server.
|
||
- The proxy shapes traffic by pacing chunk delivery through backpressured streams and injecting round-trip latency and stalls.
|
||
- **Network Profiles**:
|
||
- **`lte`**: 1.6 Mbps (200 KB/s) download, 750 kbps upload, 150 ms round-trip latency.
|
||
- **`lossy`**: 1.0 Mbps (125 KB/s) download, 250 ms latency, with intermittent connection stalls (every 7th request delayed by 1,500 ms).
|
||
|
||
### Shell Fixture Extractor (`perf/make-shell-fixture.sh`)
|
||
- Archives the frontend from git commits (`git archive <commit> frontend`) into `perf/fixtures/shell-<tag>/`.
|
||
- Computes the build tag using the exact sha256 walk algorithm from `server/server.js computeBuildTag()`.
|
||
- Validates the computed tag against live production (`https://worship.hesed.sbs/api/version`).
|
||
|
||
---
|
||
|
||
## 2. Test Scenarios
|
||
|
||
1. **`cold` (Cold First Load)**:
|
||
- Fresh browser context with empty cache, no service worker, and clean storage.
|
||
- Measures initial page fetch, FCP, LCP, DOMContentLoaded, boot milestone (`#cards` render), media playback readiness, service worker precaching duration, and origin storage quota.
|
||
2. **`warm` (Warm Reload under SW)**:
|
||
- Re-navigates the page with active Service Worker and primed HTTP cache.
|
||
- Measures cached asset retrieval efficiency and second-load boot performance.
|
||
3. **`offline` (Offline Reload)**:
|
||
- Sets offline state and reloads the application.
|
||
- Verifies the app boots entirely from `CacheStorage` precached shell assets without network connectivity.
|
||
4. **`update-js` (Update N → N+1 with 1-line JS comment)**:
|
||
- Boots client against Tree N (base frontend), establishing an active Service Worker.
|
||
- Swaps server to Tree N+1 (one comment line appended to `app.js`, altering `BUILD_TAG`).
|
||
- Triggers `registration.update()` and measures total wire bytes and time until the new Service Worker reaches `waiting` state (`installed`).
|
||
5. **`update-css` (Update N → N+1 with 1-line CSS rule)**:
|
||
- Identical to `update-js`, but Tree N+1 has one CSS rule appended to `theme-glass.css`.
|
||
- Measures the full shell re-download penalty caused by any stylesheet modification.
|
||
|
||
---
|
||
|
||
## 3. Metrics Collected
|
||
|
||
Each scenario records the following metrics across **5 independent runs** (reporting min, max, and median):
|
||
- **Request Count**: Total HTTP requests made across the wire.
|
||
- **Wire Bytes**: Total compressed bytes transferred over the network.
|
||
- **Wire Bytes by Type**: Categorized byte breakdown (`html`, `js`, `css`, `fonts`, `images`, `media`, `api`, `other`).
|
||
- **FCP (First Contentful Paint)**: Time from navigation start to first rendered content (ms).
|
||
- **LCP (Largest Contentful Paint)**: Time to largest contentful paint via `PerformanceObserver` (ms).
|
||
- **DOMContentLoaded**: Time to DOMContentLoaded event end (ms).
|
||
- **Boot Done**: Time from navigation start until the first `#cards` view rendering is completed (ms).
|
||
- **Long Tasks Total**: Cumulative execution time of main thread tasks exceeding 50 ms (ms, Chromium).
|
||
- **SW Install Duration**: Time from Service Worker registration until precaching completes (ms).
|
||
- **SW Install Wire Bytes**: Network bytes transmitted during Service Worker installation.
|
||
- **Storage Estimate**: Quota and usage reported by `navigator.storage.estimate()` (bytes).
|
||
- **Media Ready**: Elapsed time for deterministic media fixture (`benchmedia01`) to reach `readyState >= 3` / playing state. Notes if autoplay policy blocked playback.
|
||
|
||
---
|
||
|
||
## 4. How to Run
|
||
|
||
### Prerequisites
|
||
Symlink dependencies and ensure server dependencies are installed:
|
||
```bash
|
||
# Worktree root
|
||
ln -s ~/development/personal/ytplayer/node_modules node_modules
|
||
cd server && bun install && cd ..
|
||
```
|
||
|
||
### Basic Commands
|
||
Run the complete baseline suite (5 runs, all browsers, LTE profile):
|
||
```bash
|
||
node perf/baseline.mjs
|
||
```
|
||
|
||
Run a specific browser or profile:
|
||
```bash
|
||
# Chromium on LTE profile
|
||
node perf/baseline.mjs --browser chromium --profile lte
|
||
|
||
# WebKit on LTE profile
|
||
node perf/baseline.mjs --browser webkit --profile lte
|
||
|
||
# Both browsers on lossy profile
|
||
node perf/baseline.mjs --browser all --profile lossy
|
||
```
|
||
|
||
Run a single scenario:
|
||
```bash
|
||
# Cold load only
|
||
node perf/baseline.mjs --scenario cold
|
||
|
||
# Update scenario with 1-line JS change
|
||
node perf/baseline.mjs --scenario update-js
|
||
|
||
# Update scenario with 1-line CSS change
|
||
node perf/baseline.mjs --scenario update-css
|
||
```
|
||
|
||
Configure run count:
|
||
```bash
|
||
# Quick sanity check (1 iteration)
|
||
node perf/baseline.mjs --runs 1
|
||
|
||
# High precision benchmark (7 iterations)
|
||
node perf/baseline.mjs --runs 7
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Phase 5 Comparison Mode (`--compare`)
|
||
|
||
To verify performance gains and detect regressions in later phases:
|
||
```bash
|
||
node perf/baseline.mjs --compare perf/results/baseline-2026-10-07.json
|
||
```
|
||
Prints an ASCII/Markdown delta comparison table highlighting percentage changes across wire bytes, boot duration, and update overhead.
|
||
|
||
---
|
||
|
||
## 6. Hidden Timers Verification (`perf/hidden-timers.mjs`)
|
||
|
||
Verifies that non-playback UI intervals (`chips`, `reanchor`, `rotateChips`, `flush`, `checkBuildTag`) suspend while the document is hidden and restore upon visibility, while essential background intervals (`maybeCheck`, 120s resume check) remain active:
|
||
```bash
|
||
node perf/hidden-timers.mjs
|
||
```
|
||
Exits with code 0 on success, or code 1 if any interval fails to pause, fails to resume, or if page errors occur.
|
||
|
||
## Phase 1 production-client migration
|
||
|
||
Build the production fixture locally without contacting the unavailable production
|
||
site (COMMON.md takes precedence over the older live-tag instructions above):
|
||
|
||
```bash
|
||
perf/make-shell-fixture.sh b77938a 1175f1a1d2c1
|
||
node perf/migration.mjs # Chromium and WebKit
|
||
node perf/migration.mjs --browser webkit # one browser
|
||
```
|
||
|
||
The harness archives the actual `b77938a` server and uses the reconstructed old
|
||
frontend, isolated database/media/upload paths, and an ephemeral origin behind the
|
||
`lossy` throttle/stall proxy. It restarts onto the current server and frontend at
|
||
that same origin, then invokes the OLD application's update check and clicks its
|
||
actual Refresh UI button. Assertions cover one banner, exactly one reload, cleared
|
||
update-attempt state, all new SHELL entries with correct hashes, no script errors,
|
||
and offline boot. Plain, stale `?v=<oldTag>` and `?__ytpfresh=` asset responses are
|
||
also compared byte-for-byte with the current files. Results are stored in
|
||
`perf/results/phase1-migration-2026-10-07.json`; scratch trees are removed on exit.
|
||
No service-worker source is patched or simulated. Offline checks stop Bun and
|
||
block the proxy; WebKit avoids its broken native setOffline API. Known WebKit
|
||
transport errors for disconnected version/recommendation requests are verified
|
||
separately from script errors.
|
||
|
||
The current server snapshots static bytes at boot; restart it after frontend
|
||
changes. `ASSET_HASHING=0` restores the previous single-tag stamp/cache behavior.
|
||
The manifest's tag hashes sorted canonical metadata: CSS/JS URLs are stamped
|
||
before hashing the index, with its derived build-meta placeholder intact. The
|
||
published index/SW hashes describe their final served bytes after tag injection;
|
||
these derived hashes are not fed recursively into their own build identity.
|
||
|
||
JS-created workers/importScripts/dynamic imports retain their existing plain URLs
|
||
in Phase 1. Adding a separate loader would require changing the intentionally
|
||
unchanged SHELL; worker import URL propagation belongs with the client migration.
|
||
WASM binaries are embedded in the existing `sha256-wasm.js`/`loudness-wasm.js`,
|
||
which receive hashed URLs as HTML scripts. Font preloads retain plain URLs to match
|
||
`fonts.css`; the webmanifest and its icon URLs also stay plain and precached.
|
||
`piano-engine.mjs` is manifested but is historically absent from SHELL; this
|
||
existing offline piano limitation is deferred to the loader phase.
|
||
|
||
## Phase 2 incremental-sync verification
|
||
|
||
The current migration harness verifies the production page's own ready fast path
|
||
into `ytplayer-assets`: no `__ytpfresh` downloads, no reload before Refresh UI,
|
||
one banner/reload, an open N-1 tab, playback guard, incremental refresh after
|
||
an eviction, and offline launch. It preserves the legacy user-triggered reload
|
||
exception. Async CacheStorage/registration conditions are polled with awaited
|
||
`page.evaluate`; Playwright's `waitForFunction` does not poll async predicates.
|
||
|
||
```bash
|
||
node perf/migration.mjs --browser all --profile lossy
|
||
node perf/migration.mjs --browser all --profile unthrottled
|
||
node perf/baseline.mjs --runs 1 --browser all --profile lte --scenario all \
|
||
--out perf/results/phase2-final-2026-10-07.json \
|
||
--compare perf/results/after-phase0-2026-10-07.json
|
||
ASSET_SYNC=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase2-rollback-2026-10-07.json
|
||
ASSET_HASHING=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase2-hashing-rollback-2026-10-07.json
|
||
```
|
||
|
||
Results now include response paths/statuses and compressed response-body bytes
|
||
for cold/update runs. The total includes service-worker script checks/imports;
|
||
inspect individual responses to distinguish those from app asset downloads.
|
||
Migration results are `phase2-migration-<profile>-2026-10-07.json`.
|
||
`ASSET_SYNC=0` selects the legacy path while preserving the persistent utility
|
||
cache. `ASSET_HASHING=0` also selects legacy sync because single-tag URLs cannot
|
||
pass per-file hash checks. In Phase 2 all assets.json groups blocked; Phase 3 selects core plus the active
|
||
layout and warms background groups at idle. Incremental mode now caches the manifested
|
||
`piano-engine.mjs` even though it remains absent from the legacy SHELL.
|
||
|
||
|
||
## Phase 3 lazy loading and staged apply
|
||
|
||
```bash
|
||
node perf/lazy.mjs --browser all
|
||
node perf/migration.mjs --browser all --profile lossy
|
||
node perf/migration.mjs --browser all --profile unthrottled
|
||
node perf/baseline.mjs --runs 1 --browser all --profile lte --scenario all \
|
||
--out perf/results/phase3-final-2026-10-08.json \
|
||
--compare perf/results/after-phase0-2026-10-07.json
|
||
node perf/baseline.mjs --runs 1 --browser all --profile lte --scenario cold \
|
||
--frontend-commit 7bc5b4f8b50d78ece3cca5100d54a97fece2748a \
|
||
--out perf/results/phase3-phase0-initial-2026-10-08.json
|
||
ASSET_SYNC=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase3-sync-rollback-2026-10-08.json
|
||
ASSET_HASHING=0 node perf/baseline.mjs --runs 1 --browser all --profile lte \
|
||
--scenario offline --out perf/results/phase3-hashing-rollback-2026-10-08.json
|
||
```
|
||
|
||
`initialJsCss` sums compressed Resource Timing bodies for page JS/CSS initiated
|
||
before DOMContentLoaded, excluding SW precache and later idle feature parsing.
|
||
The saved Phase 0 results lack that metric: `--frontend-commit` reconstructs its
|
||
frontend in scratch using the current delivery backend for a matched initial-byte
|
||
comparison. Historical boot/FCP comparisons still use `after-phase0`. Reduced
|
||
single-run measurements are directional; Phase 5 supplies repeated medians.
|
||
Update scenarios now wait for every build-N group to finish idle caching before
|
||
starting the counter, excluding unfinished initial downloads. Waiting-worker
|
||
bytes represent blocking readiness; an inactive layout's changed CSS downloads
|
||
later at idle after activation.
|
||
|
||
The browser harness checks pre-body layout selection, immediate offline reload,
|
||
four distinct Car mode controls, optional Settings search, all extension panels,
|
||
zero network for warmed first use/layout switches, contract guards and pinned
|
||
instances. Known WebKit disconnected API transport errors are recorded separately;
|
||
script errors still fail. Save-Data prevents automatic warming.
|
||
|
||
The index contains inert JSON and an external head bootstrap under unchanged
|
||
CSP. Source JSON mirrors assets.json for native/static shells; the server replaces
|
||
it with hashed build-local URLs. No boot-time manifest network request is added.
|
||
Shared layout visibility defaults remain eager in layout-base.css.
|
||
Pure EQ settings metadata and DeviceDB remain eager to preserve synchronous
|
||
player defaults and offline file bookkeeping. P2P/direct parse on actual use or
|
||
the existing eight-second sharing startup. Searching Settings explicitly loads
|
||
optional panels so their original labels remain searchable.
|
||
|
||
| JS-requested assets | Phase 3 URL treatment |
|
||
|---|---|
|
||
| opfs-worker.js, hash-worker.js, direct-recv-worker.js, p2p-recv-worker.js | Lazy.worker uses embedded per-file URL |
|
||
| Workers' sha256.js, resume-core.js, sha256-wasm.js | First postMessage carries originating build URLs; importScripts uses them |
|
||
| piano-engine.mjs | Dynamic import uses Lazy.url; incremental cache warms it |
|
||
| vendor/framework7-swipe.min.js | Ordered layout group and fallback use Lazy.url |
|
||
| loudness-wasm.js | Remains an eager stamped script; embedded binary needs no separate URL |
|
||
| fonts/fonts.css and font preloads; webmanifest/icons | Plain URLs stay aligned with CSS/manifest references; all belong to manifested cache groups |
|
||
|
||
Legacy pages cannot report their selected layout. Their one-time migration blocks
|
||
all layouts to preserve immediate offline selection; modern pages block only the
|
||
reported layout and its shared controls. Executing groups never re-evaluate live.
|
||
See plans/phase3-staged-apply-decision.md: Phase 4 must add disposal/state handoff.
|
||
|
||
## Phase 4 seams and verification
|
||
|
||
The dependency inventory in `plans/phase4-seams.md` records Phase 3 declarations,
|
||
resolved outside references and byte/coupling rankings. Reproduce its source using
|
||
`npm install --prefix perf/.tmp/analysis --no-audit --no-fund acorn@8 eslint-scope@8`
|
||
and `node perf/analyze-app.cjs cd4a25a` (development scratch dependencies only).
|
||
Shell/shared view/Settings/section-rail definitions remain eager before app.js.
|
||
Presenter, Remote client, Watch party, Share/External and Stats view register
|
||
classic singletons on `window.YT`; synchronous playback hooks remain facades.
|
||
Executing chunks remain pinned: disposal/state handoff is still required before
|
||
any live replacement can be enabled.
|
||
|
||
```bash
|
||
node --test frontend/*.test.js
|
||
(cd server && bun install && bun run test)
|
||
node perf/hidden-timers.mjs
|
||
node perf/seams.mjs --browser all
|
||
node perf/lazy.mjs --browser all
|
||
node perf/migration.mjs --browser all --profile lossy \
|
||
--out perf/results/phase4-migration-lossy-2026-10-08.json
|
||
node perf/migration.mjs --browser all --profile unthrottled \
|
||
--out perf/results/phase4-migration-unthrottled-2026-10-08.json
|
||
node perf/baseline.mjs --runs 1 --browser all --profile lte --scenario all \
|
||
--out perf/results/phase4-final-2026-10-08.json \
|
||
--compare perf/results/after-phase0-2026-10-07.json
|
||
node perf/baseline.mjs --runs 3 --browser chromium --profile lte --scenario cold \
|
||
--out perf/results/phase4-cold-isolated-2026-10-08.json
|
||
```
|
||
|
||
`update-feature` appends one comment to presenter.js in scratch, installs the worker,
|
||
explicitly activates, then waits for the changed background chunk to be verified
|
||
in CacheStorage. It asserts that no other application JS/CSS/font/icon payload
|
||
transfers; derived index/manifest and worker script checks are included in totals.
|
||
`all` includes this scenario when the source contains presenter.js. Cold records
|
||
now include individual long-task timing without changing the aggregate metric,
|
||
plus the source frontend build tag. WebKit does not expose the long-task API.
|
||
Migration supports `--out` to preserve earlier phase evidence. Existing lazy
|
||
harness results are copied to `phase4-lazy-2026-10-08.json` after each Phase 4 run
|
||
and the Phase 3 result is restored. No production requests or deploys are needed.
|
||
|
||
## Phase 5 complete measurements
|
||
|
||
```bash
|
||
node --test perf/measurement.test.mjs
|
||
node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all \
|
||
--out perf/results/final-2026-10-08.json \
|
||
--compare perf/results/baseline-2026-10-07.json
|
||
node perf/lazy.mjs --browser all --out perf/results/phase5-lazy-2026-10-08.json
|
||
```
|
||
|
||
Warm reloads now wait for verified idle asset caching before counters reset and
|
||
assert that no application asset body transfers. CSS update totals now include
|
||
activation and the changed inactive stylesheet's verified background cache fill,
|
||
matching the feature-update boundary. `readyWireBytes`/`readyDuration` retain the
|
||
blocking readiness snapshot separately. Core updates stop at blocking readiness
|
||
because app.js is blocking. Every update asserts that the only application asset
|
||
payload is the edited file; worker script checks/imports and index/manifest count
|
||
as overhead in the total. Historical Phase 3/4 CSS numbers were readiness only.
|
||
No product behavior changes are involved in these measurement corrections.
|
||
|
||
Keep browser/build/test processes idle during full timing measurements. WebKit's
|
||
zero long-task field means unavailable, not zero work. Autoplay-blocked/media-ready
|
||
null results do not establish playback performance or iPhone audio continuity.
|