Add performance baseline harness, throttling proxy, and documentation
This commit is contained in:
Jonathan Sykes
132
perf/README.md
Normal file
132
perf/README.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# Performance Baseline Harness (`perf/`)
|
||||
|
||||
Automated performance baseline and regression harness for the **ytplayer PWA**.
|
||||
|
||||
The harness drives the real Bun backend server (`server/server.js`) serving static Brotli-compressed assets, combined with Playwright browser automation on Chromium and WebKit.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture
|
||||
|
||||
### Backend Server (`server/server.js`)
|
||||
- Runs as an isolated child process using **Bun** on an ephemeral port.
|
||||
- Configured with dedicated scratch directories under `perf/.tmp/` (`DATA_DIR`, `DB_PATH`, `MEDIA_DIR`) so no production or developer data is ever touched.
|
||||
- Employs the real static asset delivery stack (`Brotli` encoding, `ETag` generation, `Cache-Control` immutable caching for version-stamped assets, and `computeBuildTag()`).
|
||||
|
||||
### Browser Automation & Network Throttling
|
||||
- **Chromium**:
|
||||
- Network emulation via Chrome DevTools Protocol (`Network.emulateNetworkConditions`).
|
||||
- CPU throttling at **4× slowdown** (`Emulation.setCPUThrottlingRate`).
|
||||
- Unrestricted audio/video playback via `--autoplay-policy=no-user-gesture-required`.
|
||||
- **WebKit**:
|
||||
- WebKit on Linux lacks CDP network emulation capabilities.
|
||||
- Throttling and stalling are handled by a dedicated local proxy (`perf/proxy.mjs`) sitting between WebKit and the Bun server.
|
||||
- The proxy shapes traffic by pacing chunk delivery through backpressured streams and injecting round-trip latency and stalls.
|
||||
- **Network Profiles**:
|
||||
- **`lte`**: 1.6 Mbps (200 KB/s) download, 750 kbps upload, 150 ms round-trip latency.
|
||||
- **`lossy`**: 1.0 Mbps (125 KB/s) download, 250 ms latency, with intermittent connection stalls (every 7th request delayed by 1,500 ms).
|
||||
|
||||
### Shell Fixture Extractor (`perf/make-shell-fixture.sh`)
|
||||
- Archives the frontend from git commits (`git archive <commit> frontend`) into `perf/fixtures/shell-<tag>/`.
|
||||
- Computes the build tag using the exact sha256 walk algorithm from `server/server.js computeBuildTag()`.
|
||||
- Validates the computed tag against live production (`https://worship.hesed.sbs/api/version`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Test Scenarios
|
||||
|
||||
1. **`cold` (Cold First Load)**:
|
||||
- Fresh browser context with empty cache, no service worker, and clean storage.
|
||||
- Measures initial page fetch, FCP, LCP, DOMContentLoaded, boot milestone (`#cards` render), media playback readiness, service worker precaching duration, and origin storage quota.
|
||||
2. **`warm` (Warm Reload under SW)**:
|
||||
- Re-navigates the page with active Service Worker and primed HTTP cache.
|
||||
- Measures cached asset retrieval efficiency and second-load boot performance.
|
||||
3. **`offline` (Offline Reload)**:
|
||||
- Sets offline state and reloads the application.
|
||||
- Verifies the app boots entirely from `CacheStorage` precached shell assets without network connectivity.
|
||||
4. **`update-js` (Update N → N+1 with 1-line JS comment)**:
|
||||
- Boots client against Tree N (base frontend), establishing an active Service Worker.
|
||||
- Swaps server to Tree N+1 (one comment line appended to `app.js`, altering `BUILD_TAG`).
|
||||
- Triggers `registration.update()` and measures total wire bytes and time until the new Service Worker reaches `waiting` state (`installed`).
|
||||
5. **`update-css` (Update N → N+1 with 1-line CSS rule)**:
|
||||
- Identical to `update-js`, but Tree N+1 has one CSS rule appended to `theme-glass.css`.
|
||||
- Measures the full shell re-download penalty caused by any stylesheet modification.
|
||||
|
||||
---
|
||||
|
||||
## 3. Metrics Collected
|
||||
|
||||
Each scenario records the following metrics across **5 independent runs** (reporting min, max, and median):
|
||||
- **Request Count**: Total HTTP requests made across the wire.
|
||||
- **Wire Bytes**: Total compressed bytes transferred over the network.
|
||||
- **Wire Bytes by Type**: Categorized byte breakdown (`html`, `js`, `css`, `fonts`, `images`, `media`, `api`, `other`).
|
||||
- **FCP (First Contentful Paint)**: Time from navigation start to first rendered content (ms).
|
||||
- **LCP (Largest Contentful Paint)**: Time to largest contentful paint via `PerformanceObserver` (ms).
|
||||
- **DOMContentLoaded**: Time to DOMContentLoaded event end (ms).
|
||||
- **Boot Done**: Time from navigation start until the first `#cards` view rendering is completed (ms).
|
||||
- **Long Tasks Total**: Cumulative execution time of main thread tasks exceeding 50 ms (ms, Chromium).
|
||||
- **SW Install Duration**: Time from Service Worker registration until precaching completes (ms).
|
||||
- **SW Install Wire Bytes**: Network bytes transmitted during Service Worker installation.
|
||||
- **Storage Estimate**: Quota and usage reported by `navigator.storage.estimate()` (bytes).
|
||||
- **Media Ready**: Elapsed time for deterministic media fixture (`benchmedia01`) to reach `readyState >= 3` / playing state. Notes if autoplay policy blocked playback.
|
||||
|
||||
---
|
||||
|
||||
## 4. How to Run
|
||||
|
||||
### Prerequisites
|
||||
Symlink dependencies and ensure server dependencies are installed:
|
||||
```bash
|
||||
# Worktree root
|
||||
ln -s ~/development/personal/ytplayer/node_modules node_modules
|
||||
cd server && bun install && cd ..
|
||||
```
|
||||
|
||||
### Basic Commands
|
||||
Run the complete baseline suite (5 runs, all browsers, LTE profile):
|
||||
```bash
|
||||
node perf/baseline.mjs
|
||||
```
|
||||
|
||||
Run a specific browser or profile:
|
||||
```bash
|
||||
# Chromium on LTE profile
|
||||
node perf/baseline.mjs --browser chromium --profile lte
|
||||
|
||||
# WebKit on LTE profile
|
||||
node perf/baseline.mjs --browser webkit --profile lte
|
||||
|
||||
# Both browsers on lossy profile
|
||||
node perf/baseline.mjs --browser all --profile lossy
|
||||
```
|
||||
|
||||
Run a single scenario:
|
||||
```bash
|
||||
# Cold load only
|
||||
node perf/baseline.mjs --scenario cold
|
||||
|
||||
# Update scenario with 1-line JS change
|
||||
node perf/baseline.mjs --scenario update-js
|
||||
|
||||
# Update scenario with 1-line CSS change
|
||||
node perf/baseline.mjs --scenario update-css
|
||||
```
|
||||
|
||||
Configure run count:
|
||||
```bash
|
||||
# Quick sanity check (1 iteration)
|
||||
node perf/baseline.mjs --runs 1
|
||||
|
||||
# High precision benchmark (7 iterations)
|
||||
node perf/baseline.mjs --runs 7
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 5 Comparison Mode (`--compare`)
|
||||
|
||||
To verify performance gains and detect regressions in later phases:
|
||||
```bash
|
||||
node perf/baseline.mjs --compare perf/results/baseline-2026-10-07.json
|
||||
```
|
||||
Prints an ASCII/Markdown delta comparison table highlighting percentage changes across wire bytes, boot duration, and update overhead.
|
||||
Reference in New Issue
Block a user