josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7bc5b4f8b50d78ece3cca5100d54a97fece2748a
ytplayer/plans/perf-baseline.md
Jonathan Sykes 4bc15a55e3 Add baseline benchmark results and report
2026-10-07 04:25:14 +08:00

176 lines
9.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Performance Baseline Report (Phase 0 / UNCHANGED code)
**Date**: 2026-10-07
**Commit**: `b77938a`
**Dataset**: `perf/results/baseline-2026-10-07.json`
---
## 1. Executive Summary & Headline Baseline Numbers
This baseline captures the performance metrics of the unoptimized ytplayer codebase (commit `b77938a`, matching production tag `1175f1a1d2c1`) across 5 scenarios on both Chromium and WebKit under simulated LTE and Lossy network profiles (median of 5 runs).
### Headline Baseline Numbers
- **Cold First-Load Wire Transfer**:
- **Chromium**: 164 requests, **1,015.0 KB** compressed wire bytes (boot-done median: **10,163 ms** under 4× CPU throttle + LTE).
- **WebKit**: 162 requests, **1,152.0 KB (1.12 MB)** wire bytes (boot-done median: **3,201 ms** under proxy LTE).
- Breakdown: ~477 KB JavaScript, ~397 KB Web Fonts, ~131 KB CSS, ~27 KB HTML.
- **Warm Reload (under Service Worker)**:
- **Chromium**: 4 requests, **5.2 KB** wire bytes (periodic API version poll), boot-done median: **675 ms** (LTE) / **651 ms** (Lossy).
- **WebKit**: 0 requests, **0 KB** wire bytes, boot-done median: **131 ms** (LTE) / **131 ms** (Lossy).
- **Offline Reload (under Service Worker)**:
- **Chromium**: 0 requests, **0 B** wire bytes, reload median: **240 ms**, boot-done median: **219 ms**.
- **WebKit**: 0 requests, **0 B** wire bytes, reload median: **135 ms**, boot-done median: **128 ms**.
- **Update N -> N+1 Wire Cost (The Invalidation Penalty)**:
- **1-line `app.js` change**:
- **Chromium**: 88 requests, **596.6 KB** wire bytes (~611 KB uncompressed/brotli), duration: **2,013 ms** (LTE) / **2,031 ms** (Lossy).
- **WebKit**: 86 requests, **664.3 KB** wire bytes (~680 KB wire), duration: **4,457 ms** (LTE) / **8,135 ms** (Lossy).
- **1-line `theme-glass.css` change**:
- **Chromium**: 88 requests, **596.7 KB** wire bytes, duration: **1,970 ms** (LTE) / **2,004 ms** (Lossy).
- **WebKit**: 86-87 requests, **664.3 KB** wire bytes, duration: **4,510 ms** (LTE) / **8,269 ms** (Lossy).
> **Core Invalidation Finding**: A single 1-line change to *any* static file (`app.js` or `theme-glass.css`) mutates `BUILD_TAG` calculated by `computeBuildTag()`. This causes `sw.js` to re-fetch and re-cache all 86-88 shell assets (~600-664 KB wire payload) rather than just the mutated resource.
---
## 2. Benchmark Harness Execution
The benchmark harness is located in `perf/baseline.mjs` and driven by Playwright with an isolated ephemeral Bun server and a custom pacing proxy (`perf/proxy.mjs`).
### Quick Start / Exact Commands
```bash
# Ensure node_modules is symlinked to root
ln -s ~/development/personal/ytplayer/node_modules node_modules
# Run full baseline suite (all scenarios, both browsers, both profiles, 5 runs)
node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --out perf/results/baseline-2026-10-07.json
# Run individual browser or scenario
node perf/baseline.mjs --runs 5 --browser chromium --profile lte --scenario cold
node perf/baseline.mjs --runs 5 --browser webkit --profile lte --scenario update-js
# Run comparison against baseline after optimizations
node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --compare perf/results/baseline-2026-10-07.json
```
---
## 3. Results by Browser and Profile (Median of 5 Runs)
### Chromium — LTE (1.6 Mbps down, 750 kbps up, 150 ms RTT, 4× CPU throttle)
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|---|---|---|---|---|---|---|---|
| **Cold First Load** | 164 | 1,015.0 KB | 1,540 | 1,896 | 10,163 | 38,446 | 397 |
| **Warm Reload (SW)** | 4 | 5.2 KB | 180 | 684 | 675 | - | 156 |
| **Offline Reload** | 0 | 0.0 KB | 72 | 108 | 219 | 240 | - |
| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,013 | - |
| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 1,970 | - |
- **Storage Estimate**: Quota = 4,298,447,325 B (~4.0 GB), Usage = 3,480,029 B (~3.31 MB).
- **Cold Wire Bytes by Type**: JS 477.2 KB, Fonts 397.1 KB, CSS 130.7 KB, HTML 27.1 KB, Images 6.3 KB, API/Other 1.0 KB.
---
### Chromium — Lossy (1.0 Mbps down, 500 kbps up, 250 ms RTT, 4× CPU throttle)
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|---|---|---|---|---|---|---|---|
| **Cold First Load** | 164 | 1,015.0 KB | 2,344 | 2,912 | 16,015 | 42,518 | 432 |
| **Warm Reload (SW)** | 4 | 5.2 KB | 184 | 644 | 651 | - | 155 |
| **Offline Reload** | 0 | 0.0 KB | 80 | 92 | 237 | 253 | - |
| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,031 | - |
| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 2,004 | - |
---
### WebKit — LTE (1.6 Mbps down, 150 ms RTT via proxy)
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|---|---|---|---|---|---|---|---|
| **Cold First Load** | 162 | 1.12 MB | 1,088 | 1,287 | 3,201 | 30,508 | 0* |
| **Warm Reload (SW)** | 0 | 0.0 KB | 110 | 110 | 131 | - | 0* |
| **Offline Reload** | 0 | 0.0 KB | 107 | 107 | 128 | 135 | - |
| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 4,457 | - |
| **Update N->N+1 (CSS)** | 86 | 664.3 KB | - | - | - | 4,510 | - |
---
### WebKit — Lossy (1.0 Mbps down, 250 ms RTT + 1,500 ms stalls via proxy)
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|---|---|---|---|---|---|---|---|
| **Cold First Load** | 162 | 1.12 MB | 3,557 | 3,735 | 7,303 | 35,449 | 0* |
| **Warm Reload (SW)** | 0 | 0.0 KB | 111 | 112 | 131 | - | 0* |
| **Offline Reload** | 0 | 0.0 KB | 109 | 119 | 129 | 141 | - |
| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 8,135 | - |
| **Update N->N+1 (CSS)** | 87 | 664.4 KB | - | - | - | 8,269 | - |
*\*Note: WebKit does not expose `PerformanceLongTaskTiming` (`longtask` entries), so long task total evaluates to 0 ms.*
---
## 4. Variance & Noise Observed
Over the 5 runs per configuration, measurements demonstrated high stability:
- **Wire Bytes & Request Counts**: Deterministic (0% variance across identical runs).
- **Cold Boot-Done**:
- Chromium LTE: Min 10,129 ms, Max 10,798 ms (<6% variance).
- Chromium Lossy: Min 15,871 ms, Max 16,031 ms (<1% variance).
- WebKit LTE: Min 3,138 ms, Max 3,245 ms (<3% variance).
- WebKit Lossy: Min 6,963 ms, Max 8,710 ms (~20% variance due to proxy random stalls).
- **Warm & Offline Reload**:
- Chromium Offline Reload: Min 238 ms, Max 257 ms (<8% variance).
- WebKit Offline Reload: Min 131 ms, Max 141 ms (<7% variance).
- **Update Duration**:
- Chromium: Runs cluster around ~2,000 ms (2,013 ms median). Occasional outlier runs (~45–200 ms) occur if the browser resolves `reg.update()` immediately before network completion reporting.
- WebKit: Steady scaling between LTE (~4.5 s) and Lossy (~8.2 s) pacing.
---
## 5. Limitations & Caveats
1. **WebKit Network Throttling**:
WebKit lacks Chrome DevTools Protocol (CDP) support for hardware network emulation and CPU throttling. We paced throughput (1.6 Mbps) and injected latency/stalls using a custom local reverse proxy (`perf/proxy.mjs`). This throttles HTTP response chunks truthfully but cannot simulate OS-level socket buffer delays or 4× CPU throttling.
2. **Autoplay Policy in WebKit**:
Playwright WebKit on Linux rejects `--autoplay-policy=no-user-gesture-required` and enforces Safari-style gesture requirements on unmuted audio elements. The media playback benchmark detects this and correctly reports `autoplayBlocked: true` without failing the test run.
3. **WebKit SetOffline Incompatibility**:
Calling Playwright's `context.setOffline(true)` on Linux WebKit causes an unrecoverable engine crash (`WebKit encountered an internal error`). Offline mode for WebKit is simulated by terminating the proxy upstream connection (`proxy.setOffline(true)`).
---
## 6. Manual iPhone Testing Protocol (Phase 0 / iOS Safari PWA)
Because automated headless WebKit cannot fully replicate iOS Safari's mobile background constraints and battery telemetry, the following manual protocol is required:
### Protocol Steps
1. **Install PWA**:
- Open Safari on iPhone and navigate to `https://worship.hesed.sbs`.
- Tap the Share button -> **"Add to Home Screen"**.
- Launch ytplayer directly from the Home Screen icon.
2. **Verify Version & Cache**:
- Verify the footer / version string matches `1175f1a1d2c1` (or current production build tag).
- Turn on Airplane Mode and force-reload to confirm offline readiness. Turn Airplane Mode back off.
3. **30-Minute Screen-Off Battery Test**:
- Note baseline battery percentage (e.g., 85%).
- Select a pre-cached playlist or track and tap **Play**. Ensure continuous audio is heard.
- Lock the screen (display off) and let playback run uninterrupted for **30 minutes**.
- Note battery percentage at 30 minutes (calculate drain %/hr).
- Listen for any audio dropouts, stuttering, or premature process suspension.
4. **Interruption & Resumption Test**:
- Trigger an interruption while locked (e.g., initiate an incoming phone call or start Siri).
- End the interruption and observe if playback automatically resumes via `MediaSession` handlers.
- Test Lock Screen Controls (Play/Pause, Next Track, scrubber).
5. **Log Findings**:
- Record iPhone Model, iOS Version, Battery Drain (% per 30m), Dropouts Count (0 = pass), and Resumption Status (Pass/Fail).
Reference in New Issue View Git Blame Copy Permalink