diff --git a/.gitignore b/.gitignore index c0b2614..c9390e1 100755 --- a/.gitignore +++ b/.gitignore @@ -40,3 +40,7 @@ legacy/releases/ __pycache__/ plans/codex-shots*/ plans/codex-report*.md + +# performance harness artifacts and temp files +perf/fixtures/ +perf/.tmp/ diff --git a/perf/README.md b/perf/README.md new file mode 100644 index 0000000..24b9a32 --- /dev/null +++ b/perf/README.md @@ -0,0 +1,132 @@ +# Performance Baseline Harness (`perf/`) + +Automated performance baseline and regression harness for the **ytplayer PWA**. + +The harness drives the real Bun backend server (`server/server.js`) serving static Brotli-compressed assets, combined with Playwright browser automation on Chromium and WebKit. + +--- + +## 1. Architecture + +### Backend Server (`server/server.js`) +- Runs as an isolated child process using **Bun** on an ephemeral port. +- Configured with dedicated scratch directories under `perf/.tmp/` (`DATA_DIR`, `DB_PATH`, `MEDIA_DIR`) so no production or developer data is ever touched. +- Employs the real static asset delivery stack (`Brotli` encoding, `ETag` generation, `Cache-Control` immutable caching for version-stamped assets, and `computeBuildTag()`). + +### Browser Automation & Network Throttling +- **Chromium**: + - Network emulation via Chrome DevTools Protocol (`Network.emulateNetworkConditions`). + - CPU throttling at **4× slowdown** (`Emulation.setCPUThrottlingRate`). + - Unrestricted audio/video playback via `--autoplay-policy=no-user-gesture-required`. +- **WebKit**: + - WebKit on Linux lacks CDP network emulation capabilities. + - Throttling and stalling are handled by a dedicated local proxy (`perf/proxy.mjs`) sitting between WebKit and the Bun server. + - The proxy shapes traffic by pacing chunk delivery through backpressured streams and injecting round-trip latency and stalls. +- **Network Profiles**: + - **`lte`**: 1.6 Mbps (200 KB/s) download, 750 kbps upload, 150 ms round-trip latency. + - **`lossy`**: 1.0 Mbps (125 KB/s) download, 250 ms latency, with intermittent connection stalls (every 7th request delayed by 1,500 ms). + +### Shell Fixture Extractor (`perf/make-shell-fixture.sh`) +- Archives the frontend from git commits (`git archive frontend`) into `perf/fixtures/shell-/`. +- Computes the build tag using the exact sha256 walk algorithm from `server/server.js computeBuildTag()`. +- Validates the computed tag against live production (`https://worship.hesed.sbs/api/version`). + +--- + +## 2. Test Scenarios + +1. **`cold` (Cold First Load)**: + - Fresh browser context with empty cache, no service worker, and clean storage. + - Measures initial page fetch, FCP, LCP, DOMContentLoaded, boot milestone (`#cards` render), media playback readiness, service worker precaching duration, and origin storage quota. +2. **`warm` (Warm Reload under SW)**: + - Re-navigates the page with active Service Worker and primed HTTP cache. + - Measures cached asset retrieval efficiency and second-load boot performance. +3. **`offline` (Offline Reload)**: + - Sets offline state and reloads the application. + - Verifies the app boots entirely from `CacheStorage` precached shell assets without network connectivity. +4. **`update-js` (Update N → N+1 with 1-line JS comment)**: + - Boots client against Tree N (base frontend), establishing an active Service Worker. + - Swaps server to Tree N+1 (one comment line appended to `app.js`, altering `BUILD_TAG`). + - Triggers `registration.update()` and measures total wire bytes and time until the new Service Worker reaches `waiting` state (`installed`). +5. **`update-css` (Update N → N+1 with 1-line CSS rule)**: + - Identical to `update-js`, but Tree N+1 has one CSS rule appended to `theme-glass.css`. + - Measures the full shell re-download penalty caused by any stylesheet modification. + +--- + +## 3. Metrics Collected + +Each scenario records the following metrics across **5 independent runs** (reporting min, max, and median): +- **Request Count**: Total HTTP requests made across the wire. +- **Wire Bytes**: Total compressed bytes transferred over the network. +- **Wire Bytes by Type**: Categorized byte breakdown (`html`, `js`, `css`, `fonts`, `images`, `media`, `api`, `other`). +- **FCP (First Contentful Paint)**: Time from navigation start to first rendered content (ms). +- **LCP (Largest Contentful Paint)**: Time to largest contentful paint via `PerformanceObserver` (ms). +- **DOMContentLoaded**: Time to DOMContentLoaded event end (ms). +- **Boot Done**: Time from navigation start until the first `#cards` view rendering is completed (ms). +- **Long Tasks Total**: Cumulative execution time of main thread tasks exceeding 50 ms (ms, Chromium). +- **SW Install Duration**: Time from Service Worker registration until precaching completes (ms). +- **SW Install Wire Bytes**: Network bytes transmitted during Service Worker installation. +- **Storage Estimate**: Quota and usage reported by `navigator.storage.estimate()` (bytes). +- **Media Ready**: Elapsed time for deterministic media fixture (`benchmedia01`) to reach `readyState >= 3` / playing state. Notes if autoplay policy blocked playback. + +--- + +## 4. How to Run + +### Prerequisites +Symlink dependencies and ensure server dependencies are installed: +```bash +# Worktree root +ln -s ~/development/personal/ytplayer/node_modules node_modules +cd server && bun install && cd .. +``` + +### Basic Commands +Run the complete baseline suite (5 runs, all browsers, LTE profile): +```bash +node perf/baseline.mjs +``` + +Run a specific browser or profile: +```bash +# Chromium on LTE profile +node perf/baseline.mjs --browser chromium --profile lte + +# WebKit on LTE profile +node perf/baseline.mjs --browser webkit --profile lte + +# Both browsers on lossy profile +node perf/baseline.mjs --browser all --profile lossy +``` + +Run a single scenario: +```bash +# Cold load only +node perf/baseline.mjs --scenario cold + +# Update scenario with 1-line JS change +node perf/baseline.mjs --scenario update-js + +# Update scenario with 1-line CSS change +node perf/baseline.mjs --scenario update-css +``` + +Configure run count: +```bash +# Quick sanity check (1 iteration) +node perf/baseline.mjs --runs 1 + +# High precision benchmark (7 iterations) +node perf/baseline.mjs --runs 7 +``` + +--- + +## 5. Phase 5 Comparison Mode (`--compare`) + +To verify performance gains and detect regressions in later phases: +```bash +node perf/baseline.mjs --compare perf/results/baseline-2026-10-07.json +``` +Prints an ASCII/Markdown delta comparison table highlighting percentage changes across wire bytes, boot duration, and update overhead. diff --git a/perf/baseline.mjs b/perf/baseline.mjs new file mode 100755 index 0000000..54fd48f --- /dev/null +++ b/perf/baseline.mjs @@ -0,0 +1,948 @@ +#!/usr/bin/env node + +/** + * perf/baseline.mjs + * + * Performance baseline harness for ytplayer PWA. + * Drives the real Bun server and runs Playwright (Chromium with CDP throttling + CPU 4x; + * WebKit with a local throttling & stalling proxy). + * + * Scenarios: + * - cold: fresh context, network download, SW install, boot, media playback check + * - warm: re-navigate with active SW and primed HTTP cache + * - offline: context.setOffline(true), reload under SW cache + * - update-js: update N->N+1 (1-line comment appended to app.js) -> wire bytes + time until new SW ready + * - update-css: update N->N+1 (1-line rule appended to theme-glass.css) -> wire bytes + time until new SW ready + * + * Usage: + * node perf/baseline.mjs [options] + * Options: + * --runs Runs per scenario for median calculation (default: 5) + * --browser Browser(s) to test (default: all) + * --profile Network profile (default: lte) + * --scenario Specific scenario or all (default: all) + * --out Output JSON path + * --compare Compare results against a previous baseline JSON + */ + +import { spawn, spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, rmSync, cpSync, appendFileSync, statSync, writeFileSync, readFileSync, symlinkSync, unlinkSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createThrottleProxy } from './proxy.mjs'; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); +const REPO_ROOT = path.resolve(__dirname, '..'); + +// Dynamically resolve playwright from node_modules +let pw; +try { + pw = await import('playwright'); +} catch { + pw = await import(path.join(REPO_ROOT, 'node_modules', 'playwright', 'index.mjs')); +} + +import { createRequire } from 'node:module'; +const req = createRequire(import.meta.url); +const { createClient } = req(path.join(REPO_ROOT, 'server', 'node_modules', '@libsql/client')); + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +function parseArgs() { + const args = process.argv.slice(2); + const options = { + runs: 5, + browser: 'all', // 'chromium', 'webkit', or 'all' + profile: 'lte', // 'lte', 'lossy', or 'all' + scenario: 'all', // 'cold', 'warm', 'offline', 'update-js', 'update-css', or 'all' + out: null, + compare: null, + }; + + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--runs' && i + 1 < args.length) options.runs = parseInt(args[++i], 10); + else if (a === '--browser' && i + 1 < args.length) options.browser = args[++i]; + else if (a === '--profile' && i + 1 < args.length) options.profile = args[++i]; + else if (a === '--scenario' && i + 1 < args.length) options.scenario = args[++i]; + else if (a === '--out' && i + 1 < args.length) options.out = args[++i]; + else if (a === '--compare' && i + 1 < args.length) options.compare = args[++i]; + } + + return options; +} + +function median(values) { + const nums = values.filter((v) => typeof v === 'number' && !Number.isNaN(v)).sort((a, b) => a - b); + if (!nums.length) return null; + const mid = Math.floor(nums.length / 2); + return nums.length % 2 !== 0 ? nums[mid] : Math.round(((nums[mid - 1] + nums[mid]) / 2) * 10) / 10; +} + +function summarizeList(values) { + const valid = values.filter((v) => v !== null && v !== undefined && !Number.isNaN(v)); + if (!valid.length) return { median: null, min: null, max: null, values }; + return { + median: median(valid), + min: Math.min(...valid), + max: Math.max(...valid), + values, + }; +} + +/** + * Setup deterministic local media fixture + */ +function prepareMediaFiles(mediaDir) { + mkdirSync(mediaDir, { recursive: true }); + const mp4 = path.join(mediaDir, 'benchmedia01.1.mp4'); + const m4a = path.join(mediaDir, 'benchmedia01.1.m4a'); + + if (!existsSync(mp4)) { + spawnSync('ffmpeg', [ + '-y', '-f', 'lavfi', '-i', 'testsrc=size=640x360:rate=30:duration=5', + '-f', 'lavfi', '-i', 'sine=frequency=440:duration=5', + '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', + '-c:a', 'aac', '-movflags', '+faststart', '-shortest', mp4, + ]); + } + + if (!existsSync(m4a)) { + spawnSync('ffmpeg', [ + '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=5', + '-c:a', 'aac', '-shortest', m4a, + ]); + } + + return { mp4, m4a }; +} + +async function seedDatabase(dbPath, mediaFiles) { + const db = createClient({ url: 'file:' + dbPath }); + const mp4Size = statSync(mediaFiles.mp4).size; + const m4aSize = statSync(mediaFiles.m4a).size; + + await db.execute({ + sql: `INSERT OR REPLACE INTO media_cache ( + video_id, status, gen, size, height, vcodec, acodec, duration, meta, last_access + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + args: [ + 'benchmedia01', + 'ready', + 1, + mp4Size + m4aSize, + 360, + 'h264', + 'aac', + 5.0, + JSON.stringify({ title: 'Baseline Deterministic Clip', channel: 'Perf Benchmark' }), + Date.now(), + ], + }); +} + +/** + * Bun Server Manager + */ +class ServerManager { + constructor(scratchDir, publicDir) { + this.scratchDir = scratchDir; + this.publicDir = publicDir; + this.dataDir = path.join(scratchDir, 'data'); + this.mediaDir = path.join(this.dataDir, 'media'); + this.dbPath = path.join(this.dataDir, 'ytplayer.db'); + this.serverProcess = null; + this.port = 0; + } + + async start() { + mkdirSync(this.dataDir, { recursive: true }); + const fixtureDir = path.join(REPO_ROOT, 'perf', 'fixtures', 'media'); + const mediaFiles = prepareMediaFiles(fixtureDir); + + // Create a dedicated server working dir with public symlink + const srvCwd = path.join(this.scratchDir, 'srv'); + mkdirSync(srvCwd, { recursive: true }); + const publicLink = path.join(srvCwd, 'public'); + if (existsSync(publicLink)) unlinkSync(publicLink); + symlinkSync(this.publicDir, publicLink); + + // Pick random available ephemeral port once and reuse across restarts + this.port = this.port || (4300 + Math.floor(Math.random() * 2000)); + + const env = { + ...process.env, + PORT: String(this.port), + DATA_DIR: this.dataDir, + MEDIA_DIR: this.mediaDir, + DB_PATH: this.dbPath, + }; + + const serverScript = path.join(REPO_ROOT, 'server', 'server.js'); + this.serverProcess = spawn('bun', [serverScript], { + cwd: srvCwd, + env, + stdio: 'ignore', + }); + + // Wait for server readiness + let ready = false; + for (let i = 0; i < 50; i++) { + try { + const res = await fetch(`http://127.0.0.1:${this.port}/api/version`); + if (res.ok) { + ready = true; + break; + } + } catch {} + await sleep(100); + } + + if (!ready) { + throw new Error(`Server failed to start on port ${this.port}`); + } + + // Copy media files to mediaDir and seed media table AFTER server startup sweep + mkdirSync(this.mediaDir, { recursive: true }); + const targetMp4 = path.join(this.mediaDir, 'benchmedia01.1.mp4'); + const targetM4a = path.join(this.mediaDir, 'benchmedia01.1.m4a'); + cpSync(mediaFiles.mp4, targetMp4); + cpSync(mediaFiles.m4a, targetM4a); + await seedDatabase(this.dbPath, { mp4: targetMp4, m4a: targetM4a }); + } + + async stop() { + if (this.serverProcess) { + this.serverProcess.kill('SIGKILL'); + this.serverProcess = null; + await sleep(100); + } + } + + async restartWithPublic(newPublicDir) { + await this.stop(); + this.publicDir = newPublicDir; + await this.start(); + } +} + +/** + * Configure page for metrics collection + */ +async function setupPageMetrics(page) { + await page.addInitScript(() => { + window.__bootDone = null; + window.__fcp = null; + window.__lcp = null; + window.__longTasksTotal = 0; + + // Observe boot completion landmark (#cards view set by renderList()) + const markBoot = () => { + const cards = document.getElementById('cards'); + if (cards && cards.dataset && cards.dataset.view) { + if (!window.__bootDone) window.__bootDone = performance.now(); + return true; + } + return false; + }; + const t = setInterval(() => { + if (markBoot()) clearInterval(t); + }, 5); + + // Observe Long Tasks API (supported on Chromium) + try { + new PerformanceObserver((list) => { + for (const entry of list.getEntries()) { + window.__longTasksTotal += entry.duration; + } + }).observe({ type: 'longtask', buffered: true }); + } catch {} + + // Observe LCP + try { + new PerformanceObserver((list) => { + const entries = list.getEntries(); + if (entries.length > 0) { + window.__lcp = entries[entries.length - 1].startTime; + } + }).observe({ type: 'largest-contentful-paint', buffered: true }); + } catch {} + }); +} + +/** + * Run Scenario: Cold Load + */ +async function runColdScenario({ browserType, server, proxy, profile }) { + const isWebKit = browserType.name() === 'webkit'; + const launchArgs = isWebKit ? [] : ['--autoplay-policy=no-user-gesture-required']; + const browser = await browserType.launch({ + headless: true, + args: launchArgs, + }); + + const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } }; + const context = await browser.newContext(contextOptions); + const page = await context.newPage(); + + // If Chromium, configure CDP Network conditions and 4x CPU throttle + if (!isWebKit) { + const cdp = await context.newCDPSession(page); + await cdp.send('Network.enable'); + const isLossy = profile === 'lossy'; + await cdp.send('Network.emulateNetworkConditions', { + offline: false, + latency: isLossy ? 250 : 150, + downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8, + uploadThroughput: (750 * 1000) / 8, + connectionType: 'cellular4g', + }); + await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 }); + } + + await setupPageMetrics(page); + + proxy.resetStats(); + proxy.startTracking(); + + const swInstallStart = Date.now(); + const navUrl = `http://127.0.0.1:${server.port}/?v=benchmedia01`; + + await page.goto(navUrl); + + // Wait for boot landmark + await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 }); + + // Check deterministic media playing-ready + let mediaReadyMs = null; + let autoplayBlocked = false; + try { + const mediaCheck = await page.waitForFunction(() => { + const v = document.querySelector('video') || document.querySelector('audio'); + if (!v) return false; + return v.readyState >= 3 || (!v.paused && v.currentTime > 0); + }, { timeout: 8000 }); + if (mediaCheck) { + const state = await page.evaluate(() => { + const v = document.querySelector('video') || document.querySelector('audio'); + return { paused: v.paused, readyState: v.readyState }; + }); + autoplayBlocked = state.paused; + mediaReadyMs = await page.evaluate(() => performance.now()); + } + } catch { + autoplayBlocked = true; + } + + // Wait for initial Service Worker activation + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + if (reg.active && reg.active.state === 'activated') return; + await new Promise((resolve) => { + const w = reg.active || reg.installing; + if (w.state === 'activated') return resolve(); + w.addEventListener('statechange', () => { + if (w.state === 'activated') resolve(); + }); + }); + }); + + const swInstallDuration = Date.now() - swInstallStart; + proxy.stopTracking(); + + const stats = proxy.getStats(); + + const pageMetrics = await page.evaluate(() => { + const nav = performance.getEntriesByType('navigation')[0]; + const paint = performance.getEntriesByType('paint'); + const fcp = paint.find((p) => p.name === 'first-contentful-paint'); + + return { + fcp: fcp ? Math.round(fcp.startTime) : null, + lcp: window.__lcp ? Math.round(window.__lcp) : null, + domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null, + bootDone: window.__bootDone ? Math.round(window.__bootDone) : null, + longTaskTotal: Math.round(window.__longTasksTotal || 0), + }; + }); + + const storage = await page.evaluate(async () => { + if (navigator.storage && navigator.storage.estimate) { + const est = await navigator.storage.estimate(); + return { usage: est.usage || 0, quota: est.quota || 0 }; + } + return { usage: null, quota: null }; + }); + + await browser.close(); + + return { + requestCount: stats.requestCount, + wireBytes: stats.totalWireBytes, + wireBytesByType: stats.bytesByType, + fcp: pageMetrics.fcp, + lcp: pageMetrics.lcp, + domContentLoaded: pageMetrics.domContentLoaded, + bootDone: pageMetrics.bootDone, + longTaskTotal: pageMetrics.longTaskTotal, + swInstallDuration, + swInstallBytes: stats.totalWireBytes, + storageEstimate: storage, + mediaReady: mediaReadyMs ? Math.round(mediaReadyMs) : null, + autoplayBlocked, + }; +} + +/** + * Run Scenario: Warm Load (re-navigation with active SW) + */ +async function runWarmScenario({ browserType, server, proxy, profile }) { + const isWebKit = browserType.name() === 'webkit'; + const launchArgs = isWebKit ? [] : ['--autoplay-policy=no-user-gesture-required']; + const browser = await browserType.launch({ + headless: true, + args: launchArgs, + }); + + const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } }; + const context = await browser.newContext(contextOptions); + const page = await context.newPage(); + + if (!isWebKit) { + const cdp = await context.newCDPSession(page); + await cdp.send('Network.enable'); + const isLossy = profile === 'lossy'; + await cdp.send('Network.emulateNetworkConditions', { + offline: false, + latency: isLossy ? 250 : 150, + downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8, + uploadThroughput: (750 * 1000) / 8, + connectionType: 'cellular4g', + }); + await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 }); + } + + await setupPageMetrics(page); + + const navUrl = `http://127.0.0.1:${server.port}/`; + + // Prime context with cold load + await page.goto(navUrl); + await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 }); + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + if (reg.active && reg.active.state === 'activated') return; + await new Promise((resolve) => { + const w = reg.active || reg.installing; + if (w.state === 'activated') return resolve(); + w.addEventListener('statechange', () => { + if (w.state === 'activated') resolve(); + }); + }); + }); + + // Now measure warm re-navigation + proxy.resetStats(); + proxy.startTracking(); + + await page.goto(navUrl); + await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 }); + + proxy.stopTracking(); + const stats = proxy.getStats(); + + const pageMetrics = await page.evaluate(() => { + const nav = performance.getEntriesByType('navigation')[0]; + const paint = performance.getEntriesByType('paint'); + const fcp = paint.find((p) => p.name === 'first-contentful-paint'); + + return { + fcp: fcp ? Math.round(fcp.startTime) : null, + lcp: window.__lcp ? Math.round(window.__lcp) : null, + domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null, + bootDone: window.__bootDone ? Math.round(window.__bootDone) : null, + longTaskTotal: Math.round(window.__longTasksTotal || 0), + }; + }); + + await browser.close(); + + return { + requestCount: stats.requestCount, + wireBytes: stats.totalWireBytes, + wireBytesByType: stats.bytesByType, + fcp: pageMetrics.fcp, + lcp: pageMetrics.lcp, + domContentLoaded: pageMetrics.domContentLoaded, + bootDone: pageMetrics.bootDone, + longTaskTotal: pageMetrics.longTaskTotal, + }; +} + +/** + * Run Scenario: Offline Reload under SW + */ +async function runOfflineScenario({ browserType, server, proxy, profile }) { + const browser = await browserType.launch({ headless: true }); + const isWebKit = browserType.name() === 'webkit'; + const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } }; + const context = await browser.newContext(contextOptions); + const page = await context.newPage(); + + await setupPageMetrics(page); + const navUrl = `http://127.0.0.1:${server.port}/`; + + // Prime SW + await page.goto(navUrl); + await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 }); + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + if (reg.active && reg.active.state === 'activated') return; + await new Promise((resolve) => { + const w = reg.active || reg.installing; + if (w.state === 'activated') return resolve(); + w.addEventListener('statechange', () => { + if (w.state === 'activated') resolve(); + }); + }); + }); + + // Switch to offline mode: + // On Chromium, context.setOffline(true) works. + // On Linux WebKit, Playwright context.setOffline throws an internal browser error, + // so proxy.setOffline(true) cuts upstream network to simulate offline truthfully. + if (!isWebKit) { + await context.setOffline(true); + } + proxy.setOffline(true); + + proxy.resetStats(); + proxy.startTracking(); + + const t0 = Date.now(); + await page.reload(); + await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 }); + const reloadDuration = Date.now() - t0; + + proxy.stopTracking(); + proxy.setOffline(false); + const stats = proxy.getStats(); + + const pageMetrics = await page.evaluate(() => { + const nav = performance.getEntriesByType('navigation')[0]; + const paint = performance.getEntriesByType('paint'); + const fcp = paint.find((p) => p.name === 'first-contentful-paint'); + + return { + fcp: fcp ? Math.round(fcp.startTime) : null, + lcp: window.__lcp ? Math.round(window.__lcp) : null, + domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null, + bootDone: window.__bootDone ? Math.round(window.__bootDone) : null, + }; + }); + + await browser.close(); + + return { + requestCount: stats.requestCount, + wireBytes: stats.totalWireBytes, + fcp: pageMetrics.fcp, + lcp: pageMetrics.lcp, + domContentLoaded: pageMetrics.domContentLoaded, + bootDone: pageMetrics.bootDone, + reloadDuration, + }; +} + +/** + * Run Scenario: Update N -> N+1 + */ +async function runUpdateScenario({ browserType, server, proxy, profile, updateKind, trees }) { + const browser = await browserType.launch({ headless: true }); + const isWebKit = browserType.name() === 'webkit'; + const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } }; + const context = await browser.newContext(contextOptions); + const page = await context.newPage(); + + if (!isWebKit) { + const cdp = await context.newCDPSession(page); + await cdp.send('Network.enable'); + const isLossy = profile === 'lossy'; + await cdp.send('Network.emulateNetworkConditions', { + offline: false, + latency: isLossy ? 250 : 150, + downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8, + uploadThroughput: (750 * 1000) / 8, + connectionType: 'cellular4g', + }); + } + + // Start on Tree N (base) + await server.restartWithPublic(trees.treeBase); + + const navUrl = `http://127.0.0.1:${server.port}/`; + + await page.goto(navUrl); + + // Wait for steady state: active worker present, no waiting worker + while (true) { + const ready = await page.evaluate(async () => { + const reg = await navigator.serviceWorker.getRegistration(); + return reg?.active?.state === 'activated' && !reg?.installing && !reg?.waiting; + }); + if (ready) break; + await sleep(50); + } + + // Switch server to Tree N+1 (app.js or theme-glass.css modification) + const targetTree = updateKind === 'js' ? trees.treeUpdateJs : trees.treeUpdateCss; + await server.restartWithPublic(targetTree); + + // Measure wire bytes and duration from reg.update() until new SW is waiting/installed + proxy.resetStats(); + proxy.startTracking(); + + const t0 = Date.now(); + await page.evaluate(() => navigator.serviceWorker.getRegistration().then((r) => r.update())); + + // Wait until new worker reaches waiting state + while (true) { + const waiting = await page.evaluate(async () => { + const reg = await navigator.serviceWorker.getRegistration(); + return reg?.waiting?.state === 'installed'; + }); + if (waiting) break; + await sleep(50); + } + + const duration = Date.now() - t0; + proxy.stopTracking(); + const stats = proxy.getStats(); + + await browser.close(); + + return { + updateKind, + duration, + requestCount: stats.requestCount, + wireBytes: stats.totalWireBytes, + wireBytesByType: stats.bytesByType, + }; +} + +/** + * Main execution + */ +async function main() { + const options = parseArgs(); + console.log('=== YTPlayer Performance Baseline Harness ==='); + console.log(`Runs per scenario: ${options.runs}`); + console.log(`Browser target: ${options.browser}`); + console.log(`Network profile: ${options.profile}`); + console.log(`Scenario target: ${options.scenario}`); + + const browsers = []; + if (options.browser === 'all' || options.browser === 'chromium') browsers.push(pw.chromium); + if (options.browser === 'all' || options.browser === 'webkit') browsers.push(pw.webkit); + + const profiles = []; + if (options.profile === 'all' || options.profile === 'lte') profiles.push('lte'); + if (options.profile === 'all' || options.profile === 'lossy') profiles.push('lossy'); + + const tmpRoot = path.join(REPO_ROOT, 'perf', '.tmp', `bench-${Date.now()}`); + mkdirSync(tmpRoot, { recursive: true }); + + // Setup scratch trees for N -> N+1 update scenarios + const treeBase = path.join(tmpRoot, 'tree-base'); + const treeUpdateJs = path.join(tmpRoot, 'tree-update-js'); + const treeUpdateCss = path.join(tmpRoot, 'tree-update-css'); + + cpSync(path.join(REPO_ROOT, 'frontend'), treeBase, { recursive: true }); + cpSync(path.join(REPO_ROOT, 'frontend'), treeUpdateJs, { recursive: true }); + cpSync(path.join(REPO_ROOT, 'frontend'), treeUpdateCss, { recursive: true }); + + // Append 1 comment line to app.js in treeUpdateJs + appendFileSync(path.join(treeUpdateJs, 'app.js'), '\n// baseline bench: 1-line app.js comment\n'); + // Append 1 rule to theme-glass.css in treeUpdateCss + appendFileSync(path.join(treeUpdateCss, 'theme-glass.css'), '\n/* baseline bench: 1-line css rule */\n.baseline-marker { opacity: 0.99; }\n'); + + const trees = { treeBase, treeUpdateJs, treeUpdateCss }; + + const serverScratch = path.join(tmpRoot, 'server-run'); + const server = new ServerManager(serverScratch, treeBase); + await server.start(); + console.log(`Bun server running on ephemeral port ${server.port}`); + + const results = { + date: new Date().toISOString(), + commit: spawnSync('git', ['rev-parse', 'HEAD'], { cwd: REPO_ROOT }).stdout.toString().trim(), + runsConfigured: options.runs, + browsers: {}, + }; + + try { + for (const bType of browsers) { + const bName = bType.name(); + results.browsers[bName] = {}; + + for (const prof of profiles) { + console.log(`\n>>> Running Benchmark: Browser=[${bName}] Profile=[${prof}]`); + results.browsers[bName][prof] = {}; + + const proxyProfile = bName === 'chromium' ? 'unthrottled' : prof; + const proxy = createThrottleProxy({ targetPort: server.port, profile: proxyProfile }); + proxy.port = await proxy.listen(); + console.log(`Proxy listening on port ${proxy.port} (mode: ${proxyProfile})`); + + // 1. Cold Scenario + if (options.scenario === 'all' || options.scenario === 'cold') { + console.log(` [Scenario: Cold Load] running ${options.runs} iterations...`); + const coldRuns = []; + for (let r = 0; r < options.runs; r++) { + process.stdout.write(` run ${r + 1}/${options.runs}... `); + const res = await runColdScenario({ browserType: bType, server, proxy, profile: prof }); + coldRuns.push(res); + console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, boot=${res.bootDone}ms)`); + } + + results.browsers[bName][prof].cold = { + requestCount: summarizeList(coldRuns.map((r) => r.requestCount)), + wireBytes: summarizeList(coldRuns.map((r) => r.wireBytes)), + fcp: summarizeList(coldRuns.map((r) => r.fcp)), + lcp: summarizeList(coldRuns.map((r) => r.lcp)), + domContentLoaded: summarizeList(coldRuns.map((r) => r.domContentLoaded)), + bootDone: summarizeList(coldRuns.map((r) => r.bootDone)), + longTaskTotal: summarizeList(coldRuns.map((r) => r.longTaskTotal)), + swInstallDuration: summarizeList(coldRuns.map((r) => r.swInstallDuration)), + swInstallBytes: summarizeList(coldRuns.map((r) => r.swInstallBytes)), + storageEstimate: coldRuns[0]?.storageEstimate || null, + mediaReady: summarizeList(coldRuns.map((r) => r.mediaReady)), + autoplayBlocked: coldRuns.some((r) => r.autoplayBlocked), + wireBytesByType: coldRuns[0]?.wireBytesByType || {}, + }; + } + + // 2. Warm Scenario + if (options.scenario === 'all' || options.scenario === 'warm') { + console.log(` [Scenario: Warm Load] running ${options.runs} iterations...`); + const warmRuns = []; + for (let r = 0; r < options.runs; r++) { + process.stdout.write(` run ${r + 1}/${options.runs}... `); + const res = await runWarmScenario({ browserType: bType, server, proxy, profile: prof }); + warmRuns.push(res); + console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, boot=${res.bootDone}ms)`); + } + + results.browsers[bName][prof].warm = { + requestCount: summarizeList(warmRuns.map((r) => r.requestCount)), + wireBytes: summarizeList(warmRuns.map((r) => r.wireBytes)), + fcp: summarizeList(warmRuns.map((r) => r.fcp)), + lcp: summarizeList(warmRuns.map((r) => r.lcp)), + domContentLoaded: summarizeList(warmRuns.map((r) => r.domContentLoaded)), + bootDone: summarizeList(warmRuns.map((r) => r.bootDone)), + longTaskTotal: summarizeList(warmRuns.map((r) => r.longTaskTotal)), + }; + } + + // 3. Offline Scenario + if (options.scenario === 'all' || options.scenario === 'offline') { + console.log(` [Scenario: Offline Reload] running ${options.runs} iterations...`); + const offRuns = []; + for (let r = 0; r < options.runs; r++) { + process.stdout.write(` run ${r + 1}/${options.runs}... `); + const res = await runOfflineScenario({ browserType: bType, server, proxy, profile: prof }); + offRuns.push(res); + console.log(`done (reload=${res.reloadDuration}ms, boot=${res.bootDone}ms)`); + } + + results.browsers[bName][prof].offline = { + requestCount: summarizeList(offRuns.map((r) => r.requestCount)), + wireBytes: summarizeList(offRuns.map((r) => r.wireBytes)), + fcp: summarizeList(offRuns.map((r) => r.fcp)), + lcp: summarizeList(offRuns.map((r) => r.lcp)), + domContentLoaded: summarizeList(offRuns.map((r) => r.domContentLoaded)), + bootDone: summarizeList(offRuns.map((r) => r.bootDone)), + reloadDuration: summarizeList(offRuns.map((r) => r.reloadDuration)), + }; + } + + // 4. Update N -> N+1 (JS) + if (options.scenario === 'all' || options.scenario === 'update-js') { + console.log(` [Scenario: Update N->N+1 (1-line app.js)] running ${options.runs} iterations...`); + const updJsRuns = []; + for (let r = 0; r < options.runs; r++) { + process.stdout.write(` run ${r + 1}/${options.runs}... `); + const res = await runUpdateScenario({ + browserType: bType, + server, + proxy, + profile: prof, + updateKind: 'js', + trees, + }); + updJsRuns.push(res); + console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, dur=${res.duration}ms)`); + } + + results.browsers[bName][prof].updateJs = { + requestCount: summarizeList(updJsRuns.map((r) => r.requestCount)), + wireBytes: summarizeList(updJsRuns.map((r) => r.wireBytes)), + duration: summarizeList(updJsRuns.map((r) => r.duration)), + wireBytesByType: updJsRuns[0]?.wireBytesByType || {}, + }; + } + + // 5. Update N -> N+1 (CSS) + if (options.scenario === 'all' || options.scenario === 'update-css') { + console.log(` [Scenario: Update N->N+1 (1-line theme-glass.css)] running ${options.runs} iterations...`); + const updCssRuns = []; + for (let r = 0; r < options.runs; r++) { + process.stdout.write(` run ${r + 1}/${options.runs}... `); + const res = await runUpdateScenario({ + browserType: bType, + server, + proxy, + profile: prof, + updateKind: 'css', + trees, + }); + updCssRuns.push(res); + console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, dur=${res.duration}ms)`); + } + + results.browsers[bName][prof].updateCss = { + requestCount: summarizeList(updCssRuns.map((r) => r.requestCount)), + wireBytes: summarizeList(updCssRuns.map((r) => r.wireBytes)), + duration: summarizeList(updCssRuns.map((r) => r.duration)), + wireBytesByType: updCssRuns[0]?.wireBytesByType || {}, + }; + } + + await proxy.close(); + } + } + } finally { + await server.stop(); + rmSync(tmpRoot, { recursive: true, force: true }); + } + + // Write results JSON + const now = new Date(); + const dateStr = `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, '0')}-${String(now.getDate()).padStart(2, '0')}`; + const outPath = options.out || path.join(REPO_ROOT, 'perf', 'results', `baseline-${dateStr}.json`); + mkdirSync(path.dirname(outPath), { recursive: true }); + writeFileSync(outPath, JSON.stringify(results, null, 2)); + console.log(`\nBaseline results saved to: ${outPath}`); + + // Print Summary Table + printSummaryTable(results); + + // Compare mode if requested + if (options.compare) { + compareResults(options.compare, results); + } +} + +function printSummaryTable(results) { + console.log('\n============================= HEADLINE BASELINE SUMMARY ============================='); + for (const [browser, profiles] of Object.entries(results.browsers)) { + for (const [profile, scenarios] of Object.entries(profiles)) { + console.log(`\n[Browser: ${browser} | Profile: ${profile}]`); + console.log('| Scenario | Req Count | Wire Bytes | Boot / Ready | SW Dur / Reload |'); + console.log('|---------------------------|-----------|------------|--------------|-----------------|'); + + if (scenarios.cold) { + const c = scenarios.cold; + console.log( + `| Cold First Load | ${String(c.requestCount.median).padEnd(9)} | ${formatBytes(c.wireBytes.median).padEnd(10)} | ${String(c.bootDone.median + 'ms').padEnd(12)} | ${String(c.swInstallDuration.median + 'ms').padEnd(15)} |` + ); + } + if (scenarios.warm) { + const w = scenarios.warm; + console.log( + `| Warm Reload (under SW) | ${String(w.requestCount.median).padEnd(9)} | ${formatBytes(w.wireBytes.median).padEnd(10)} | ${String(w.bootDone.median + 'ms').padEnd(12)} | ${'-'.padEnd(15)} |` + ); + } + if (scenarios.offline) { + const o = scenarios.offline; + console.log( + `| Offline Reload (under SW) | ${String(o.requestCount.median).padEnd(9)} | ${formatBytes(o.wireBytes.median).padEnd(10)} | ${String(o.bootDone.median + 'ms').padEnd(12)} | ${String(o.reloadDuration.median + 'ms').padEnd(15)} |` + ); + } + if (scenarios.updateJs) { + const u = scenarios.updateJs; + console.log( + `| Update N->N+1 (1-line JS) | ${String(u.requestCount.median).padEnd(9)} | ${formatBytes(u.wireBytes.median).padEnd(10)} | ${'-'.padEnd(12)} | ${String(u.duration.median + 'ms').padEnd(15)} |` + ); + } + if (scenarios.updateCss) { + const u = scenarios.updateCss; + console.log( + `| Update N->N+1 (1-line CSS)| ${String(u.requestCount.median).padEnd(9)} | ${formatBytes(u.wireBytes.median).padEnd(10)} | ${'-'.padEnd(12)} | ${String(u.duration.median + 'ms').padEnd(15)} |` + ); + } + } + } + console.log('\n====================================================================================='); +} + +function formatBytes(bytes) { + if (bytes === null || bytes === undefined) return 'N/A'; + if (bytes >= 1024 * 1024) return (bytes / (1024 * 1024)).toFixed(2) + ' MB'; + if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB'; + return bytes + ' B'; +} + +function compareResults(baselinePath, currentResults) { + console.log(`\n================ Phase 5 Comparison against: ${baselinePath} ================`); + if (!existsSync(baselinePath)) { + console.error(`Error: Baseline file not found: ${baselinePath}`); + return; + } + const base = JSON.parse(readFileSync(baselinePath, 'utf8')); + + for (const [browser, profiles] of Object.entries(currentResults.browsers)) { + for (const [profile, scenarios] of Object.entries(profiles)) { + console.log(`\n[Comparison: ${browser} / ${profile}]`); + console.log('| Metric | Baseline | Current | Delta (%) |'); + console.log('|------------------------------------|-------------|-------------|--------------|'); + + const baseScen = base.browsers?.[browser]?.[profile]; + if (!baseScen) { + console.log('| (No matching baseline data) | | | |'); + continue; + } + + const rows = [ + ['Cold: Wire Bytes', baseScen.cold?.wireBytes?.median, scenarios.cold?.wireBytes?.median, true], + ['Cold: Boot Done (ms)', baseScen.cold?.bootDone?.median, scenarios.cold?.bootDone?.median, false], + ['Cold: Requests', baseScen.cold?.requestCount?.median, scenarios.cold?.requestCount?.median, false], + ['Warm: Boot Done (ms)', baseScen.warm?.bootDone?.median, scenarios.warm?.bootDone?.median, false], + ['Update (1-line JS): Wire Bytes', baseScen.updateJs?.wireBytes?.median, scenarios.updateJs?.wireBytes?.median, true], + ['Update (1-line JS): Time (ms)', baseScen.updateJs?.duration?.median, scenarios.updateJs?.duration?.median, false], + ['Update (1-line CSS): Wire Bytes', baseScen.updateCss?.wireBytes?.median, scenarios.updateCss?.wireBytes?.median, true], + ['Update (1-line CSS): Time (ms)', baseScen.updateCss?.duration?.median, scenarios.updateCss?.duration?.median, false], + ]; + + for (const [name, bVal, cVal, isBytes] of rows) { + if (bVal === undefined || cVal === undefined) continue; + const bStr = isBytes ? formatBytes(bVal) : bVal !== null ? `${bVal}` : 'N/A'; + const cStr = isBytes ? formatBytes(cVal) : cVal !== null ? `${cVal}` : 'N/A'; + let deltaStr = '-'; + if (typeof bVal === 'number' && typeof cVal === 'number' && bVal > 0) { + const pct = (((cVal - bVal) / bVal) * 100).toFixed(1); + const sign = cVal >= bVal ? '+' : ''; + deltaStr = `${sign}${pct}%`; + } + console.log(`| ${name.padEnd(34)} | ${bStr.padEnd(11)} | ${cStr.padEnd(11)} | ${deltaStr.padEnd(12)} |`); + } + } + } + console.log('=============================================================================='); +} + +main().catch((err) => { + console.error('Benchmark fatal error:', err); + process.exit(1); +}); diff --git a/perf/make-shell-fixture.sh b/perf/make-shell-fixture.sh new file mode 100755 index 0000000..93553e3 --- /dev/null +++ b/perf/make-shell-fixture.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +set -euo pipefail + +# ============================================================================== +# perf/make-shell-fixture.sh +# Archives frontend from into perf/fixtures/shell-/, +# computes the build tag the way server.js computeBuildTag() does, +# and verifies that it matches expected-tag (or live prod tag). +# ============================================================================== + +if [ $# -lt 1 ]; then + echo "Usage: $0 [expected-tag]" >&2 + exit 1 +fi + +COMMIT="$1" +EXPECTED_TAG="${2:-}" + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +PERF_DIR="$REPO_ROOT/perf" +FIXTURES_DIR="$PERF_DIR/fixtures" +TMP_DIR="$PERF_DIR/.tmp" + +mkdir -p "$FIXTURES_DIR" "$TMP_DIR" + +PROD_URL="https://worship.hesed.sbs/api/version" + +fetch_prod_tag() { + curl -s -S --max-time 10 "$PROD_URL" | node -e ' + let data = ""; + process.stdin.on("data", chunk => { data += chunk; }); + process.stdin.on("end", () => { + try { + const j = JSON.parse(data); + if (j.buildTag) { + process.stdout.write(j.buildTag); + process.exit(0); + } + } catch {} + process.exit(1); + }); + ' 2>/dev/null || echo "" +} + +# Verify the commit exists in git +if ! git rev-parse --verify "$COMMIT^{commit}" >/dev/null 2>&1; then + echo "Error: commit '$COMMIT' not found in git repository" >&2 + exit 1 +fi + +RESOLVED_COMMIT="$(git rev-parse --short=7 "$COMMIT")" +echo "Extracting frontend from commit $RESOLVED_COMMIT..." + +STAGING_DIR="$(mktemp -d "$TMP_DIR/fixture-staging-XXXXXX")" +cleanup() { + rm -rf "$STAGING_DIR" +} +trap cleanup EXIT + +# Archive frontend tree into staging dir +git archive "$COMMIT" frontend | tar -x -C "$STAGING_DIR" --strip-components=1 + +# Compute build tag using the exact logic from server/server.js computeBuildTag() +COMPUTED_TAG=$(node -e ' +const { createHash } = require("crypto"); +const { readdirSync, statSync, readFileSync } = require("fs"); + +function computeTag(dir) { + const hash = createHash("sha256"); + const walk = (d, prefix) => { + for (const name of readdirSync(d).sort()) { + const full = `${d}/${name}`; + const rel = `${prefix}/${name}`; + if (statSync(full).isDirectory()) { + walk(full, rel); + } else { + hash.update(rel); + hash.update(readFileSync(full)); + } + } + }; + walk(dir, "./public"); + return hash.digest("hex").slice(0, 12); +} + +const tag = computeTag(process.argv[1]); +process.stdout.write(tag); +' "$STAGING_DIR") + +echo "Computed build tag for $RESOLVED_COMMIT: $COMPUTED_TAG" + +# Determine target expected tag +if [ -z "$EXPECTED_TAG" ]; then + echo "Querying live production tag from $PROD_URL..." + EXPECTED_TAG="$(fetch_prod_tag)" + if [ -z "$EXPECTED_TAG" ]; then + echo "Error: failed to fetch live buildTag from $PROD_URL" >&2 + exit 1 + fi + echo "Live production build tag: $EXPECTED_TAG" + + # If mismatch, deploy might be in progress: wait 5 minutes and retry once + if [ "$COMPUTED_TAG" != "$EXPECTED_TAG" ]; then + echo "Tag mismatch detected (computed=$COMPUTED_TAG vs prod=$EXPECTED_TAG)." + echo "Deploy may be in progress. Waiting 300 seconds to retry once..." + sleep 300 + EXPECTED_TAG="$(fetch_prod_tag)" + echo "Retried production build tag: $EXPECTED_TAG" + fi +fi + +if [ "$COMPUTED_TAG" != "$EXPECTED_TAG" ]; then + echo "Error: build tag mismatch! Computed '$COMPUTED_TAG' does not match expected '$EXPECTED_TAG'" >&2 + exit 1 +fi + +TARGET_FIXTURE_DIR="$FIXTURES_DIR/shell-$COMPUTED_TAG" +echo "Installing fixture into $TARGET_FIXTURE_DIR..." +rm -rf "$TARGET_FIXTURE_DIR" +mkdir -p "$TARGET_FIXTURE_DIR" +cp -a "$STAGING_DIR"/. "$TARGET_FIXTURE_DIR"/ + +FILE_COUNT=$(find "$TARGET_FIXTURE_DIR" -type f | wc -l) +echo "Success: shell fixture created at $TARGET_FIXTURE_DIR with $FILE_COUNT files." +echo "Verified commit: $RESOLVED_COMMIT" +echo "Verified build tag: $COMPUTED_TAG" diff --git a/perf/proxy.mjs b/perf/proxy.mjs new file mode 100644 index 0000000..42ec6b0 --- /dev/null +++ b/perf/proxy.mjs @@ -0,0 +1,201 @@ +import http from 'node:http'; +import { Transform } from 'node:stream'; + +/** + * ThrottleTransform + * Delays stream chunks to simulate bandwidth caps (e.g. 1.6 Mbps = 200,000 B/s). + */ +class ThrottleTransform extends Transform { + constructor(bytesPerSec) { + super(); + this.bytesPerSec = bytesPerSec; + } + + _transform(chunk, encoding, callback) { + if (!this.bytesPerSec || this.bytesPerSec <= 0) { + this.push(chunk); + return callback(); + } + const delayMs = (chunk.length / this.bytesPerSec) * 1000; + setTimeout(() => { + this.push(chunk); + callback(); + }, Math.min(delayMs, 2000)); + } +} + +/** + * Categorize Content-Type into standard metric buckets. + */ +export function categorizeContentType(contentType, urlPath) { + const ct = (contentType || '').toLowerCase(); + const path = (urlPath || '').toLowerCase(); + + if (ct.includes('text/html') || path.endsWith('.html')) return 'html'; + if (ct.includes('javascript') || path.endsWith('.js') || path.endsWith('.mjs')) return 'js'; + if (ct.includes('text/css') || path.endsWith('.css')) return 'css'; + if (ct.includes('font') || path.endsWith('.woff2') || path.endsWith('.woff') || path.endsWith('.ttf')) return 'fonts'; + if (ct.includes('image/') || path.endsWith('.png') || path.endsWith('.jpg') || path.endsWith('.svg') || path.endsWith('.webp') || path.endsWith('.ico')) return 'images'; + if (ct.includes('video/') || ct.includes('audio/') || path.endsWith('.mp4') || path.endsWith('.m4a') || path.endsWith('.mp3')) return 'media'; + if (ct.includes('application/json') || path.startsWith('/api/')) return 'api'; + return 'other'; +} + +/** + * Creates an HTTP throttling & stalling proxy for WebKit (and browser network tracing). + * Supports both forward proxying and reverse proxying to targetPort. + */ +export function createThrottleProxy({ targetPort, profile = 'lte' }) { + let activeTracking = false; + let requestCount = 0; + let totalWireBytes = 0; + let bytesByType = { + html: 0, + js: 0, + css: 0, + fonts: 0, + images: 0, + media: 0, + api: 0, + other: 0, + }; + + let reqIndex = 0; + let isOfflineMode = false; + + // Profile configuration: + // LTE: 150ms roundtrip latency, 1.6 Mbps (200 KB/s) download, 750 kbps upload + // Lossy: 250ms latency, 1.0 Mbps (125 KB/s) download, stall every 7th request by 1500ms + // Unthrottled: 0 latency, 0 throttle (for Chromium where CDP throttles network/CPU) + const isLossy = profile === 'lossy'; + const isUnthrottled = profile === 'unthrottled'; + const latencyMs = isUnthrottled ? 0 : (isLossy ? 250 : 150); + const downloadBytesPerSec = isUnthrottled ? 0 : (isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8); + + const server = http.createServer((clientReq, clientRes) => { + if (isOfflineMode) { + clientReq.destroy(); + return; + } + + reqIndex++; + const currentReqIndex = reqIndex; + + let stallMs = 0; + if (isLossy && currentReqIndex % 7 === 0) { + stallMs = 1500; // Simulated lossy stall + } + + const totalDelay = isUnthrottled ? 0 : (latencyMs + stallMs); + + const rawUrl = clientReq.url; + const targetUrl = rawUrl.startsWith('http://') || rawUrl.startsWith('https://') + ? rawUrl + : `http://127.0.0.1:${targetPort}${rawUrl}`; + + const parsedTarget = new URL(targetUrl); + const pathname = parsedTarget.pathname; + + const options = { + protocol: parsedTarget.protocol, + hostname: parsedTarget.hostname, + port: parsedTarget.port, + path: parsedTarget.pathname + parsedTarget.search, + method: clientReq.method, + headers: { ...clientReq.headers, host: parsedTarget.host }, + }; + + const forward = () => { + const upReq = http.request(options, (upRes) => { + const ct = upRes.headers['content-type'] || ''; + const cat = categorizeContentType(ct, pathname); + + if (activeTracking) { + requestCount++; + } + + clientRes.writeHead(upRes.statusCode, upRes.headers); + + if (isUnthrottled) { + upRes.on('data', (chunk) => { + if (activeTracking) { + totalWireBytes += chunk.length; + bytesByType[cat] = (bytesByType[cat] || 0) + chunk.length; + } + clientRes.write(chunk); + }); + upRes.on('end', () => clientRes.end()); + } else { + const throttler = new ThrottleTransform(downloadBytesPerSec); + + throttler.on('data', (chunk) => { + if (activeTracking) { + totalWireBytes += chunk.length; + bytesByType[cat] = (bytesByType[cat] || 0) + chunk.length; + } + }); + + upRes.pipe(throttler).pipe(clientRes); + } + }); + + upReq.on('error', (err) => { + try { + clientRes.statusCode = 502; + clientRes.end(err.message); + } catch {} + }); + + clientReq.pipe(upReq); + }; + + if (totalDelay > 0) { + setTimeout(forward, totalDelay); + } else { + forward(); + } + }); + + return { + server, + listen: () => + new Promise((resolve) => { + server.listen(0, '127.0.0.1', () => { + const port = server.address().port; + resolve(port); + }); + }), + close: () => + new Promise((resolve) => { + server.close(resolve); + }), + startTracking: () => { + activeTracking = true; + }, + stopTracking: () => { + activeTracking = false; + }, + resetStats: () => { + requestCount = 0; + totalWireBytes = 0; + bytesByType = { + html: 0, + js: 0, + css: 0, + fonts: 0, + images: 0, + media: 0, + api: 0, + other: 0, + }; + }, + setOffline: (val) => { + isOfflineMode = !!val; + }, + getStats: () => ({ + requestCount, + totalWireBytes, + bytesByType: { ...bytesByType }, + }), + }; +} diff --git a/perf/results/baseline-2026-10-07.json b/perf/results/baseline-2026-10-07.json new file mode 100644 index 0000000..3c83f1a --- /dev/null +++ b/perf/results/baseline-2026-10-07.json @@ -0,0 +1,1639 @@ +{ + "date": "2026-10-06T19:53:30.775Z", + "commit": "64f8eae92bc69857875a91e3f79be9c166766577", + "runsConfigured": 5, + "browsers": { + "chromium": { + "lte": { + "cold": { + "requestCount": { + "median": 164, + "min": 164, + "max": 164, + "values": [ + 164, + 164, + 164, + 164, + 164 + ] + }, + "wireBytes": { + "median": 1039315, + "min": 1039315, + "max": 1039315, + "values": [ + 1039315, + 1039315, + 1039315, + 1039315, + 1039315 + ] + }, + "fcp": { + "median": 1540, + "min": 1524, + "max": 1556, + "values": [ + 1544, + 1524, + 1540, + 1532, + 1556 + ] + }, + "lcp": { + "median": 1896, + "min": 1880, + "max": 1920, + "values": [ + 1904, + 1880, + 1896, + 1880, + 1920 + ] + }, + "domContentLoaded": { + "median": 10033, + "min": 10004, + "max": 10677, + "values": [ + 10677, + 10033, + 10004, + 10005, + 10048 + ] + }, + "bootDone": { + "median": 10163, + "min": 10129, + "max": 10798, + "values": [ + 10798, + 10163, + 10129, + 10138, + 10180 + ] + }, + "longTaskTotal": { + "median": 397, + "min": 385, + "max": 422, + "values": [ + 397, + 385, + 393, + 398, + 422 + ] + }, + "swInstallDuration": { + "median": 38446, + "min": 36687, + "max": 39157, + "values": [ + 39157, + 36687, + 38446, + 38498, + 37723 + ] + }, + "swInstallBytes": { + "median": 1039315, + "min": 1039315, + "max": 1039315, + "values": [ + 1039315, + 1039315, + 1039315, + 1039315, + 1039315 + ] + }, + "storageEstimate": { + "usage": 3480029, + "quota": 4298447325 + }, + "mediaReady": { + "median": null, + "min": null, + "max": null, + "values": [ + null, + null, + null, + null, + null + ] + }, + "autoplayBlocked": true, + "wireBytesByType": { + "html": 27057, + "js": 477223, + "css": 130718, + "fonts": 397072, + "images": 6278, + "media": 0, + "api": 336, + "other": 631 + } + }, + "warm": { + "requestCount": { + "median": 4, + "min": 3, + "max": 5, + "values": [ + 4, + 3, + 4, + 5, + 4 + ] + }, + "wireBytes": { + "median": 5321, + "min": 97, + "max": 10545, + "values": [ + 5321, + 97, + 5321, + 10545, + 5321 + ] + }, + "fcp": { + "median": 180, + "min": 172, + "max": 216, + "values": [ + 216, + 176, + 188, + 172, + 180 + ] + }, + "lcp": { + "median": 684, + "min": 172, + "max": 704, + "values": [ + 704, + 176, + 704, + 172, + 684 + ] + }, + "domContentLoaded": { + "median": 558, + "min": 507, + "max": 573, + "values": [ + 573, + 508, + 566, + 507, + 558 + ] + }, + "bootDone": { + "median": 675, + "min": 622, + "max": 692, + "values": [ + 692, + 622, + 691, + 625, + 675 + ] + }, + "longTaskTotal": { + "median": 156, + "min": 148, + "max": 165, + "values": [ + 156, + 148, + 163, + 152, + 165 + ] + } + }, + "offline": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 72, + "min": 68, + "max": 80, + "values": [ + 76, + 80, + 72, + 68, + 72 + ] + }, + "lcp": { + "median": 108, + "min": 72, + "max": 228, + "values": [ + 76, + 108, + 228, + 228, + 72 + ] + }, + "domContentLoaded": { + "median": 199, + "min": 197, + "max": 221, + "values": [ + 221, + 199, + 199, + 197, + 197 + ] + }, + "bootDone": { + "median": 219, + "min": 219, + "max": 228, + "values": [ + 228, + 220, + 219, + 219, + 219 + ] + }, + "reloadDuration": { + "median": 240, + "min": 238, + "max": 257, + "values": [ + 257, + 240, + 241, + 239, + 238 + ] + } + }, + "updateJs": { + "requestCount": { + "median": 88, + "min": 88, + "max": 88, + "values": [ + 88, + 88, + 88, + 88, + 88 + ] + }, + "wireBytes": { + "median": 610965, + "min": 610965, + "max": 610965, + "values": [ + 610965, + 610965, + 610965, + 610965, + 610965 + ] + }, + "duration": { + "median": 2013, + "min": 45, + "max": 2073, + "values": [ + 2073, + 2013, + 45, + 2063, + 1958 + ] + }, + "wireBytesByType": { + "html": 17998, + "js": 266589, + "css": 65359, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 62, + "other": 631 + } + }, + "updateCss": { + "requestCount": { + "median": 88, + "min": 86, + "max": 88, + "values": [ + 88, + 86, + 88, + 88, + 87 + ] + }, + "wireBytes": { + "median": 611005, + "min": 605714, + "max": 611005, + "values": [ + 611005, + 605714, + 611005, + 611005, + 605776 + ] + }, + "duration": { + "median": 1970, + "min": 1956, + "max": 2071, + "values": [ + 1989, + 1961, + 2071, + 1970, + 1956 + ] + }, + "wireBytesByType": { + "html": 17990, + "js": 266609, + "css": 65387, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 62, + "other": 631 + } + } + }, + "lossy": { + "cold": { + "requestCount": { + "median": 164, + "min": 164, + "max": 164, + "values": [ + 164, + 164, + 164, + 164, + 164 + ] + }, + "wireBytes": { + "median": 1039309, + "min": 1039309, + "max": 1039309, + "values": [ + 1039309, + 1039309, + 1039309, + 1039309, + 1039309 + ] + }, + "fcp": { + "median": 2344, + "min": 2080, + "max": 2368, + "values": [ + 2348, + 2344, + 2368, + 2336, + 2080 + ] + }, + "lcp": { + "median": 2912, + "min": 2228, + "max": 2940, + "values": [ + 2916, + 2912, + 2940, + 2904, + 2228 + ] + }, + "domContentLoaded": { + "median": 15875, + "min": 15753, + "max": 15903, + "values": [ + 15850, + 15903, + 15875, + 15875, + 15753 + ] + }, + "bootDone": { + "median": 16015, + "min": 15871, + "max": 16031, + "values": [ + 15975, + 16031, + 16015, + 16018, + 15871 + ] + }, + "longTaskTotal": { + "median": 432, + "min": 379, + "max": 434, + "values": [ + 379, + 404, + 433, + 434, + 432 + ] + }, + "swInstallDuration": { + "median": 42518, + "min": 42354, + "max": 44366, + "values": [ + 42469, + 44313, + 42518, + 44366, + 42354 + ] + }, + "swInstallBytes": { + "median": 1039309, + "min": 1039309, + "max": 1039309, + "values": [ + 1039309, + 1039309, + 1039309, + 1039309, + 1039309 + ] + }, + "storageEstimate": { + "usage": 3480104, + "quota": 5372189224 + }, + "mediaReady": { + "median": null, + "min": null, + "max": null, + "values": [ + null, + null, + null, + null, + null + ] + }, + "autoplayBlocked": true, + "wireBytesByType": { + "html": 26985, + "js": 477233, + "css": 130774, + "fonts": 397072, + "images": 6278, + "media": 0, + "api": 336, + "other": 631 + } + }, + "warm": { + "requestCount": { + "median": 4, + "min": 3, + "max": 4, + "values": [ + 3, + 4, + 4, + 4, + 3 + ] + }, + "wireBytes": { + "median": 5326, + "min": 97, + "max": 5326, + "values": [ + 97, + 5326, + 5326, + 5326, + 97 + ] + }, + "fcp": { + "median": 184, + "min": 180, + "max": 196, + "values": [ + 196, + 180, + 184, + 180, + 184 + ] + }, + "lcp": { + "median": 644, + "min": 184, + "max": 708, + "values": [ + 644, + 708, + 656, + 628, + 184 + ] + }, + "domContentLoaded": { + "median": 531, + "min": 500, + "max": 567, + "values": [ + 514, + 567, + 531, + 500, + 539 + ] + }, + "bootDone": { + "median": 651, + "min": 618, + "max": 697, + "values": [ + 632, + 697, + 651, + 618, + 651 + ] + }, + "longTaskTotal": { + "median": 155, + "min": 148, + "max": 167, + "values": [ + 155, + 167, + 155, + 155, + 148 + ] + } + }, + "offline": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 80, + "min": 68, + "max": 92, + "values": [ + 88, + 68, + 72, + 80, + 92 + ] + }, + "lcp": { + "median": 92, + "min": 68, + "max": 244, + "values": [ + 88, + 68, + 244, + 244, + 92 + ] + }, + "domContentLoaded": { + "median": 214, + "min": 194, + "max": 279, + "values": [ + 220, + 194, + 210, + 214, + 279 + ] + }, + "bootDone": { + "median": 237, + "min": 216, + "max": 323, + "values": [ + 241, + 216, + 233, + 237, + 323 + ] + }, + "reloadDuration": { + "median": 253, + "min": 237, + "max": 343, + "values": [ + 255, + 237, + 251, + 253, + 343 + ] + } + }, + "updateJs": { + "requestCount": { + "median": 88, + "min": 86, + "max": 88, + "values": [ + 88, + 86, + 88, + 88, + 88 + ] + }, + "wireBytes": { + "median": 610965, + "min": 605674, + "max": 610965, + "values": [ + 610965, + 605674, + 610965, + 610965, + 610965 + ] + }, + "duration": { + "median": 2031, + "min": 207, + "max": 2071, + "values": [ + 2002, + 2031, + 2034, + 207, + 2071 + ] + }, + "wireBytesByType": { + "html": 17998, + "js": 266589, + "css": 65359, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 62, + "other": 631 + } + }, + "updateCss": { + "requestCount": { + "median": 88, + "min": 88, + "max": 88, + "values": [ + 88, + 88, + 88, + 88, + 88 + ] + }, + "wireBytes": { + "median": 611005, + "min": 611005, + "max": 611005, + "values": [ + 611005, + 611005, + 611005, + 611005, + 611005 + ] + }, + "duration": { + "median": 2004, + "min": 1955, + "max": 2043, + "values": [ + 1955, + 1984, + 2004, + 2043, + 2035 + ] + }, + "wireBytesByType": { + "html": 17990, + "js": 266609, + "css": 65387, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 62, + "other": 631 + } + } + } + }, + "webkit": { + "lte": { + "cold": { + "requestCount": { + "median": 162, + "min": 162, + "max": 162, + "values": [ + 162, + 162, + 162, + 162, + 162 + ] + }, + "wireBytes": { + "median": 1179624, + "min": 1179624, + "max": 1179624, + "values": [ + 1179624, + 1179624, + 1179624, + 1179624, + 1179624 + ] + }, + "fcp": { + "median": 1088, + "min": 1026, + "max": 1105, + "values": [ + 1100, + 1026, + 1105, + 1083, + 1088 + ] + }, + "lcp": { + "median": 1287, + "min": 1216, + "max": 1304, + "values": [ + 1304, + 1216, + 1297, + 1272, + 1287 + ] + }, + "domContentLoaded": { + "median": 3178, + "min": 3118, + "max": 3210, + "values": [ + 3178, + 3118, + 3179, + 3161, + 3210 + ] + }, + "bootDone": { + "median": 3201, + "min": 3138, + "max": 3245, + "values": [ + 3201, + 3138, + 3201, + 3172, + 3245 + ] + }, + "longTaskTotal": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "swInstallDuration": { + "median": 30508, + "min": 30391, + "max": 32376, + "values": [ + 30508, + 30391, + 32376, + 30425, + 31458 + ] + }, + "swInstallBytes": { + "median": 1179624, + "min": 1179624, + "max": 1179624, + "values": [ + 1179624, + 1179624, + 1179624, + 1179624, + 1179624 + ] + }, + "storageEstimate": { + "usage": null, + "quota": null + }, + "mediaReady": { + "median": null, + "min": null, + "max": null, + "values": [ + null, + null, + null, + null, + null + ] + }, + "autoplayBlocked": true, + "wireBytesByType": { + "html": 32535, + "js": 585292, + "css": 156760, + "fonts": 397072, + "images": 6278, + "media": 0, + "api": 121, + "other": 1566 + } + }, + "warm": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 110, + "min": 104, + "max": 121, + "values": [ + 104, + 110, + 105, + 111, + 121 + ] + }, + "lcp": { + "median": 110, + "min": 104, + "max": 157, + "values": [ + 104, + 110, + 105, + 111, + 157 + ] + }, + "domContentLoaded": { + "median": 101, + "min": 96, + "max": 111, + "values": [ + 96, + 101, + 97, + 102, + 111 + ] + }, + "bootDone": { + "median": 131, + "min": 122, + "max": 149, + "values": [ + 124, + 131, + 122, + 131, + 149 + ] + }, + "longTaskTotal": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + } + }, + "offline": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 107, + "min": 104, + "max": 109, + "values": [ + 105, + 107, + 109, + 104, + 108 + ] + }, + "lcp": { + "median": 107, + "min": 104, + "max": 109, + "values": [ + 105, + 107, + 109, + 104, + 108 + ] + }, + "domContentLoaded": { + "median": 98, + "min": 95, + "max": 101, + "values": [ + 96, + 98, + 101, + 95, + 100 + ] + }, + "bootDone": { + "median": 128, + "min": 124, + "max": 130, + "values": [ + 128, + 128, + 130, + 124, + 127 + ] + }, + "reloadDuration": { + "median": 135, + "min": 131, + "max": 136, + "values": [ + 135, + 135, + 136, + 131, + 134 + ] + } + }, + "updateJs": { + "requestCount": { + "median": 86, + "min": 86, + "max": 86, + "values": [ + 86, + 86, + 86, + 86, + 86 + ] + }, + "wireBytes": { + "median": 680250, + "min": 680250, + "max": 680250, + "values": [ + 680250, + 680250, + 680250, + 680250, + 680250 + ] + }, + "duration": { + "median": 4457, + "min": 2579, + "max": 4520, + "values": [ + 4478, + 4457, + 3625, + 4520, + 2579 + ] + }, + "wireBytesByType": { + "html": 21688, + "js": 319109, + "css": 78344, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 0, + "other": 783 + } + }, + "updateCss": { + "requestCount": { + "median": 86, + "min": 86, + "max": 86, + "values": [ + 86, + 86, + 86, + 86, + 86 + ] + }, + "wireBytes": { + "median": 680265, + "min": 680265, + "max": 680265, + "values": [ + 680265, + 680265, + 680265, + 680265, + 680265 + ] + }, + "duration": { + "median": 4510, + "min": 4449, + "max": 4554, + "values": [ + 4528, + 4554, + 4510, + 4449, + 4458 + ] + }, + "wireBytesByType": { + "html": 21690, + "js": 319086, + "css": 78380, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 0, + "other": 783 + } + } + }, + "lossy": { + "cold": { + "requestCount": { + "median": 162, + "min": 162, + "max": 162, + "values": [ + 162, + 162, + 162, + 162, + 162 + ] + }, + "wireBytes": { + "median": 1179624, + "min": 1179624, + "max": 1179624, + "values": [ + 1179624, + 1179624, + 1179624, + 1179624, + 1179624 + ] + }, + "fcp": { + "median": 3557, + "min": 3216, + "max": 3838, + "values": [ + 3838, + 3229, + 3768, + 3216, + 3557 + ] + }, + "lcp": { + "median": 3735, + "min": 3413, + "max": 4034, + "values": [ + 4034, + 3413, + 3959, + 3415, + 3735 + ] + }, + "domContentLoaded": { + "median": 7281, + "min": 6948, + "max": 8688, + "values": [ + 6948, + 8688, + 8655, + 7281, + 7181 + ] + }, + "bootDone": { + "median": 7303, + "min": 6963, + "max": 8710, + "values": [ + 6963, + 8710, + 8685, + 7303, + 7202 + ] + }, + "longTaskTotal": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "swInstallDuration": { + "median": 35449, + "min": 35013, + "max": 36906, + "values": [ + 35621, + 36906, + 35013, + 35449, + 35405 + ] + }, + "swInstallBytes": { + "median": 1179624, + "min": 1179624, + "max": 1179624, + "values": [ + 1179624, + 1179624, + 1179624, + 1179624, + 1179624 + ] + }, + "storageEstimate": { + "usage": null, + "quota": null + }, + "mediaReady": { + "median": null, + "min": null, + "max": null, + "values": [ + null, + null, + null, + null, + null + ] + }, + "autoplayBlocked": true, + "wireBytesByType": { + "html": 32535, + "js": 585292, + "css": 156760, + "fonts": 397072, + "images": 6278, + "media": 0, + "api": 121, + "other": 1566 + } + }, + "warm": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 111, + "min": 104, + "max": 124, + "values": [ + 112, + 107, + 104, + 124, + 111 + ] + }, + "lcp": { + "median": 112, + "min": 107, + "max": 130, + "values": [ + 112, + 107, + 130, + 124, + 111 + ] + }, + "domContentLoaded": { + "median": 102, + "min": 96, + "max": 115, + "values": [ + 104, + 96, + 96, + 115, + 102 + ] + }, + "bootDone": { + "median": 131, + "min": 124, + "max": 144, + "values": [ + 131, + 124, + 124, + 144, + 134 + ] + }, + "longTaskTotal": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + } + }, + "offline": { + "requestCount": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "wireBytes": { + "median": 0, + "min": 0, + "max": 0, + "values": [ + 0, + 0, + 0, + 0, + 0 + ] + }, + "fcp": { + "median": 109, + "min": 104, + "max": 119, + "values": [ + 119, + 113, + 109, + 108, + 104 + ] + }, + "lcp": { + "median": 119, + "min": 104, + "max": 142, + "values": [ + 119, + 142, + 136, + 108, + 104 + ] + }, + "domContentLoaded": { + "median": 101, + "min": 96, + "max": 111, + "values": [ + 111, + 104, + 101, + 100, + 96 + ] + }, + "bootDone": { + "median": 129, + "min": 123, + "max": 140, + "values": [ + 140, + 135, + 129, + 128, + 123 + ] + }, + "reloadDuration": { + "median": 141, + "min": 130, + "max": 147, + "values": [ + 147, + 147, + 141, + 134, + 130 + ] + } + }, + "updateJs": { + "requestCount": { + "median": 86, + "min": 86, + "max": 87, + "values": [ + 86, + 86, + 86, + 87, + 86 + ] + }, + "wireBytes": { + "median": 680250, + "min": 680250, + "max": 686464, + "values": [ + 680250, + 680250, + 680250, + 686464, + 680250 + ] + }, + "duration": { + "median": 8135, + "min": 7243, + "max": 8398, + "values": [ + 8135, + 8398, + 7880, + 8313, + 7243 + ] + }, + "wireBytesByType": { + "html": 21688, + "js": 319109, + "css": 78344, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 0, + "other": 783 + } + }, + "updateCss": { + "requestCount": { + "median": 87, + "min": 86, + "max": 87, + "values": [ + 86, + 87, + 87, + 86, + 87 + ] + }, + "wireBytes": { + "median": 680327, + "min": 680265, + "max": 686479, + "values": [ + 680265, + 686479, + 686479, + 680265, + 680327 + ] + }, + "duration": { + "median": 8269, + "min": 7987, + "max": 8799, + "values": [ + 8269, + 8791, + 7987, + 8157, + 8799 + ] + }, + "wireBytesByType": { + "html": 21690, + "js": 319086, + "css": 78380, + "fonts": 254048, + "images": 6278, + "media": 0, + "api": 0, + "other": 783 + } + } + } + } + } +} \ No newline at end of file diff --git a/plans/perf-baseline.md b/plans/perf-baseline.md new file mode 100644 index 0000000..1b120c7 --- /dev/null +++ b/plans/perf-baseline.md @@ -0,0 +1,175 @@ +# Performance Baseline Report (Phase 0 / UNCHANGED code) + +**Date**: 2026-10-07 +**Commit**: `b77938a` +**Dataset**: `perf/results/baseline-2026-10-07.json` + +--- + +## 1. Executive Summary & Headline Baseline Numbers + +This baseline captures the performance metrics of the unoptimized ytplayer codebase (commit `b77938a`, matching production tag `1175f1a1d2c1`) across 5 scenarios on both Chromium and WebKit under simulated LTE and Lossy network profiles (median of 5 runs). + +### Headline Baseline Numbers + +- **Cold First-Load Wire Transfer**: + - **Chromium**: 164 requests, **1,015.0 KB** compressed wire bytes (boot-done median: **10,163 ms** under 4× CPU throttle + LTE). + - **WebKit**: 162 requests, **1,152.0 KB (1.12 MB)** wire bytes (boot-done median: **3,201 ms** under proxy LTE). + - Breakdown: ~477 KB JavaScript, ~397 KB Web Fonts, ~131 KB CSS, ~27 KB HTML. + +- **Warm Reload (under Service Worker)**: + - **Chromium**: 4 requests, **5.2 KB** wire bytes (periodic API version poll), boot-done median: **675 ms** (LTE) / **651 ms** (Lossy). + - **WebKit**: 0 requests, **0 KB** wire bytes, boot-done median: **131 ms** (LTE) / **131 ms** (Lossy). + +- **Offline Reload (under Service Worker)**: + - **Chromium**: 0 requests, **0 B** wire bytes, reload median: **240 ms**, boot-done median: **219 ms**. + - **WebKit**: 0 requests, **0 B** wire bytes, reload median: **135 ms**, boot-done median: **128 ms**. + +- **Update N -> N+1 Wire Cost (The Invalidation Penalty)**: + - **1-line `app.js` change**: + - **Chromium**: 88 requests, **596.6 KB** wire bytes (~611 KB uncompressed/brotli), duration: **2,013 ms** (LTE) / **2,031 ms** (Lossy). + - **WebKit**: 86 requests, **664.3 KB** wire bytes (~680 KB wire), duration: **4,457 ms** (LTE) / **8,135 ms** (Lossy). + - **1-line `theme-glass.css` change**: + - **Chromium**: 88 requests, **596.7 KB** wire bytes, duration: **1,970 ms** (LTE) / **2,004 ms** (Lossy). + - **WebKit**: 86-87 requests, **664.3 KB** wire bytes, duration: **4,510 ms** (LTE) / **8,269 ms** (Lossy). + +> **Core Invalidation Finding**: A single 1-line change to *any* static file (`app.js` or `theme-glass.css`) mutates `BUILD_TAG` calculated by `computeBuildTag()`. This causes `sw.js` to re-fetch and re-cache all 86-88 shell assets (~600-664 KB wire payload) rather than just the mutated resource. + +--- + +## 2. Benchmark Harness Execution + +The benchmark harness is located in `perf/baseline.mjs` and driven by Playwright with an isolated ephemeral Bun server and a custom pacing proxy (`perf/proxy.mjs`). + +### Quick Start / Exact Commands + +```bash +# Ensure node_modules is symlinked to root +ln -s ~/development/personal/ytplayer/node_modules node_modules + +# Run full baseline suite (all scenarios, both browsers, both profiles, 5 runs) +node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --out perf/results/baseline-2026-10-07.json + +# Run individual browser or scenario +node perf/baseline.mjs --runs 5 --browser chromium --profile lte --scenario cold +node perf/baseline.mjs --runs 5 --browser webkit --profile lte --scenario update-js + +# Run comparison against baseline after optimizations +node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --compare perf/results/baseline-2026-10-07.json +``` + +--- + +## 3. Results by Browser and Profile (Median of 5 Runs) + +### Chromium — LTE (1.6 Mbps down, 750 kbps up, 150 ms RTT, 4× CPU throttle) + +| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) | +|---|---|---|---|---|---|---|---| +| **Cold First Load** | 164 | 1,015.0 KB | 1,540 | 1,896 | 10,163 | 38,446 | 397 | +| **Warm Reload (SW)** | 4 | 5.2 KB | 180 | 684 | 675 | - | 156 | +| **Offline Reload** | 0 | 0.0 KB | 72 | 108 | 219 | 240 | - | +| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,013 | - | +| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 1,970 | - | + +- **Storage Estimate**: Quota = 4,298,447,325 B (~4.0 GB), Usage = 3,480,029 B (~3.31 MB). +- **Cold Wire Bytes by Type**: JS 477.2 KB, Fonts 397.1 KB, CSS 130.7 KB, HTML 27.1 KB, Images 6.3 KB, API/Other 1.0 KB. + +--- + +### Chromium — Lossy (1.0 Mbps down, 500 kbps up, 250 ms RTT, 4× CPU throttle) + +| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) | +|---|---|---|---|---|---|---|---| +| **Cold First Load** | 164 | 1,015.0 KB | 2,344 | 2,912 | 16,015 | 42,518 | 432 | +| **Warm Reload (SW)** | 4 | 5.2 KB | 184 | 644 | 651 | - | 155 | +| **Offline Reload** | 0 | 0.0 KB | 80 | 92 | 237 | 253 | - | +| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,031 | - | +| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 2,004 | - | + +--- + +### WebKit — LTE (1.6 Mbps down, 150 ms RTT via proxy) + +| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) | +|---|---|---|---|---|---|---|---| +| **Cold First Load** | 162 | 1.12 MB | 1,088 | 1,287 | 3,201 | 30,508 | 0* | +| **Warm Reload (SW)** | 0 | 0.0 KB | 110 | 110 | 131 | - | 0* | +| **Offline Reload** | 0 | 0.0 KB | 107 | 107 | 128 | 135 | - | +| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 4,457 | - | +| **Update N->N+1 (CSS)** | 86 | 664.3 KB | - | - | - | 4,510 | - | + +--- + +### WebKit — Lossy (1.0 Mbps down, 250 ms RTT + 1,500 ms stalls via proxy) + +| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) | +|---|---|---|---|---|---|---|---| +| **Cold First Load** | 162 | 1.12 MB | 3,557 | 3,735 | 7,303 | 35,449 | 0* | +| **Warm Reload (SW)** | 0 | 0.0 KB | 111 | 112 | 131 | - | 0* | +| **Offline Reload** | 0 | 0.0 KB | 109 | 119 | 129 | 141 | - | +| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 8,135 | - | +| **Update N->N+1 (CSS)** | 87 | 664.4 KB | - | - | - | 8,269 | - | + +*\*Note: WebKit does not expose `PerformanceLongTaskTiming` (`longtask` entries), so long task total evaluates to 0 ms.* + +--- + +## 4. Variance & Noise Observed + +Over the 5 runs per configuration, measurements demonstrated high stability: +- **Wire Bytes & Request Counts**: Deterministic (0% variance across identical runs). +- **Cold Boot-Done**: + - Chromium LTE: Min 10,129 ms, Max 10,798 ms (<6% variance). + - Chromium Lossy: Min 15,871 ms, Max 16,031 ms (<1% variance). + - WebKit LTE: Min 3,138 ms, Max 3,245 ms (<3% variance). + - WebKit Lossy: Min 6,963 ms, Max 8,710 ms (~20% variance due to proxy random stalls). +- **Warm & Offline Reload**: + - Chromium Offline Reload: Min 238 ms, Max 257 ms (<8% variance). + - WebKit Offline Reload: Min 131 ms, Max 141 ms (<7% variance). +- **Update Duration**: + - Chromium: Runs cluster around ~2,000 ms (2,013 ms median). Occasional outlier runs (~45–200 ms) occur if the browser resolves `reg.update()` immediately before network completion reporting. + - WebKit: Steady scaling between LTE (~4.5 s) and Lossy (~8.2 s) pacing. + +--- + +## 5. Limitations & Caveats + +1. **WebKit Network Throttling**: + WebKit lacks Chrome DevTools Protocol (CDP) support for hardware network emulation and CPU throttling. We paced throughput (1.6 Mbps) and injected latency/stalls using a custom local reverse proxy (`perf/proxy.mjs`). This throttles HTTP response chunks truthfully but cannot simulate OS-level socket buffer delays or 4× CPU throttling. +2. **Autoplay Policy in WebKit**: + Playwright WebKit on Linux rejects `--autoplay-policy=no-user-gesture-required` and enforces Safari-style gesture requirements on unmuted audio elements. The media playback benchmark detects this and correctly reports `autoplayBlocked: true` without failing the test run. +3. **WebKit SetOffline Incompatibility**: + Calling Playwright's `context.setOffline(true)` on Linux WebKit causes an unrecoverable engine crash (`WebKit encountered an internal error`). Offline mode for WebKit is simulated by terminating the proxy upstream connection (`proxy.setOffline(true)`). + +--- + +## 6. Manual iPhone Testing Protocol (Phase 0 / iOS Safari PWA) + +Because automated headless WebKit cannot fully replicate iOS Safari's mobile background constraints and battery telemetry, the following manual protocol is required: + +### Protocol Steps + +1. **Install PWA**: + - Open Safari on iPhone and navigate to `https://worship.hesed.sbs`. + - Tap the Share button -> **"Add to Home Screen"**. + - Launch ytplayer directly from the Home Screen icon. + +2. **Verify Version & Cache**: + - Verify the footer / version string matches `1175f1a1d2c1` (or current production build tag). + - Turn on Airplane Mode and force-reload to confirm offline readiness. Turn Airplane Mode back off. + +3. **30-Minute Screen-Off Battery Test**: + - Note baseline battery percentage (e.g., 85%). + - Select a pre-cached playlist or track and tap **Play**. Ensure continuous audio is heard. + - Lock the screen (display off) and let playback run uninterrupted for **30 minutes**. + - Note battery percentage at 30 minutes (calculate drain %/hr). + - Listen for any audio dropouts, stuttering, or premature process suspension. + +4. **Interruption & Resumption Test**: + - Trigger an interruption while locked (e.g., initiate an incoming phone call or start Siri). + - End the interruption and observe if playback automatically resumes via `MediaSession` handlers. + - Test Lock Screen Controls (Play/Pause, Next Track, scrubber). + +5. **Log Findings**: + - Record iPhone Model, iOS Version, Battery Drain (% per 30m), Dropouts Count (0 = pass), and Resumption Status (Pass/Fail).