Merge branch 'agy/phase0-perf'
This commit is contained in:
Jonathan Sykes
4
.gitignore
vendored
4
.gitignore
vendored
@@ -40,3 +40,7 @@ legacy/releases/
|
||||
__pycache__/
|
||||
plans/codex-shots*/
|
||||
plans/codex-report*.md
|
||||
|
||||
# performance harness artifacts and temp files
|
||||
perf/fixtures/
|
||||
perf/.tmp/
|
||||
|
||||
132
perf/README.md
Normal file
132
perf/README.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# Performance Baseline Harness (`perf/`)
|
||||
|
||||
Automated performance baseline and regression harness for the **ytplayer PWA**.
|
||||
|
||||
The harness drives the real Bun backend server (`server/server.js`) serving static Brotli-compressed assets, combined with Playwright browser automation on Chromium and WebKit.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture
|
||||
|
||||
### Backend Server (`server/server.js`)
|
||||
- Runs as an isolated child process using **Bun** on an ephemeral port.
|
||||
- Configured with dedicated scratch directories under `perf/.tmp/` (`DATA_DIR`, `DB_PATH`, `MEDIA_DIR`) so no production or developer data is ever touched.
|
||||
- Employs the real static asset delivery stack (`Brotli` encoding, `ETag` generation, `Cache-Control` immutable caching for version-stamped assets, and `computeBuildTag()`).
|
||||
|
||||
### Browser Automation & Network Throttling
|
||||
- **Chromium**:
|
||||
- Network emulation via Chrome DevTools Protocol (`Network.emulateNetworkConditions`).
|
||||
- CPU throttling at **4× slowdown** (`Emulation.setCPUThrottlingRate`).
|
||||
- Unrestricted audio/video playback via `--autoplay-policy=no-user-gesture-required`.
|
||||
- **WebKit**:
|
||||
- WebKit on Linux lacks CDP network emulation capabilities.
|
||||
- Throttling and stalling are handled by a dedicated local proxy (`perf/proxy.mjs`) sitting between WebKit and the Bun server.
|
||||
- The proxy shapes traffic by pacing chunk delivery through backpressured streams and injecting round-trip latency and stalls.
|
||||
- **Network Profiles**:
|
||||
- **`lte`**: 1.6 Mbps (200 KB/s) download, 750 kbps upload, 150 ms round-trip latency.
|
||||
- **`lossy`**: 1.0 Mbps (125 KB/s) download, 250 ms latency, with intermittent connection stalls (every 7th request delayed by 1,500 ms).
|
||||
|
||||
### Shell Fixture Extractor (`perf/make-shell-fixture.sh`)
|
||||
- Archives the frontend from git commits (`git archive <commit> frontend`) into `perf/fixtures/shell-<tag>/`.
|
||||
- Computes the build tag using the exact sha256 walk algorithm from `server/server.js computeBuildTag()`.
|
||||
- Validates the computed tag against live production (`https://worship.hesed.sbs/api/version`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Test Scenarios
|
||||
|
||||
1. **`cold` (Cold First Load)**:
|
||||
- Fresh browser context with empty cache, no service worker, and clean storage.
|
||||
- Measures initial page fetch, FCP, LCP, DOMContentLoaded, boot milestone (`#cards` render), media playback readiness, service worker precaching duration, and origin storage quota.
|
||||
2. **`warm` (Warm Reload under SW)**:
|
||||
- Re-navigates the page with active Service Worker and primed HTTP cache.
|
||||
- Measures cached asset retrieval efficiency and second-load boot performance.
|
||||
3. **`offline` (Offline Reload)**:
|
||||
- Sets offline state and reloads the application.
|
||||
- Verifies the app boots entirely from `CacheStorage` precached shell assets without network connectivity.
|
||||
4. **`update-js` (Update N → N+1 with 1-line JS comment)**:
|
||||
- Boots client against Tree N (base frontend), establishing an active Service Worker.
|
||||
- Swaps server to Tree N+1 (one comment line appended to `app.js`, altering `BUILD_TAG`).
|
||||
- Triggers `registration.update()` and measures total wire bytes and time until the new Service Worker reaches `waiting` state (`installed`).
|
||||
5. **`update-css` (Update N → N+1 with 1-line CSS rule)**:
|
||||
- Identical to `update-js`, but Tree N+1 has one CSS rule appended to `theme-glass.css`.
|
||||
- Measures the full shell re-download penalty caused by any stylesheet modification.
|
||||
|
||||
---
|
||||
|
||||
## 3. Metrics Collected
|
||||
|
||||
Each scenario records the following metrics across **5 independent runs** (reporting min, max, and median):
|
||||
- **Request Count**: Total HTTP requests made across the wire.
|
||||
- **Wire Bytes**: Total compressed bytes transferred over the network.
|
||||
- **Wire Bytes by Type**: Categorized byte breakdown (`html`, `js`, `css`, `fonts`, `images`, `media`, `api`, `other`).
|
||||
- **FCP (First Contentful Paint)**: Time from navigation start to first rendered content (ms).
|
||||
- **LCP (Largest Contentful Paint)**: Time to largest contentful paint via `PerformanceObserver` (ms).
|
||||
- **DOMContentLoaded**: Time to DOMContentLoaded event end (ms).
|
||||
- **Boot Done**: Time from navigation start until the first `#cards` view rendering is completed (ms).
|
||||
- **Long Tasks Total**: Cumulative execution time of main thread tasks exceeding 50 ms (ms, Chromium).
|
||||
- **SW Install Duration**: Time from Service Worker registration until precaching completes (ms).
|
||||
- **SW Install Wire Bytes**: Network bytes transmitted during Service Worker installation.
|
||||
- **Storage Estimate**: Quota and usage reported by `navigator.storage.estimate()` (bytes).
|
||||
- **Media Ready**: Elapsed time for deterministic media fixture (`benchmedia01`) to reach `readyState >= 3` / playing state. Notes if autoplay policy blocked playback.
|
||||
|
||||
---
|
||||
|
||||
## 4. How to Run
|
||||
|
||||
### Prerequisites
|
||||
Symlink dependencies and ensure server dependencies are installed:
|
||||
```bash
|
||||
# Worktree root
|
||||
ln -s ~/development/personal/ytplayer/node_modules node_modules
|
||||
cd server && bun install && cd ..
|
||||
```
|
||||
|
||||
### Basic Commands
|
||||
Run the complete baseline suite (5 runs, all browsers, LTE profile):
|
||||
```bash
|
||||
node perf/baseline.mjs
|
||||
```
|
||||
|
||||
Run a specific browser or profile:
|
||||
```bash
|
||||
# Chromium on LTE profile
|
||||
node perf/baseline.mjs --browser chromium --profile lte
|
||||
|
||||
# WebKit on LTE profile
|
||||
node perf/baseline.mjs --browser webkit --profile lte
|
||||
|
||||
# Both browsers on lossy profile
|
||||
node perf/baseline.mjs --browser all --profile lossy
|
||||
```
|
||||
|
||||
Run a single scenario:
|
||||
```bash
|
||||
# Cold load only
|
||||
node perf/baseline.mjs --scenario cold
|
||||
|
||||
# Update scenario with 1-line JS change
|
||||
node perf/baseline.mjs --scenario update-js
|
||||
|
||||
# Update scenario with 1-line CSS change
|
||||
node perf/baseline.mjs --scenario update-css
|
||||
```
|
||||
|
||||
Configure run count:
|
||||
```bash
|
||||
# Quick sanity check (1 iteration)
|
||||
node perf/baseline.mjs --runs 1
|
||||
|
||||
# High precision benchmark (7 iterations)
|
||||
node perf/baseline.mjs --runs 7
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 5 Comparison Mode (`--compare`)
|
||||
|
||||
To verify performance gains and detect regressions in later phases:
|
||||
```bash
|
||||
node perf/baseline.mjs --compare perf/results/baseline-2026-10-07.json
|
||||
```
|
||||
Prints an ASCII/Markdown delta comparison table highlighting percentage changes across wire bytes, boot duration, and update overhead.
|
||||
948
perf/baseline.mjs
Executable file
948
perf/baseline.mjs
Executable file
@@ -0,0 +1,948 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* perf/baseline.mjs
|
||||
*
|
||||
* Performance baseline harness for ytplayer PWA.
|
||||
* Drives the real Bun server and runs Playwright (Chromium with CDP throttling + CPU 4x;
|
||||
* WebKit with a local throttling & stalling proxy).
|
||||
*
|
||||
* Scenarios:
|
||||
* - cold: fresh context, network download, SW install, boot, media playback check
|
||||
* - warm: re-navigate with active SW and primed HTTP cache
|
||||
* - offline: context.setOffline(true), reload under SW cache
|
||||
* - update-js: update N->N+1 (1-line comment appended to app.js) -> wire bytes + time until new SW ready
|
||||
* - update-css: update N->N+1 (1-line rule appended to theme-glass.css) -> wire bytes + time until new SW ready
|
||||
*
|
||||
* Usage:
|
||||
* node perf/baseline.mjs [options]
|
||||
* Options:
|
||||
* --runs <number> Runs per scenario for median calculation (default: 5)
|
||||
* --browser <chr|wk|all> Browser(s) to test (default: all)
|
||||
* --profile <lte|lossy|all> Network profile (default: lte)
|
||||
* --scenario <name|all> Specific scenario or all (default: all)
|
||||
* --out <path> Output JSON path
|
||||
* --compare <json-path> Compare results against a previous baseline JSON
|
||||
*/
|
||||
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, rmSync, cpSync, appendFileSync, statSync, writeFileSync, readFileSync, symlinkSync, unlinkSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createThrottleProxy } from './proxy.mjs';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
const REPO_ROOT = path.resolve(__dirname, '..');
|
||||
|
||||
// Dynamically resolve playwright from node_modules
|
||||
let pw;
|
||||
try {
|
||||
pw = await import('playwright');
|
||||
} catch {
|
||||
pw = await import(path.join(REPO_ROOT, 'node_modules', 'playwright', 'index.mjs'));
|
||||
}
|
||||
|
||||
import { createRequire } from 'node:module';
|
||||
const req = createRequire(import.meta.url);
|
||||
const { createClient } = req(path.join(REPO_ROOT, 'server', 'node_modules', '@libsql/client'));
|
||||
|
||||
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
function parseArgs() {
|
||||
const args = process.argv.slice(2);
|
||||
const options = {
|
||||
runs: 5,
|
||||
browser: 'all', // 'chromium', 'webkit', or 'all'
|
||||
profile: 'lte', // 'lte', 'lossy', or 'all'
|
||||
scenario: 'all', // 'cold', 'warm', 'offline', 'update-js', 'update-css', or 'all'
|
||||
out: null,
|
||||
compare: null,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--runs' && i + 1 < args.length) options.runs = parseInt(args[++i], 10);
|
||||
else if (a === '--browser' && i + 1 < args.length) options.browser = args[++i];
|
||||
else if (a === '--profile' && i + 1 < args.length) options.profile = args[++i];
|
||||
else if (a === '--scenario' && i + 1 < args.length) options.scenario = args[++i];
|
||||
else if (a === '--out' && i + 1 < args.length) options.out = args[++i];
|
||||
else if (a === '--compare' && i + 1 < args.length) options.compare = args[++i];
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
function median(values) {
|
||||
const nums = values.filter((v) => typeof v === 'number' && !Number.isNaN(v)).sort((a, b) => a - b);
|
||||
if (!nums.length) return null;
|
||||
const mid = Math.floor(nums.length / 2);
|
||||
return nums.length % 2 !== 0 ? nums[mid] : Math.round(((nums[mid - 1] + nums[mid]) / 2) * 10) / 10;
|
||||
}
|
||||
|
||||
function summarizeList(values) {
|
||||
const valid = values.filter((v) => v !== null && v !== undefined && !Number.isNaN(v));
|
||||
if (!valid.length) return { median: null, min: null, max: null, values };
|
||||
return {
|
||||
median: median(valid),
|
||||
min: Math.min(...valid),
|
||||
max: Math.max(...valid),
|
||||
values,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Setup deterministic local media fixture
|
||||
*/
|
||||
function prepareMediaFiles(mediaDir) {
|
||||
mkdirSync(mediaDir, { recursive: true });
|
||||
const mp4 = path.join(mediaDir, 'benchmedia01.1.mp4');
|
||||
const m4a = path.join(mediaDir, 'benchmedia01.1.m4a');
|
||||
|
||||
if (!existsSync(mp4)) {
|
||||
spawnSync('ffmpeg', [
|
||||
'-y', '-f', 'lavfi', '-i', 'testsrc=size=640x360:rate=30:duration=5',
|
||||
'-f', 'lavfi', '-i', 'sine=frequency=440:duration=5',
|
||||
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p',
|
||||
'-c:a', 'aac', '-movflags', '+faststart', '-shortest', mp4,
|
||||
]);
|
||||
}
|
||||
|
||||
if (!existsSync(m4a)) {
|
||||
spawnSync('ffmpeg', [
|
||||
'-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=5',
|
||||
'-c:a', 'aac', '-shortest', m4a,
|
||||
]);
|
||||
}
|
||||
|
||||
return { mp4, m4a };
|
||||
}
|
||||
|
||||
async function seedDatabase(dbPath, mediaFiles) {
|
||||
const db = createClient({ url: 'file:' + dbPath });
|
||||
const mp4Size = statSync(mediaFiles.mp4).size;
|
||||
const m4aSize = statSync(mediaFiles.m4a).size;
|
||||
|
||||
await db.execute({
|
||||
sql: `INSERT OR REPLACE INTO media_cache (
|
||||
video_id, status, gen, size, height, vcodec, acodec, duration, meta, last_access
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
args: [
|
||||
'benchmedia01',
|
||||
'ready',
|
||||
1,
|
||||
mp4Size + m4aSize,
|
||||
360,
|
||||
'h264',
|
||||
'aac',
|
||||
5.0,
|
||||
JSON.stringify({ title: 'Baseline Deterministic Clip', channel: 'Perf Benchmark' }),
|
||||
Date.now(),
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Bun Server Manager
|
||||
*/
|
||||
class ServerManager {
|
||||
constructor(scratchDir, publicDir) {
|
||||
this.scratchDir = scratchDir;
|
||||
this.publicDir = publicDir;
|
||||
this.dataDir = path.join(scratchDir, 'data');
|
||||
this.mediaDir = path.join(this.dataDir, 'media');
|
||||
this.dbPath = path.join(this.dataDir, 'ytplayer.db');
|
||||
this.serverProcess = null;
|
||||
this.port = 0;
|
||||
}
|
||||
|
||||
async start() {
|
||||
mkdirSync(this.dataDir, { recursive: true });
|
||||
const fixtureDir = path.join(REPO_ROOT, 'perf', 'fixtures', 'media');
|
||||
const mediaFiles = prepareMediaFiles(fixtureDir);
|
||||
|
||||
// Create a dedicated server working dir with public symlink
|
||||
const srvCwd = path.join(this.scratchDir, 'srv');
|
||||
mkdirSync(srvCwd, { recursive: true });
|
||||
const publicLink = path.join(srvCwd, 'public');
|
||||
if (existsSync(publicLink)) unlinkSync(publicLink);
|
||||
symlinkSync(this.publicDir, publicLink);
|
||||
|
||||
// Pick random available ephemeral port once and reuse across restarts
|
||||
this.port = this.port || (4300 + Math.floor(Math.random() * 2000));
|
||||
|
||||
const env = {
|
||||
...process.env,
|
||||
PORT: String(this.port),
|
||||
DATA_DIR: this.dataDir,
|
||||
MEDIA_DIR: this.mediaDir,
|
||||
DB_PATH: this.dbPath,
|
||||
};
|
||||
|
||||
const serverScript = path.join(REPO_ROOT, 'server', 'server.js');
|
||||
this.serverProcess = spawn('bun', [serverScript], {
|
||||
cwd: srvCwd,
|
||||
env,
|
||||
stdio: 'ignore',
|
||||
});
|
||||
|
||||
// Wait for server readiness
|
||||
let ready = false;
|
||||
for (let i = 0; i < 50; i++) {
|
||||
try {
|
||||
const res = await fetch(`http://127.0.0.1:${this.port}/api/version`);
|
||||
if (res.ok) {
|
||||
ready = true;
|
||||
break;
|
||||
}
|
||||
} catch {}
|
||||
await sleep(100);
|
||||
}
|
||||
|
||||
if (!ready) {
|
||||
throw new Error(`Server failed to start on port ${this.port}`);
|
||||
}
|
||||
|
||||
// Copy media files to mediaDir and seed media table AFTER server startup sweep
|
||||
mkdirSync(this.mediaDir, { recursive: true });
|
||||
const targetMp4 = path.join(this.mediaDir, 'benchmedia01.1.mp4');
|
||||
const targetM4a = path.join(this.mediaDir, 'benchmedia01.1.m4a');
|
||||
cpSync(mediaFiles.mp4, targetMp4);
|
||||
cpSync(mediaFiles.m4a, targetM4a);
|
||||
await seedDatabase(this.dbPath, { mp4: targetMp4, m4a: targetM4a });
|
||||
}
|
||||
|
||||
async stop() {
|
||||
if (this.serverProcess) {
|
||||
this.serverProcess.kill('SIGKILL');
|
||||
this.serverProcess = null;
|
||||
await sleep(100);
|
||||
}
|
||||
}
|
||||
|
||||
async restartWithPublic(newPublicDir) {
|
||||
await this.stop();
|
||||
this.publicDir = newPublicDir;
|
||||
await this.start();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Configure page for metrics collection
|
||||
*/
|
||||
async function setupPageMetrics(page) {
|
||||
await page.addInitScript(() => {
|
||||
window.__bootDone = null;
|
||||
window.__fcp = null;
|
||||
window.__lcp = null;
|
||||
window.__longTasksTotal = 0;
|
||||
|
||||
// Observe boot completion landmark (#cards view set by renderList())
|
||||
const markBoot = () => {
|
||||
const cards = document.getElementById('cards');
|
||||
if (cards && cards.dataset && cards.dataset.view) {
|
||||
if (!window.__bootDone) window.__bootDone = performance.now();
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
const t = setInterval(() => {
|
||||
if (markBoot()) clearInterval(t);
|
||||
}, 5);
|
||||
|
||||
// Observe Long Tasks API (supported on Chromium)
|
||||
try {
|
||||
new PerformanceObserver((list) => {
|
||||
for (const entry of list.getEntries()) {
|
||||
window.__longTasksTotal += entry.duration;
|
||||
}
|
||||
}).observe({ type: 'longtask', buffered: true });
|
||||
} catch {}
|
||||
|
||||
// Observe LCP
|
||||
try {
|
||||
new PerformanceObserver((list) => {
|
||||
const entries = list.getEntries();
|
||||
if (entries.length > 0) {
|
||||
window.__lcp = entries[entries.length - 1].startTime;
|
||||
}
|
||||
}).observe({ type: 'largest-contentful-paint', buffered: true });
|
||||
} catch {}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Scenario: Cold Load
|
||||
*/
|
||||
async function runColdScenario({ browserType, server, proxy, profile }) {
|
||||
const isWebKit = browserType.name() === 'webkit';
|
||||
const launchArgs = isWebKit ? [] : ['--autoplay-policy=no-user-gesture-required'];
|
||||
const browser = await browserType.launch({
|
||||
headless: true,
|
||||
args: launchArgs,
|
||||
});
|
||||
|
||||
const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } };
|
||||
const context = await browser.newContext(contextOptions);
|
||||
const page = await context.newPage();
|
||||
|
||||
// If Chromium, configure CDP Network conditions and 4x CPU throttle
|
||||
if (!isWebKit) {
|
||||
const cdp = await context.newCDPSession(page);
|
||||
await cdp.send('Network.enable');
|
||||
const isLossy = profile === 'lossy';
|
||||
await cdp.send('Network.emulateNetworkConditions', {
|
||||
offline: false,
|
||||
latency: isLossy ? 250 : 150,
|
||||
downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8,
|
||||
uploadThroughput: (750 * 1000) / 8,
|
||||
connectionType: 'cellular4g',
|
||||
});
|
||||
await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 });
|
||||
}
|
||||
|
||||
await setupPageMetrics(page);
|
||||
|
||||
proxy.resetStats();
|
||||
proxy.startTracking();
|
||||
|
||||
const swInstallStart = Date.now();
|
||||
const navUrl = `http://127.0.0.1:${server.port}/?v=benchmedia01`;
|
||||
|
||||
await page.goto(navUrl);
|
||||
|
||||
// Wait for boot landmark
|
||||
await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 });
|
||||
|
||||
// Check deterministic media playing-ready
|
||||
let mediaReadyMs = null;
|
||||
let autoplayBlocked = false;
|
||||
try {
|
||||
const mediaCheck = await page.waitForFunction(() => {
|
||||
const v = document.querySelector('video') || document.querySelector('audio');
|
||||
if (!v) return false;
|
||||
return v.readyState >= 3 || (!v.paused && v.currentTime > 0);
|
||||
}, { timeout: 8000 });
|
||||
if (mediaCheck) {
|
||||
const state = await page.evaluate(() => {
|
||||
const v = document.querySelector('video') || document.querySelector('audio');
|
||||
return { paused: v.paused, readyState: v.readyState };
|
||||
});
|
||||
autoplayBlocked = state.paused;
|
||||
mediaReadyMs = await page.evaluate(() => performance.now());
|
||||
}
|
||||
} catch {
|
||||
autoplayBlocked = true;
|
||||
}
|
||||
|
||||
// Wait for initial Service Worker activation
|
||||
await page.evaluate(async () => {
|
||||
const reg = await navigator.serviceWorker.ready;
|
||||
if (reg.active && reg.active.state === 'activated') return;
|
||||
await new Promise((resolve) => {
|
||||
const w = reg.active || reg.installing;
|
||||
if (w.state === 'activated') return resolve();
|
||||
w.addEventListener('statechange', () => {
|
||||
if (w.state === 'activated') resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
const swInstallDuration = Date.now() - swInstallStart;
|
||||
proxy.stopTracking();
|
||||
|
||||
const stats = proxy.getStats();
|
||||
|
||||
const pageMetrics = await page.evaluate(() => {
|
||||
const nav = performance.getEntriesByType('navigation')[0];
|
||||
const paint = performance.getEntriesByType('paint');
|
||||
const fcp = paint.find((p) => p.name === 'first-contentful-paint');
|
||||
|
||||
return {
|
||||
fcp: fcp ? Math.round(fcp.startTime) : null,
|
||||
lcp: window.__lcp ? Math.round(window.__lcp) : null,
|
||||
domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null,
|
||||
bootDone: window.__bootDone ? Math.round(window.__bootDone) : null,
|
||||
longTaskTotal: Math.round(window.__longTasksTotal || 0),
|
||||
};
|
||||
});
|
||||
|
||||
const storage = await page.evaluate(async () => {
|
||||
if (navigator.storage && navigator.storage.estimate) {
|
||||
const est = await navigator.storage.estimate();
|
||||
return { usage: est.usage || 0, quota: est.quota || 0 };
|
||||
}
|
||||
return { usage: null, quota: null };
|
||||
});
|
||||
|
||||
await browser.close();
|
||||
|
||||
return {
|
||||
requestCount: stats.requestCount,
|
||||
wireBytes: stats.totalWireBytes,
|
||||
wireBytesByType: stats.bytesByType,
|
||||
fcp: pageMetrics.fcp,
|
||||
lcp: pageMetrics.lcp,
|
||||
domContentLoaded: pageMetrics.domContentLoaded,
|
||||
bootDone: pageMetrics.bootDone,
|
||||
longTaskTotal: pageMetrics.longTaskTotal,
|
||||
swInstallDuration,
|
||||
swInstallBytes: stats.totalWireBytes,
|
||||
storageEstimate: storage,
|
||||
mediaReady: mediaReadyMs ? Math.round(mediaReadyMs) : null,
|
||||
autoplayBlocked,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Scenario: Warm Load (re-navigation with active SW)
|
||||
*/
|
||||
async function runWarmScenario({ browserType, server, proxy, profile }) {
|
||||
const isWebKit = browserType.name() === 'webkit';
|
||||
const launchArgs = isWebKit ? [] : ['--autoplay-policy=no-user-gesture-required'];
|
||||
const browser = await browserType.launch({
|
||||
headless: true,
|
||||
args: launchArgs,
|
||||
});
|
||||
|
||||
const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } };
|
||||
const context = await browser.newContext(contextOptions);
|
||||
const page = await context.newPage();
|
||||
|
||||
if (!isWebKit) {
|
||||
const cdp = await context.newCDPSession(page);
|
||||
await cdp.send('Network.enable');
|
||||
const isLossy = profile === 'lossy';
|
||||
await cdp.send('Network.emulateNetworkConditions', {
|
||||
offline: false,
|
||||
latency: isLossy ? 250 : 150,
|
||||
downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8,
|
||||
uploadThroughput: (750 * 1000) / 8,
|
||||
connectionType: 'cellular4g',
|
||||
});
|
||||
await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 });
|
||||
}
|
||||
|
||||
await setupPageMetrics(page);
|
||||
|
||||
const navUrl = `http://127.0.0.1:${server.port}/`;
|
||||
|
||||
// Prime context with cold load
|
||||
await page.goto(navUrl);
|
||||
await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 });
|
||||
await page.evaluate(async () => {
|
||||
const reg = await navigator.serviceWorker.ready;
|
||||
if (reg.active && reg.active.state === 'activated') return;
|
||||
await new Promise((resolve) => {
|
||||
const w = reg.active || reg.installing;
|
||||
if (w.state === 'activated') return resolve();
|
||||
w.addEventListener('statechange', () => {
|
||||
if (w.state === 'activated') resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// Now measure warm re-navigation
|
||||
proxy.resetStats();
|
||||
proxy.startTracking();
|
||||
|
||||
await page.goto(navUrl);
|
||||
await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 });
|
||||
|
||||
proxy.stopTracking();
|
||||
const stats = proxy.getStats();
|
||||
|
||||
const pageMetrics = await page.evaluate(() => {
|
||||
const nav = performance.getEntriesByType('navigation')[0];
|
||||
const paint = performance.getEntriesByType('paint');
|
||||
const fcp = paint.find((p) => p.name === 'first-contentful-paint');
|
||||
|
||||
return {
|
||||
fcp: fcp ? Math.round(fcp.startTime) : null,
|
||||
lcp: window.__lcp ? Math.round(window.__lcp) : null,
|
||||
domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null,
|
||||
bootDone: window.__bootDone ? Math.round(window.__bootDone) : null,
|
||||
longTaskTotal: Math.round(window.__longTasksTotal || 0),
|
||||
};
|
||||
});
|
||||
|
||||
await browser.close();
|
||||
|
||||
return {
|
||||
requestCount: stats.requestCount,
|
||||
wireBytes: stats.totalWireBytes,
|
||||
wireBytesByType: stats.bytesByType,
|
||||
fcp: pageMetrics.fcp,
|
||||
lcp: pageMetrics.lcp,
|
||||
domContentLoaded: pageMetrics.domContentLoaded,
|
||||
bootDone: pageMetrics.bootDone,
|
||||
longTaskTotal: pageMetrics.longTaskTotal,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Scenario: Offline Reload under SW
|
||||
*/
|
||||
async function runOfflineScenario({ browserType, server, proxy, profile }) {
|
||||
const browser = await browserType.launch({ headless: true });
|
||||
const isWebKit = browserType.name() === 'webkit';
|
||||
const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } };
|
||||
const context = await browser.newContext(contextOptions);
|
||||
const page = await context.newPage();
|
||||
|
||||
await setupPageMetrics(page);
|
||||
const navUrl = `http://127.0.0.1:${server.port}/`;
|
||||
|
||||
// Prime SW
|
||||
await page.goto(navUrl);
|
||||
await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 });
|
||||
await page.evaluate(async () => {
|
||||
const reg = await navigator.serviceWorker.ready;
|
||||
if (reg.active && reg.active.state === 'activated') return;
|
||||
await new Promise((resolve) => {
|
||||
const w = reg.active || reg.installing;
|
||||
if (w.state === 'activated') return resolve();
|
||||
w.addEventListener('statechange', () => {
|
||||
if (w.state === 'activated') resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// Switch to offline mode:
|
||||
// On Chromium, context.setOffline(true) works.
|
||||
// On Linux WebKit, Playwright context.setOffline throws an internal browser error,
|
||||
// so proxy.setOffline(true) cuts upstream network to simulate offline truthfully.
|
||||
if (!isWebKit) {
|
||||
await context.setOffline(true);
|
||||
}
|
||||
proxy.setOffline(true);
|
||||
|
||||
proxy.resetStats();
|
||||
proxy.startTracking();
|
||||
|
||||
const t0 = Date.now();
|
||||
await page.reload();
|
||||
await page.waitForFunction(() => window.__bootDone !== null, { timeout: 15000 });
|
||||
const reloadDuration = Date.now() - t0;
|
||||
|
||||
proxy.stopTracking();
|
||||
proxy.setOffline(false);
|
||||
const stats = proxy.getStats();
|
||||
|
||||
const pageMetrics = await page.evaluate(() => {
|
||||
const nav = performance.getEntriesByType('navigation')[0];
|
||||
const paint = performance.getEntriesByType('paint');
|
||||
const fcp = paint.find((p) => p.name === 'first-contentful-paint');
|
||||
|
||||
return {
|
||||
fcp: fcp ? Math.round(fcp.startTime) : null,
|
||||
lcp: window.__lcp ? Math.round(window.__lcp) : null,
|
||||
domContentLoaded: nav ? Math.round(nav.domContentLoadedEventEnd - nav.startTime) : null,
|
||||
bootDone: window.__bootDone ? Math.round(window.__bootDone) : null,
|
||||
};
|
||||
});
|
||||
|
||||
await browser.close();
|
||||
|
||||
return {
|
||||
requestCount: stats.requestCount,
|
||||
wireBytes: stats.totalWireBytes,
|
||||
fcp: pageMetrics.fcp,
|
||||
lcp: pageMetrics.lcp,
|
||||
domContentLoaded: pageMetrics.domContentLoaded,
|
||||
bootDone: pageMetrics.bootDone,
|
||||
reloadDuration,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Scenario: Update N -> N+1
|
||||
*/
|
||||
async function runUpdateScenario({ browserType, server, proxy, profile, updateKind, trees }) {
|
||||
const browser = await browserType.launch({ headless: true });
|
||||
const isWebKit = browserType.name() === 'webkit';
|
||||
const contextOptions = { proxy: { server: `http://127.0.0.1:${proxy.port}` } };
|
||||
const context = await browser.newContext(contextOptions);
|
||||
const page = await context.newPage();
|
||||
|
||||
if (!isWebKit) {
|
||||
const cdp = await context.newCDPSession(page);
|
||||
await cdp.send('Network.enable');
|
||||
const isLossy = profile === 'lossy';
|
||||
await cdp.send('Network.emulateNetworkConditions', {
|
||||
offline: false,
|
||||
latency: isLossy ? 250 : 150,
|
||||
downloadThroughput: isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8,
|
||||
uploadThroughput: (750 * 1000) / 8,
|
||||
connectionType: 'cellular4g',
|
||||
});
|
||||
}
|
||||
|
||||
// Start on Tree N (base)
|
||||
await server.restartWithPublic(trees.treeBase);
|
||||
|
||||
const navUrl = `http://127.0.0.1:${server.port}/`;
|
||||
|
||||
await page.goto(navUrl);
|
||||
|
||||
// Wait for steady state: active worker present, no waiting worker
|
||||
while (true) {
|
||||
const ready = await page.evaluate(async () => {
|
||||
const reg = await navigator.serviceWorker.getRegistration();
|
||||
return reg?.active?.state === 'activated' && !reg?.installing && !reg?.waiting;
|
||||
});
|
||||
if (ready) break;
|
||||
await sleep(50);
|
||||
}
|
||||
|
||||
// Switch server to Tree N+1 (app.js or theme-glass.css modification)
|
||||
const targetTree = updateKind === 'js' ? trees.treeUpdateJs : trees.treeUpdateCss;
|
||||
await server.restartWithPublic(targetTree);
|
||||
|
||||
// Measure wire bytes and duration from reg.update() until new SW is waiting/installed
|
||||
proxy.resetStats();
|
||||
proxy.startTracking();
|
||||
|
||||
const t0 = Date.now();
|
||||
await page.evaluate(() => navigator.serviceWorker.getRegistration().then((r) => r.update()));
|
||||
|
||||
// Wait until new worker reaches waiting state
|
||||
while (true) {
|
||||
const waiting = await page.evaluate(async () => {
|
||||
const reg = await navigator.serviceWorker.getRegistration();
|
||||
return reg?.waiting?.state === 'installed';
|
||||
});
|
||||
if (waiting) break;
|
||||
await sleep(50);
|
||||
}
|
||||
|
||||
const duration = Date.now() - t0;
|
||||
proxy.stopTracking();
|
||||
const stats = proxy.getStats();
|
||||
|
||||
await browser.close();
|
||||
|
||||
return {
|
||||
updateKind,
|
||||
duration,
|
||||
requestCount: stats.requestCount,
|
||||
wireBytes: stats.totalWireBytes,
|
||||
wireBytesByType: stats.bytesByType,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Main execution
|
||||
*/
|
||||
async function main() {
|
||||
const options = parseArgs();
|
||||
console.log('=== YTPlayer Performance Baseline Harness ===');
|
||||
console.log(`Runs per scenario: ${options.runs}`);
|
||||
console.log(`Browser target: ${options.browser}`);
|
||||
console.log(`Network profile: ${options.profile}`);
|
||||
console.log(`Scenario target: ${options.scenario}`);
|
||||
|
||||
const browsers = [];
|
||||
if (options.browser === 'all' || options.browser === 'chromium') browsers.push(pw.chromium);
|
||||
if (options.browser === 'all' || options.browser === 'webkit') browsers.push(pw.webkit);
|
||||
|
||||
const profiles = [];
|
||||
if (options.profile === 'all' || options.profile === 'lte') profiles.push('lte');
|
||||
if (options.profile === 'all' || options.profile === 'lossy') profiles.push('lossy');
|
||||
|
||||
const tmpRoot = path.join(REPO_ROOT, 'perf', '.tmp', `bench-${Date.now()}`);
|
||||
mkdirSync(tmpRoot, { recursive: true });
|
||||
|
||||
// Setup scratch trees for N -> N+1 update scenarios
|
||||
const treeBase = path.join(tmpRoot, 'tree-base');
|
||||
const treeUpdateJs = path.join(tmpRoot, 'tree-update-js');
|
||||
const treeUpdateCss = path.join(tmpRoot, 'tree-update-css');
|
||||
|
||||
cpSync(path.join(REPO_ROOT, 'frontend'), treeBase, { recursive: true });
|
||||
cpSync(path.join(REPO_ROOT, 'frontend'), treeUpdateJs, { recursive: true });
|
||||
cpSync(path.join(REPO_ROOT, 'frontend'), treeUpdateCss, { recursive: true });
|
||||
|
||||
// Append 1 comment line to app.js in treeUpdateJs
|
||||
appendFileSync(path.join(treeUpdateJs, 'app.js'), '\n// baseline bench: 1-line app.js comment\n');
|
||||
// Append 1 rule to theme-glass.css in treeUpdateCss
|
||||
appendFileSync(path.join(treeUpdateCss, 'theme-glass.css'), '\n/* baseline bench: 1-line css rule */\n.baseline-marker { opacity: 0.99; }\n');
|
||||
|
||||
const trees = { treeBase, treeUpdateJs, treeUpdateCss };
|
||||
|
||||
const serverScratch = path.join(tmpRoot, 'server-run');
|
||||
const server = new ServerManager(serverScratch, treeBase);
|
||||
await server.start();
|
||||
console.log(`Bun server running on ephemeral port ${server.port}`);
|
||||
|
||||
const results = {
|
||||
date: new Date().toISOString(),
|
||||
commit: spawnSync('git', ['rev-parse', 'HEAD'], { cwd: REPO_ROOT }).stdout.toString().trim(),
|
||||
runsConfigured: options.runs,
|
||||
browsers: {},
|
||||
};
|
||||
|
||||
try {
|
||||
for (const bType of browsers) {
|
||||
const bName = bType.name();
|
||||
results.browsers[bName] = {};
|
||||
|
||||
for (const prof of profiles) {
|
||||
console.log(`\n>>> Running Benchmark: Browser=[${bName}] Profile=[${prof}]`);
|
||||
results.browsers[bName][prof] = {};
|
||||
|
||||
const proxyProfile = bName === 'chromium' ? 'unthrottled' : prof;
|
||||
const proxy = createThrottleProxy({ targetPort: server.port, profile: proxyProfile });
|
||||
proxy.port = await proxy.listen();
|
||||
console.log(`Proxy listening on port ${proxy.port} (mode: ${proxyProfile})`);
|
||||
|
||||
// 1. Cold Scenario
|
||||
if (options.scenario === 'all' || options.scenario === 'cold') {
|
||||
console.log(` [Scenario: Cold Load] running ${options.runs} iterations...`);
|
||||
const coldRuns = [];
|
||||
for (let r = 0; r < options.runs; r++) {
|
||||
process.stdout.write(` run ${r + 1}/${options.runs}... `);
|
||||
const res = await runColdScenario({ browserType: bType, server, proxy, profile: prof });
|
||||
coldRuns.push(res);
|
||||
console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, boot=${res.bootDone}ms)`);
|
||||
}
|
||||
|
||||
results.browsers[bName][prof].cold = {
|
||||
requestCount: summarizeList(coldRuns.map((r) => r.requestCount)),
|
||||
wireBytes: summarizeList(coldRuns.map((r) => r.wireBytes)),
|
||||
fcp: summarizeList(coldRuns.map((r) => r.fcp)),
|
||||
lcp: summarizeList(coldRuns.map((r) => r.lcp)),
|
||||
domContentLoaded: summarizeList(coldRuns.map((r) => r.domContentLoaded)),
|
||||
bootDone: summarizeList(coldRuns.map((r) => r.bootDone)),
|
||||
longTaskTotal: summarizeList(coldRuns.map((r) => r.longTaskTotal)),
|
||||
swInstallDuration: summarizeList(coldRuns.map((r) => r.swInstallDuration)),
|
||||
swInstallBytes: summarizeList(coldRuns.map((r) => r.swInstallBytes)),
|
||||
storageEstimate: coldRuns[0]?.storageEstimate || null,
|
||||
mediaReady: summarizeList(coldRuns.map((r) => r.mediaReady)),
|
||||
autoplayBlocked: coldRuns.some((r) => r.autoplayBlocked),
|
||||
wireBytesByType: coldRuns[0]?.wireBytesByType || {},
|
||||
};
|
||||
}
|
||||
|
||||
// 2. Warm Scenario
|
||||
if (options.scenario === 'all' || options.scenario === 'warm') {
|
||||
console.log(` [Scenario: Warm Load] running ${options.runs} iterations...`);
|
||||
const warmRuns = [];
|
||||
for (let r = 0; r < options.runs; r++) {
|
||||
process.stdout.write(` run ${r + 1}/${options.runs}... `);
|
||||
const res = await runWarmScenario({ browserType: bType, server, proxy, profile: prof });
|
||||
warmRuns.push(res);
|
||||
console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, boot=${res.bootDone}ms)`);
|
||||
}
|
||||
|
||||
results.browsers[bName][prof].warm = {
|
||||
requestCount: summarizeList(warmRuns.map((r) => r.requestCount)),
|
||||
wireBytes: summarizeList(warmRuns.map((r) => r.wireBytes)),
|
||||
fcp: summarizeList(warmRuns.map((r) => r.fcp)),
|
||||
lcp: summarizeList(warmRuns.map((r) => r.lcp)),
|
||||
domContentLoaded: summarizeList(warmRuns.map((r) => r.domContentLoaded)),
|
||||
bootDone: summarizeList(warmRuns.map((r) => r.bootDone)),
|
||||
longTaskTotal: summarizeList(warmRuns.map((r) => r.longTaskTotal)),
|
||||
};
|
||||
}
|
||||
|
||||
// 3. Offline Scenario
|
||||
if (options.scenario === 'all' || options.scenario === 'offline') {
|
||||
console.log(` [Scenario: Offline Reload] running ${options.runs} iterations...`);
|
||||
const offRuns = [];
|
||||
for (let r = 0; r < options.runs; r++) {
|
||||
process.stdout.write(` run ${r + 1}/${options.runs}... `);
|
||||
const res = await runOfflineScenario({ browserType: bType, server, proxy, profile: prof });
|
||||
offRuns.push(res);
|
||||
console.log(`done (reload=${res.reloadDuration}ms, boot=${res.bootDone}ms)`);
|
||||
}
|
||||
|
||||
results.browsers[bName][prof].offline = {
|
||||
requestCount: summarizeList(offRuns.map((r) => r.requestCount)),
|
||||
wireBytes: summarizeList(offRuns.map((r) => r.wireBytes)),
|
||||
fcp: summarizeList(offRuns.map((r) => r.fcp)),
|
||||
lcp: summarizeList(offRuns.map((r) => r.lcp)),
|
||||
domContentLoaded: summarizeList(offRuns.map((r) => r.domContentLoaded)),
|
||||
bootDone: summarizeList(offRuns.map((r) => r.bootDone)),
|
||||
reloadDuration: summarizeList(offRuns.map((r) => r.reloadDuration)),
|
||||
};
|
||||
}
|
||||
|
||||
// 4. Update N -> N+1 (JS)
|
||||
if (options.scenario === 'all' || options.scenario === 'update-js') {
|
||||
console.log(` [Scenario: Update N->N+1 (1-line app.js)] running ${options.runs} iterations...`);
|
||||
const updJsRuns = [];
|
||||
for (let r = 0; r < options.runs; r++) {
|
||||
process.stdout.write(` run ${r + 1}/${options.runs}... `);
|
||||
const res = await runUpdateScenario({
|
||||
browserType: bType,
|
||||
server,
|
||||
proxy,
|
||||
profile: prof,
|
||||
updateKind: 'js',
|
||||
trees,
|
||||
});
|
||||
updJsRuns.push(res);
|
||||
console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, dur=${res.duration}ms)`);
|
||||
}
|
||||
|
||||
results.browsers[bName][prof].updateJs = {
|
||||
requestCount: summarizeList(updJsRuns.map((r) => r.requestCount)),
|
||||
wireBytes: summarizeList(updJsRuns.map((r) => r.wireBytes)),
|
||||
duration: summarizeList(updJsRuns.map((r) => r.duration)),
|
||||
wireBytesByType: updJsRuns[0]?.wireBytesByType || {},
|
||||
};
|
||||
}
|
||||
|
||||
// 5. Update N -> N+1 (CSS)
|
||||
if (options.scenario === 'all' || options.scenario === 'update-css') {
|
||||
console.log(` [Scenario: Update N->N+1 (1-line theme-glass.css)] running ${options.runs} iterations...`);
|
||||
const updCssRuns = [];
|
||||
for (let r = 0; r < options.runs; r++) {
|
||||
process.stdout.write(` run ${r + 1}/${options.runs}... `);
|
||||
const res = await runUpdateScenario({
|
||||
browserType: bType,
|
||||
server,
|
||||
proxy,
|
||||
profile: prof,
|
||||
updateKind: 'css',
|
||||
trees,
|
||||
});
|
||||
updCssRuns.push(res);
|
||||
console.log(`done (${res.wireBytes} bytes, ${res.requestCount} reqs, dur=${res.duration}ms)`);
|
||||
}
|
||||
|
||||
results.browsers[bName][prof].updateCss = {
|
||||
requestCount: summarizeList(updCssRuns.map((r) => r.requestCount)),
|
||||
wireBytes: summarizeList(updCssRuns.map((r) => r.wireBytes)),
|
||||
duration: summarizeList(updCssRuns.map((r) => r.duration)),
|
||||
wireBytesByType: updCssRuns[0]?.wireBytesByType || {},
|
||||
};
|
||||
}
|
||||
|
||||
await proxy.close();
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await server.stop();
|
||||
rmSync(tmpRoot, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
// Write results JSON
|
||||
const now = new Date();
|
||||
const dateStr = `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, '0')}-${String(now.getDate()).padStart(2, '0')}`;
|
||||
const outPath = options.out || path.join(REPO_ROOT, 'perf', 'results', `baseline-${dateStr}.json`);
|
||||
mkdirSync(path.dirname(outPath), { recursive: true });
|
||||
writeFileSync(outPath, JSON.stringify(results, null, 2));
|
||||
console.log(`\nBaseline results saved to: ${outPath}`);
|
||||
|
||||
// Print Summary Table
|
||||
printSummaryTable(results);
|
||||
|
||||
// Compare mode if requested
|
||||
if (options.compare) {
|
||||
compareResults(options.compare, results);
|
||||
}
|
||||
}
|
||||
|
||||
function printSummaryTable(results) {
|
||||
console.log('\n============================= HEADLINE BASELINE SUMMARY =============================');
|
||||
for (const [browser, profiles] of Object.entries(results.browsers)) {
|
||||
for (const [profile, scenarios] of Object.entries(profiles)) {
|
||||
console.log(`\n[Browser: ${browser} | Profile: ${profile}]`);
|
||||
console.log('| Scenario | Req Count | Wire Bytes | Boot / Ready | SW Dur / Reload |');
|
||||
console.log('|---------------------------|-----------|------------|--------------|-----------------|');
|
||||
|
||||
if (scenarios.cold) {
|
||||
const c = scenarios.cold;
|
||||
console.log(
|
||||
`| Cold First Load | ${String(c.requestCount.median).padEnd(9)} | ${formatBytes(c.wireBytes.median).padEnd(10)} | ${String(c.bootDone.median + 'ms').padEnd(12)} | ${String(c.swInstallDuration.median + 'ms').padEnd(15)} |`
|
||||
);
|
||||
}
|
||||
if (scenarios.warm) {
|
||||
const w = scenarios.warm;
|
||||
console.log(
|
||||
`| Warm Reload (under SW) | ${String(w.requestCount.median).padEnd(9)} | ${formatBytes(w.wireBytes.median).padEnd(10)} | ${String(w.bootDone.median + 'ms').padEnd(12)} | ${'-'.padEnd(15)} |`
|
||||
);
|
||||
}
|
||||
if (scenarios.offline) {
|
||||
const o = scenarios.offline;
|
||||
console.log(
|
||||
`| Offline Reload (under SW) | ${String(o.requestCount.median).padEnd(9)} | ${formatBytes(o.wireBytes.median).padEnd(10)} | ${String(o.bootDone.median + 'ms').padEnd(12)} | ${String(o.reloadDuration.median + 'ms').padEnd(15)} |`
|
||||
);
|
||||
}
|
||||
if (scenarios.updateJs) {
|
||||
const u = scenarios.updateJs;
|
||||
console.log(
|
||||
`| Update N->N+1 (1-line JS) | ${String(u.requestCount.median).padEnd(9)} | ${formatBytes(u.wireBytes.median).padEnd(10)} | ${'-'.padEnd(12)} | ${String(u.duration.median + 'ms').padEnd(15)} |`
|
||||
);
|
||||
}
|
||||
if (scenarios.updateCss) {
|
||||
const u = scenarios.updateCss;
|
||||
console.log(
|
||||
`| Update N->N+1 (1-line CSS)| ${String(u.requestCount.median).padEnd(9)} | ${formatBytes(u.wireBytes.median).padEnd(10)} | ${'-'.padEnd(12)} | ${String(u.duration.median + 'ms').padEnd(15)} |`
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log('\n=====================================================================================');
|
||||
}
|
||||
|
||||
function formatBytes(bytes) {
|
||||
if (bytes === null || bytes === undefined) return 'N/A';
|
||||
if (bytes >= 1024 * 1024) return (bytes / (1024 * 1024)).toFixed(2) + ' MB';
|
||||
if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB';
|
||||
return bytes + ' B';
|
||||
}
|
||||
|
||||
function compareResults(baselinePath, currentResults) {
|
||||
console.log(`\n================ Phase 5 Comparison against: ${baselinePath} ================`);
|
||||
if (!existsSync(baselinePath)) {
|
||||
console.error(`Error: Baseline file not found: ${baselinePath}`);
|
||||
return;
|
||||
}
|
||||
const base = JSON.parse(readFileSync(baselinePath, 'utf8'));
|
||||
|
||||
for (const [browser, profiles] of Object.entries(currentResults.browsers)) {
|
||||
for (const [profile, scenarios] of Object.entries(profiles)) {
|
||||
console.log(`\n[Comparison: ${browser} / ${profile}]`);
|
||||
console.log('| Metric | Baseline | Current | Delta (%) |');
|
||||
console.log('|------------------------------------|-------------|-------------|--------------|');
|
||||
|
||||
const baseScen = base.browsers?.[browser]?.[profile];
|
||||
if (!baseScen) {
|
||||
console.log('| (No matching baseline data) | | | |');
|
||||
continue;
|
||||
}
|
||||
|
||||
const rows = [
|
||||
['Cold: Wire Bytes', baseScen.cold?.wireBytes?.median, scenarios.cold?.wireBytes?.median, true],
|
||||
['Cold: Boot Done (ms)', baseScen.cold?.bootDone?.median, scenarios.cold?.bootDone?.median, false],
|
||||
['Cold: Requests', baseScen.cold?.requestCount?.median, scenarios.cold?.requestCount?.median, false],
|
||||
['Warm: Boot Done (ms)', baseScen.warm?.bootDone?.median, scenarios.warm?.bootDone?.median, false],
|
||||
['Update (1-line JS): Wire Bytes', baseScen.updateJs?.wireBytes?.median, scenarios.updateJs?.wireBytes?.median, true],
|
||||
['Update (1-line JS): Time (ms)', baseScen.updateJs?.duration?.median, scenarios.updateJs?.duration?.median, false],
|
||||
['Update (1-line CSS): Wire Bytes', baseScen.updateCss?.wireBytes?.median, scenarios.updateCss?.wireBytes?.median, true],
|
||||
['Update (1-line CSS): Time (ms)', baseScen.updateCss?.duration?.median, scenarios.updateCss?.duration?.median, false],
|
||||
];
|
||||
|
||||
for (const [name, bVal, cVal, isBytes] of rows) {
|
||||
if (bVal === undefined || cVal === undefined) continue;
|
||||
const bStr = isBytes ? formatBytes(bVal) : bVal !== null ? `${bVal}` : 'N/A';
|
||||
const cStr = isBytes ? formatBytes(cVal) : cVal !== null ? `${cVal}` : 'N/A';
|
||||
let deltaStr = '-';
|
||||
if (typeof bVal === 'number' && typeof cVal === 'number' && bVal > 0) {
|
||||
const pct = (((cVal - bVal) / bVal) * 100).toFixed(1);
|
||||
const sign = cVal >= bVal ? '+' : '';
|
||||
deltaStr = `${sign}${pct}%`;
|
||||
}
|
||||
console.log(`| ${name.padEnd(34)} | ${bStr.padEnd(11)} | ${cStr.padEnd(11)} | ${deltaStr.padEnd(12)} |`);
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log('==============================================================================');
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error('Benchmark fatal error:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
126
perf/make-shell-fixture.sh
Executable file
126
perf/make-shell-fixture.sh
Executable file
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# perf/make-shell-fixture.sh
|
||||
# Archives frontend from <commit> into perf/fixtures/shell-<tag>/,
|
||||
# computes the build tag the way server.js computeBuildTag() does,
|
||||
# and verifies that it matches expected-tag (or live prod tag).
|
||||
# ==============================================================================
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
echo "Usage: $0 <commit> [expected-tag]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
COMMIT="$1"
|
||||
EXPECTED_TAG="${2:-}"
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
PERF_DIR="$REPO_ROOT/perf"
|
||||
FIXTURES_DIR="$PERF_DIR/fixtures"
|
||||
TMP_DIR="$PERF_DIR/.tmp"
|
||||
|
||||
mkdir -p "$FIXTURES_DIR" "$TMP_DIR"
|
||||
|
||||
PROD_URL="https://worship.hesed.sbs/api/version"
|
||||
|
||||
fetch_prod_tag() {
|
||||
curl -s -S --max-time 10 "$PROD_URL" | node -e '
|
||||
let data = "";
|
||||
process.stdin.on("data", chunk => { data += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
try {
|
||||
const j = JSON.parse(data);
|
||||
if (j.buildTag) {
|
||||
process.stdout.write(j.buildTag);
|
||||
process.exit(0);
|
||||
}
|
||||
} catch {}
|
||||
process.exit(1);
|
||||
});
|
||||
' 2>/dev/null || echo ""
|
||||
}
|
||||
|
||||
# Verify the commit exists in git
|
||||
if ! git rev-parse --verify "$COMMIT^{commit}" >/dev/null 2>&1; then
|
||||
echo "Error: commit '$COMMIT' not found in git repository" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RESOLVED_COMMIT="$(git rev-parse --short=7 "$COMMIT")"
|
||||
echo "Extracting frontend from commit $RESOLVED_COMMIT..."
|
||||
|
||||
STAGING_DIR="$(mktemp -d "$TMP_DIR/fixture-staging-XXXXXX")"
|
||||
cleanup() {
|
||||
rm -rf "$STAGING_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# Archive frontend tree into staging dir
|
||||
git archive "$COMMIT" frontend | tar -x -C "$STAGING_DIR" --strip-components=1
|
||||
|
||||
# Compute build tag using the exact logic from server/server.js computeBuildTag()
|
||||
COMPUTED_TAG=$(node -e '
|
||||
const { createHash } = require("crypto");
|
||||
const { readdirSync, statSync, readFileSync } = require("fs");
|
||||
|
||||
function computeTag(dir) {
|
||||
const hash = createHash("sha256");
|
||||
const walk = (d, prefix) => {
|
||||
for (const name of readdirSync(d).sort()) {
|
||||
const full = `${d}/${name}`;
|
||||
const rel = `${prefix}/${name}`;
|
||||
if (statSync(full).isDirectory()) {
|
||||
walk(full, rel);
|
||||
} else {
|
||||
hash.update(rel);
|
||||
hash.update(readFileSync(full));
|
||||
}
|
||||
}
|
||||
};
|
||||
walk(dir, "./public");
|
||||
return hash.digest("hex").slice(0, 12);
|
||||
}
|
||||
|
||||
const tag = computeTag(process.argv[1]);
|
||||
process.stdout.write(tag);
|
||||
' "$STAGING_DIR")
|
||||
|
||||
echo "Computed build tag for $RESOLVED_COMMIT: $COMPUTED_TAG"
|
||||
|
||||
# Determine target expected tag
|
||||
if [ -z "$EXPECTED_TAG" ]; then
|
||||
echo "Querying live production tag from $PROD_URL..."
|
||||
EXPECTED_TAG="$(fetch_prod_tag)"
|
||||
if [ -z "$EXPECTED_TAG" ]; then
|
||||
echo "Error: failed to fetch live buildTag from $PROD_URL" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Live production build tag: $EXPECTED_TAG"
|
||||
|
||||
# If mismatch, deploy might be in progress: wait 5 minutes and retry once
|
||||
if [ "$COMPUTED_TAG" != "$EXPECTED_TAG" ]; then
|
||||
echo "Tag mismatch detected (computed=$COMPUTED_TAG vs prod=$EXPECTED_TAG)."
|
||||
echo "Deploy may be in progress. Waiting 300 seconds to retry once..."
|
||||
sleep 300
|
||||
EXPECTED_TAG="$(fetch_prod_tag)"
|
||||
echo "Retried production build tag: $EXPECTED_TAG"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$COMPUTED_TAG" != "$EXPECTED_TAG" ]; then
|
||||
echo "Error: build tag mismatch! Computed '$COMPUTED_TAG' does not match expected '$EXPECTED_TAG'" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TARGET_FIXTURE_DIR="$FIXTURES_DIR/shell-$COMPUTED_TAG"
|
||||
echo "Installing fixture into $TARGET_FIXTURE_DIR..."
|
||||
rm -rf "$TARGET_FIXTURE_DIR"
|
||||
mkdir -p "$TARGET_FIXTURE_DIR"
|
||||
cp -a "$STAGING_DIR"/. "$TARGET_FIXTURE_DIR"/
|
||||
|
||||
FILE_COUNT=$(find "$TARGET_FIXTURE_DIR" -type f | wc -l)
|
||||
echo "Success: shell fixture created at $TARGET_FIXTURE_DIR with $FILE_COUNT files."
|
||||
echo "Verified commit: $RESOLVED_COMMIT"
|
||||
echo "Verified build tag: $COMPUTED_TAG"
|
||||
201
perf/proxy.mjs
Normal file
201
perf/proxy.mjs
Normal file
@@ -0,0 +1,201 @@
|
||||
import http from 'node:http';
|
||||
import { Transform } from 'node:stream';
|
||||
|
||||
/**
|
||||
* ThrottleTransform
|
||||
* Delays stream chunks to simulate bandwidth caps (e.g. 1.6 Mbps = 200,000 B/s).
|
||||
*/
|
||||
class ThrottleTransform extends Transform {
|
||||
constructor(bytesPerSec) {
|
||||
super();
|
||||
this.bytesPerSec = bytesPerSec;
|
||||
}
|
||||
|
||||
_transform(chunk, encoding, callback) {
|
||||
if (!this.bytesPerSec || this.bytesPerSec <= 0) {
|
||||
this.push(chunk);
|
||||
return callback();
|
||||
}
|
||||
const delayMs = (chunk.length / this.bytesPerSec) * 1000;
|
||||
setTimeout(() => {
|
||||
this.push(chunk);
|
||||
callback();
|
||||
}, Math.min(delayMs, 2000));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Categorize Content-Type into standard metric buckets.
|
||||
*/
|
||||
export function categorizeContentType(contentType, urlPath) {
|
||||
const ct = (contentType || '').toLowerCase();
|
||||
const path = (urlPath || '').toLowerCase();
|
||||
|
||||
if (ct.includes('text/html') || path.endsWith('.html')) return 'html';
|
||||
if (ct.includes('javascript') || path.endsWith('.js') || path.endsWith('.mjs')) return 'js';
|
||||
if (ct.includes('text/css') || path.endsWith('.css')) return 'css';
|
||||
if (ct.includes('font') || path.endsWith('.woff2') || path.endsWith('.woff') || path.endsWith('.ttf')) return 'fonts';
|
||||
if (ct.includes('image/') || path.endsWith('.png') || path.endsWith('.jpg') || path.endsWith('.svg') || path.endsWith('.webp') || path.endsWith('.ico')) return 'images';
|
||||
if (ct.includes('video/') || ct.includes('audio/') || path.endsWith('.mp4') || path.endsWith('.m4a') || path.endsWith('.mp3')) return 'media';
|
||||
if (ct.includes('application/json') || path.startsWith('/api/')) return 'api';
|
||||
return 'other';
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an HTTP throttling & stalling proxy for WebKit (and browser network tracing).
|
||||
* Supports both forward proxying and reverse proxying to targetPort.
|
||||
*/
|
||||
export function createThrottleProxy({ targetPort, profile = 'lte' }) {
|
||||
let activeTracking = false;
|
||||
let requestCount = 0;
|
||||
let totalWireBytes = 0;
|
||||
let bytesByType = {
|
||||
html: 0,
|
||||
js: 0,
|
||||
css: 0,
|
||||
fonts: 0,
|
||||
images: 0,
|
||||
media: 0,
|
||||
api: 0,
|
||||
other: 0,
|
||||
};
|
||||
|
||||
let reqIndex = 0;
|
||||
let isOfflineMode = false;
|
||||
|
||||
// Profile configuration:
|
||||
// LTE: 150ms roundtrip latency, 1.6 Mbps (200 KB/s) download, 750 kbps upload
|
||||
// Lossy: 250ms latency, 1.0 Mbps (125 KB/s) download, stall every 7th request by 1500ms
|
||||
// Unthrottled: 0 latency, 0 throttle (for Chromium where CDP throttles network/CPU)
|
||||
const isLossy = profile === 'lossy';
|
||||
const isUnthrottled = profile === 'unthrottled';
|
||||
const latencyMs = isUnthrottled ? 0 : (isLossy ? 250 : 150);
|
||||
const downloadBytesPerSec = isUnthrottled ? 0 : (isLossy ? (1.0 * 1000 * 1000) / 8 : (1.6 * 1000 * 1000) / 8);
|
||||
|
||||
const server = http.createServer((clientReq, clientRes) => {
|
||||
if (isOfflineMode) {
|
||||
clientReq.destroy();
|
||||
return;
|
||||
}
|
||||
|
||||
reqIndex++;
|
||||
const currentReqIndex = reqIndex;
|
||||
|
||||
let stallMs = 0;
|
||||
if (isLossy && currentReqIndex % 7 === 0) {
|
||||
stallMs = 1500; // Simulated lossy stall
|
||||
}
|
||||
|
||||
const totalDelay = isUnthrottled ? 0 : (latencyMs + stallMs);
|
||||
|
||||
const rawUrl = clientReq.url;
|
||||
const targetUrl = rawUrl.startsWith('http://') || rawUrl.startsWith('https://')
|
||||
? rawUrl
|
||||
: `http://127.0.0.1:${targetPort}${rawUrl}`;
|
||||
|
||||
const parsedTarget = new URL(targetUrl);
|
||||
const pathname = parsedTarget.pathname;
|
||||
|
||||
const options = {
|
||||
protocol: parsedTarget.protocol,
|
||||
hostname: parsedTarget.hostname,
|
||||
port: parsedTarget.port,
|
||||
path: parsedTarget.pathname + parsedTarget.search,
|
||||
method: clientReq.method,
|
||||
headers: { ...clientReq.headers, host: parsedTarget.host },
|
||||
};
|
||||
|
||||
const forward = () => {
|
||||
const upReq = http.request(options, (upRes) => {
|
||||
const ct = upRes.headers['content-type'] || '';
|
||||
const cat = categorizeContentType(ct, pathname);
|
||||
|
||||
if (activeTracking) {
|
||||
requestCount++;
|
||||
}
|
||||
|
||||
clientRes.writeHead(upRes.statusCode, upRes.headers);
|
||||
|
||||
if (isUnthrottled) {
|
||||
upRes.on('data', (chunk) => {
|
||||
if (activeTracking) {
|
||||
totalWireBytes += chunk.length;
|
||||
bytesByType[cat] = (bytesByType[cat] || 0) + chunk.length;
|
||||
}
|
||||
clientRes.write(chunk);
|
||||
});
|
||||
upRes.on('end', () => clientRes.end());
|
||||
} else {
|
||||
const throttler = new ThrottleTransform(downloadBytesPerSec);
|
||||
|
||||
throttler.on('data', (chunk) => {
|
||||
if (activeTracking) {
|
||||
totalWireBytes += chunk.length;
|
||||
bytesByType[cat] = (bytesByType[cat] || 0) + chunk.length;
|
||||
}
|
||||
});
|
||||
|
||||
upRes.pipe(throttler).pipe(clientRes);
|
||||
}
|
||||
});
|
||||
|
||||
upReq.on('error', (err) => {
|
||||
try {
|
||||
clientRes.statusCode = 502;
|
||||
clientRes.end(err.message);
|
||||
} catch {}
|
||||
});
|
||||
|
||||
clientReq.pipe(upReq);
|
||||
};
|
||||
|
||||
if (totalDelay > 0) {
|
||||
setTimeout(forward, totalDelay);
|
||||
} else {
|
||||
forward();
|
||||
}
|
||||
});
|
||||
|
||||
return {
|
||||
server,
|
||||
listen: () =>
|
||||
new Promise((resolve) => {
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const port = server.address().port;
|
||||
resolve(port);
|
||||
});
|
||||
}),
|
||||
close: () =>
|
||||
new Promise((resolve) => {
|
||||
server.close(resolve);
|
||||
}),
|
||||
startTracking: () => {
|
||||
activeTracking = true;
|
||||
},
|
||||
stopTracking: () => {
|
||||
activeTracking = false;
|
||||
},
|
||||
resetStats: () => {
|
||||
requestCount = 0;
|
||||
totalWireBytes = 0;
|
||||
bytesByType = {
|
||||
html: 0,
|
||||
js: 0,
|
||||
css: 0,
|
||||
fonts: 0,
|
||||
images: 0,
|
||||
media: 0,
|
||||
api: 0,
|
||||
other: 0,
|
||||
};
|
||||
},
|
||||
setOffline: (val) => {
|
||||
isOfflineMode = !!val;
|
||||
},
|
||||
getStats: () => ({
|
||||
requestCount,
|
||||
totalWireBytes,
|
||||
bytesByType: { ...bytesByType },
|
||||
}),
|
||||
};
|
||||
}
|
||||
1639
perf/results/baseline-2026-10-07.json
Normal file
1639
perf/results/baseline-2026-10-07.json
Normal file
File diff suppressed because it is too large
Load Diff
175
plans/perf-baseline.md
Normal file
175
plans/perf-baseline.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Performance Baseline Report (Phase 0 / UNCHANGED code)
|
||||
|
||||
**Date**: 2026-10-07
|
||||
**Commit**: `b77938a`
|
||||
**Dataset**: `perf/results/baseline-2026-10-07.json`
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary & Headline Baseline Numbers
|
||||
|
||||
This baseline captures the performance metrics of the unoptimized ytplayer codebase (commit `b77938a`, matching production tag `1175f1a1d2c1`) across 5 scenarios on both Chromium and WebKit under simulated LTE and Lossy network profiles (median of 5 runs).
|
||||
|
||||
### Headline Baseline Numbers
|
||||
|
||||
- **Cold First-Load Wire Transfer**:
|
||||
- **Chromium**: 164 requests, **1,015.0 KB** compressed wire bytes (boot-done median: **10,163 ms** under 4× CPU throttle + LTE).
|
||||
- **WebKit**: 162 requests, **1,152.0 KB (1.12 MB)** wire bytes (boot-done median: **3,201 ms** under proxy LTE).
|
||||
- Breakdown: ~477 KB JavaScript, ~397 KB Web Fonts, ~131 KB CSS, ~27 KB HTML.
|
||||
|
||||
- **Warm Reload (under Service Worker)**:
|
||||
- **Chromium**: 4 requests, **5.2 KB** wire bytes (periodic API version poll), boot-done median: **675 ms** (LTE) / **651 ms** (Lossy).
|
||||
- **WebKit**: 0 requests, **0 KB** wire bytes, boot-done median: **131 ms** (LTE) / **131 ms** (Lossy).
|
||||
|
||||
- **Offline Reload (under Service Worker)**:
|
||||
- **Chromium**: 0 requests, **0 B** wire bytes, reload median: **240 ms**, boot-done median: **219 ms**.
|
||||
- **WebKit**: 0 requests, **0 B** wire bytes, reload median: **135 ms**, boot-done median: **128 ms**.
|
||||
|
||||
- **Update N -> N+1 Wire Cost (The Invalidation Penalty)**:
|
||||
- **1-line `app.js` change**:
|
||||
- **Chromium**: 88 requests, **596.6 KB** wire bytes (~611 KB uncompressed/brotli), duration: **2,013 ms** (LTE) / **2,031 ms** (Lossy).
|
||||
- **WebKit**: 86 requests, **664.3 KB** wire bytes (~680 KB wire), duration: **4,457 ms** (LTE) / **8,135 ms** (Lossy).
|
||||
- **1-line `theme-glass.css` change**:
|
||||
- **Chromium**: 88 requests, **596.7 KB** wire bytes, duration: **1,970 ms** (LTE) / **2,004 ms** (Lossy).
|
||||
- **WebKit**: 86-87 requests, **664.3 KB** wire bytes, duration: **4,510 ms** (LTE) / **8,269 ms** (Lossy).
|
||||
|
||||
> **Core Invalidation Finding**: A single 1-line change to *any* static file (`app.js` or `theme-glass.css`) mutates `BUILD_TAG` calculated by `computeBuildTag()`. This causes `sw.js` to re-fetch and re-cache all 86-88 shell assets (~600-664 KB wire payload) rather than just the mutated resource.
|
||||
|
||||
---
|
||||
|
||||
## 2. Benchmark Harness Execution
|
||||
|
||||
The benchmark harness is located in `perf/baseline.mjs` and driven by Playwright with an isolated ephemeral Bun server and a custom pacing proxy (`perf/proxy.mjs`).
|
||||
|
||||
### Quick Start / Exact Commands
|
||||
|
||||
```bash
|
||||
# Ensure node_modules is symlinked to root
|
||||
ln -s ~/development/personal/ytplayer/node_modules node_modules
|
||||
|
||||
# Run full baseline suite (all scenarios, both browsers, both profiles, 5 runs)
|
||||
node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --out perf/results/baseline-2026-10-07.json
|
||||
|
||||
# Run individual browser or scenario
|
||||
node perf/baseline.mjs --runs 5 --browser chromium --profile lte --scenario cold
|
||||
node perf/baseline.mjs --runs 5 --browser webkit --profile lte --scenario update-js
|
||||
|
||||
# Run comparison against baseline after optimizations
|
||||
node perf/baseline.mjs --runs 5 --browser all --profile all --scenario all --compare perf/results/baseline-2026-10-07.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Results by Browser and Profile (Median of 5 Runs)
|
||||
|
||||
### Chromium — LTE (1.6 Mbps down, 750 kbps up, 150 ms RTT, 4× CPU throttle)
|
||||
|
||||
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Cold First Load** | 164 | 1,015.0 KB | 1,540 | 1,896 | 10,163 | 38,446 | 397 |
|
||||
| **Warm Reload (SW)** | 4 | 5.2 KB | 180 | 684 | 675 | - | 156 |
|
||||
| **Offline Reload** | 0 | 0.0 KB | 72 | 108 | 219 | 240 | - |
|
||||
| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,013 | - |
|
||||
| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 1,970 | - |
|
||||
|
||||
- **Storage Estimate**: Quota = 4,298,447,325 B (~4.0 GB), Usage = 3,480,029 B (~3.31 MB).
|
||||
- **Cold Wire Bytes by Type**: JS 477.2 KB, Fonts 397.1 KB, CSS 130.7 KB, HTML 27.1 KB, Images 6.3 KB, API/Other 1.0 KB.
|
||||
|
||||
---
|
||||
|
||||
### Chromium — Lossy (1.0 Mbps down, 500 kbps up, 250 ms RTT, 4× CPU throttle)
|
||||
|
||||
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Cold First Load** | 164 | 1,015.0 KB | 2,344 | 2,912 | 16,015 | 42,518 | 432 |
|
||||
| **Warm Reload (SW)** | 4 | 5.2 KB | 184 | 644 | 651 | - | 155 |
|
||||
| **Offline Reload** | 0 | 0.0 KB | 80 | 92 | 237 | 253 | - |
|
||||
| **Update N->N+1 (JS)** | 88 | 596.6 KB | - | - | - | 2,031 | - |
|
||||
| **Update N->N+1 (CSS)** | 88 | 596.7 KB | - | - | - | 2,004 | - |
|
||||
|
||||
---
|
||||
|
||||
### WebKit — LTE (1.6 Mbps down, 150 ms RTT via proxy)
|
||||
|
||||
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Cold First Load** | 162 | 1.12 MB | 1,088 | 1,287 | 3,201 | 30,508 | 0* |
|
||||
| **Warm Reload (SW)** | 0 | 0.0 KB | 110 | 110 | 131 | - | 0* |
|
||||
| **Offline Reload** | 0 | 0.0 KB | 107 | 107 | 128 | 135 | - |
|
||||
| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 4,457 | - |
|
||||
| **Update N->N+1 (CSS)** | 86 | 664.3 KB | - | - | - | 4,510 | - |
|
||||
|
||||
---
|
||||
|
||||
### WebKit — Lossy (1.0 Mbps down, 250 ms RTT + 1,500 ms stalls via proxy)
|
||||
|
||||
| Scenario | Reqs (med) | Wire Bytes (med) | FCP (ms) | LCP (ms) | Boot Done (ms) | SW Ready / Reload (ms) | Long Task Total (ms) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Cold First Load** | 162 | 1.12 MB | 3,557 | 3,735 | 7,303 | 35,449 | 0* |
|
||||
| **Warm Reload (SW)** | 0 | 0.0 KB | 111 | 112 | 131 | - | 0* |
|
||||
| **Offline Reload** | 0 | 0.0 KB | 109 | 119 | 129 | 141 | - |
|
||||
| **Update N->N+1 (JS)** | 86 | 664.3 KB | - | - | - | 8,135 | - |
|
||||
| **Update N->N+1 (CSS)** | 87 | 664.4 KB | - | - | - | 8,269 | - |
|
||||
|
||||
*\*Note: WebKit does not expose `PerformanceLongTaskTiming` (`longtask` entries), so long task total evaluates to 0 ms.*
|
||||
|
||||
---
|
||||
|
||||
## 4. Variance & Noise Observed
|
||||
|
||||
Over the 5 runs per configuration, measurements demonstrated high stability:
|
||||
- **Wire Bytes & Request Counts**: Deterministic (0% variance across identical runs).
|
||||
- **Cold Boot-Done**:
|
||||
- Chromium LTE: Min 10,129 ms, Max 10,798 ms (<6% variance).
|
||||
- Chromium Lossy: Min 15,871 ms, Max 16,031 ms (<1% variance).
|
||||
- WebKit LTE: Min 3,138 ms, Max 3,245 ms (<3% variance).
|
||||
- WebKit Lossy: Min 6,963 ms, Max 8,710 ms (~20% variance due to proxy random stalls).
|
||||
- **Warm & Offline Reload**:
|
||||
- Chromium Offline Reload: Min 238 ms, Max 257 ms (<8% variance).
|
||||
- WebKit Offline Reload: Min 131 ms, Max 141 ms (<7% variance).
|
||||
- **Update Duration**:
|
||||
- Chromium: Runs cluster around ~2,000 ms (2,013 ms median). Occasional outlier runs (~45–200 ms) occur if the browser resolves `reg.update()` immediately before network completion reporting.
|
||||
- WebKit: Steady scaling between LTE (~4.5 s) and Lossy (~8.2 s) pacing.
|
||||
|
||||
---
|
||||
|
||||
## 5. Limitations & Caveats
|
||||
|
||||
1. **WebKit Network Throttling**:
|
||||
WebKit lacks Chrome DevTools Protocol (CDP) support for hardware network emulation and CPU throttling. We paced throughput (1.6 Mbps) and injected latency/stalls using a custom local reverse proxy (`perf/proxy.mjs`). This throttles HTTP response chunks truthfully but cannot simulate OS-level socket buffer delays or 4× CPU throttling.
|
||||
2. **Autoplay Policy in WebKit**:
|
||||
Playwright WebKit on Linux rejects `--autoplay-policy=no-user-gesture-required` and enforces Safari-style gesture requirements on unmuted audio elements. The media playback benchmark detects this and correctly reports `autoplayBlocked: true` without failing the test run.
|
||||
3. **WebKit SetOffline Incompatibility**:
|
||||
Calling Playwright's `context.setOffline(true)` on Linux WebKit causes an unrecoverable engine crash (`WebKit encountered an internal error`). Offline mode for WebKit is simulated by terminating the proxy upstream connection (`proxy.setOffline(true)`).
|
||||
|
||||
---
|
||||
|
||||
## 6. Manual iPhone Testing Protocol (Phase 0 / iOS Safari PWA)
|
||||
|
||||
Because automated headless WebKit cannot fully replicate iOS Safari's mobile background constraints and battery telemetry, the following manual protocol is required:
|
||||
|
||||
### Protocol Steps
|
||||
|
||||
1. **Install PWA**:
|
||||
- Open Safari on iPhone and navigate to `https://worship.hesed.sbs`.
|
||||
- Tap the Share button -> **"Add to Home Screen"**.
|
||||
- Launch ytplayer directly from the Home Screen icon.
|
||||
|
||||
2. **Verify Version & Cache**:
|
||||
- Verify the footer / version string matches `1175f1a1d2c1` (or current production build tag).
|
||||
- Turn on Airplane Mode and force-reload to confirm offline readiness. Turn Airplane Mode back off.
|
||||
|
||||
3. **30-Minute Screen-Off Battery Test**:
|
||||
- Note baseline battery percentage (e.g., 85%).
|
||||
- Select a pre-cached playlist or track and tap **Play**. Ensure continuous audio is heard.
|
||||
- Lock the screen (display off) and let playback run uninterrupted for **30 minutes**.
|
||||
- Note battery percentage at 30 minutes (calculate drain %/hr).
|
||||
- Listen for any audio dropouts, stuttering, or premature process suspension.
|
||||
|
||||
4. **Interruption & Resumption Test**:
|
||||
- Trigger an interruption while locked (e.g., initiate an incoming phone call or start Siri).
|
||||
- End the interruption and observe if playback automatically resumes via `MediaSession` handlers.
|
||||
- Test Lock Screen Controls (Play/Pause, Next Track, scrubber).
|
||||
|
||||
5. **Log Findings**:
|
||||
- Record iPhone Model, iOS Version, Battery Drain (% per 30m), Dropouts Count (0 = pass), and Resumption Status (Pass/Fail).
|
||||
Reference in New Issue
Block a user