diff --git a/docs/improvement-plan.md b/docs/improvement-plan.md index 129a0a9..3cf15d5 100644 --- a/docs/improvement-plan.md +++ b/docs/improvement-plan.md @@ -1,7 +1,25 @@ # ytplayer — Speed & Features Plan (2026-09-29) Scope: **make the app faster** (startup, search, pressing play, moving around) -and **add new features**. Hardening/refactor work is intentionally out of scope. +and **add new features**, led by **peer-to-peer video sharing**. Hardening/refactor +work is intentionally out of scope. + +## Integrated roadmap (what is queued and executable today) + +Everything below is broken into step-by-step plans a smaller model can execute +without exploring the code — see `plans/INDEX.md`. All 19 were dry-run end to end. + +| Order | Plans | What it delivers | Source | +|-------|-------|------------------|--------| +| 1 | `001` | Timing marks (`ytp:boot`, `ytp:search`, `ytp:tap-to-play`) + yt-dlp duration logs | A "Measuring" | +| 2 | `002`–`003` | Compressed + ETagged shell (app.js 426 → 94 KB), self-hosted fonts | A1.1–A1.3 | +| 3 | `004`–`005` | One yt-dlp resolve per video at a time; resolve likely next plays before the tap | A3.1–A3.2 | +| 4 | `006`–`007` | Search via InnerTube (~0.8 s, yt-dlp fallback); long-lived yt-dlp workers | A2.1, A3.3 | +| 5 | `008`–`019` | **Peer-to-peer sharing** — design and rules in `docs/p2p-architecture.md` | Part B0 | + +Not queued yet (write plans for them with `/plan-queue` when reached): A1.4 minify, +A1.5 deferred boot work, A1.6 lazy modules, A2.2 suggestions, A2.3/A2.4, A3.4/A3.5, +A4 (VPS media cache), A5 polish, and the Part B feature list. ## Where the time goes today (measured against prod) @@ -99,6 +117,21 @@ before the user taps.** ## Part B — New features +### B0. Peer-to-peer video sharing (queued: plans 008–019) + +The server keeps the user list, the video list and metadata; every validated copy +gets a content id = SHA-256 of its bytes; devices that save a video become persistent +**holders**; when YouTube and the server copy are gone, devices serve each other over +WebRTC and can restore the server's copy. Rules the owner set (2026-09-29): +**P2P is on by default**, the **server malware scan is off by default** (hashing + +media validation always run), and holder records are **persistent** — the UI shows +when each was last verified and marks old ones **stale** instead of dropping them. +Full design, data model, flows and security rules: `docs/p2p-architecture.md`. +The earlier phase 02–06 draft is superseded; its differences are listed at the end of +that document. + +### Other features (not queued yet) + Ranked by fit with how the app is actually used (worship sets, sing-alongs, offline playback). Each is sized; most reuse machinery that already exists. @@ -144,16 +177,15 @@ offline playback). Each is sized; most reuse machinery that already exists. ## Suggested order -1. **Week 1 — quick speed wins:** add the timing marks (Measuring), A1.1 - compression, A1.2 ETags, A1.3 fonts, A3.1 coalescing, A3.2 warm-on-intent, - A3.4 instant UI. These are all small, and together they fix most of what - feels slow today. -2. **Week 2:** A2.1 InnerTube search + A2.2 suggestions, A1.4 defer/minify. -3. **Week 3:** A3.3 persistent yt-dlp worker, A1.5 deferred boot work, then - measure the WireGuard link (A4) and decide on the VPS media cache. -4. **Then features**, starting with the small high-fit ones: section loops, +1. **Run the queue** (`/run-queue`): plans `001`–`007` (speed), then `008`–`019` + (peer-to-peer). Deploy after `007` and measure with the `ytp:*` marks and the + `[ytdlp]` log lines before starting P2P. +2. **Next speed plans to write:** A3.4 instant UI on tap, A2.2 suggestions, + A1.5 deferred boot work, then measure the WireGuard link (A4) and decide on the + VPS media cache. +3. **Then features**, starting with the small high-fit ones: section loops, confidence monitor, count-in, lyrics search, stats wrap-up — then service plans and chord charts. -Each row is sized to be one commit (CLAUDE.md commit rules) and can be queued -with the `plan-queue` skill. +Each item is sized to be one commit (CLAUDE.md commit rules); queue new ones with +the `plan-queue` skill. diff --git a/docs/p2p-architecture.md b/docs/p2p-architecture.md new file mode 100644 index 0000000..627f80c --- /dev/null +++ b/docs/p2p-architecture.md @@ -0,0 +1,155 @@ +# Peer-to-peer video sharing — architecture (2026-09-29) + +Source of truth for plans `008`–`019` in `plans/queue/`. Every P2P plan links here +instead of repeating the rules. It replaces the earlier "phase 02–06" draft; the +differences are listed at the end. + +## What the owner asked for + +> Peer-to-peer saving of videos. The server stores the user list, the video list and +> metadata. While the original source is online, a video is available for streaming +> and download. When it is downloaded to a device, the server records that device in +> the list of holders, so the video stays reachable from devices after the source is +> gone. Top or recent files stay on the server under a total space limit; files that +> don't meet the criteria (e.g. number of views, also stored on the server) are +> deleted first. The video id is the file hash of the highest-quality copy. The +> database grows over time. Each device has its own database that can be synced or +> added to the server's, with verification that the file exists. A file must first be +> downloaded by the server and checked before its hash is added to the server DB. + +Corrections to the earlier draft (owner, 2026-09-29): +1. **P2P is ON by default** (server and every client). +2. **The server's malware scan is OFF by default** (admin opt-in). Hashing and the + media validation gate are ALWAYS on and cannot be turned off. +3. **Holder records are persistent, not short-lived leases.** A device stays listed + as a holder until it says the file is gone, fails a check, or its reported list no + longer contains it. The UI shows when each holder was last verified and marks it + **stale** when that is older than `P2P_STALE_DAYS` (default 7). Online-right-now is + a separate, live signal. + +## Vocabulary + +| Term | Meaning | +|------|---------| +| **source** | Where bytes originally come from: YouTube (via yt-dlp) or a server upload (`upl_…`). | +| **video id** | Existing ids (`dQw4w9WgXcQ`, `upl_…`). Still used everywhere in the app and API. | +| **content id (`cid`)** | Lowercase hex SHA-256 of the exact file bytes. The P2P identity of a file. One video id can have several cids over time (a better master, the HEVC copy). | +| **master** | The best copy the server keeps for a video: today the validated ≤720p H.264+AAC faststart MP4 in `MEDIA_DIR` (`..mp4`). The compression lane's HEVC copy is a second, separately hashed file. Raising the master quality later just creates new cids linked by `video_id`. | +| **verified content** | A `p2p_content` row. Exists only after the server itself held the complete bytes, computed the SHA-256 itself and `validateMedia()` passed (plus the malware scan if enabled). | +| **holder** | A device that reported holding a cid. Row in `p2p_holders`, never deleted by time. | +| **online** | The device has an open `/ws/p2p` socket right now (in memory only). | +| **stale** | `now - last_verified_at > P2P_STALE_DAYS`. Shown in the UI, still listed. | + +## Data model (server, libsql — grows forever) + +Added by plan 008 in `server/p2p-db.js` (`initP2pSchema()` runs after `initDb()`): + +```sql +p2p_content (cid PK, video_id, size, height, vcodec, acodec, duration, meta JSON, + origin 'server'|'intake', status 'verified'|'revoked', + scan 'skipped'|'clean', created_at ms, verified_at ms) +p2p_devices (device_id PK 'dev_<16hex>', secret_hash, fingerprint, profile, + share 0|1, created_at ms, last_seen_at ms) +p2p_holders (cid, device_id, status 'active'|'removed', trust 'reported'|'challenged', + first_reported_at ms, last_verified_at ms, removed_at ms NULL, + PRIMARY KEY (cid, device_id)) +video_views (video_id, day 'YYYY-MM-DD', n, PRIMARY KEY (video_id, day)) +media_cache.sha256 -- new column: cid of the current ..mp4 +``` + +"User list" = the existing `users` (fingerprints) and `profiles` tables plus +`p2p_devices`. Nothing is ever deleted from `p2p_content`; a bad file is `revoked`. + +## Configuration (`server/p2p-config.js`, plan 008) + +| Env | Default | Meaning | +|-----|---------|---------| +| `P2P_ENABLED` | `1` (on) | `0` turns off every P2P route, the hub and client features. | +| `P2P_MALWARE_SCAN` | `0` (off) | `1` runs `P2P_SCAN_CMD ` before admission; exit 0 = clean, 1 = infected (rejected), other = error (not admitted, retried later). | +| `P2P_SCAN_CMD` | `clamscan --no-summary --infected` | Needs an image built with `--build-arg INSTALL_CLAMAV=1`. | +| `P2P_STALE_DAYS` | `7` | Holder older than this is shown as stale. | +| `P2P_KEEP_MIN_VIEWS` | `3` | Retention: views in the last `P2P_KEEP_DAYS` that make a server copy "top". | +| `P2P_KEEP_DAYS` | `30` | Window for counting views. | +| `P2P_KEEP_RECENT_DAYS` | `14` | Retention: played this recently = "recent". | +| `P2P_INTAKE_DIR` | `/p2p-intake` | Quarantine for device uploads. Never served. | +| `P2P_INTAKE_MAX_BYTES` | `3 GiB` | Largest accepted intake upload. | + +Client settings (`data.settings`, per profile): `p2pShare: true` (let other devices +download my saved videos, and report holdings), `p2pReceive: true` (fetch from other +devices when YouTube and the server can't serve). + +## Flows + +1. **Server fetch (existing media cache) → verified content** (plan 009). + `runFetch` / `runOptimize` hash the promoted file, store `media_cache.sha256`, run the + scan if enabled, then upsert `p2p_content` (`origin 'server'`). `/api/download` sends + `X-Content-SHA256`. A backfill hashes already-cached files at boot, one at a time. +2. **Device save** (plan 012). The OPFS worker hashes while it writes. If the server + sent `X-Content-SHA256` and the hash differs, the save fails (bonus integrity check). + The device DB (`IndexedDB ytp-device`, store `files`) records `{videoId, cid, size, + savedAt, lastCheckedAt, state}`; `state` is `verified` when the hashes matched, + `unverified` when the server sent no hash, `unhashed` for old saves and the + main-thread fallback path. +3. **Holdings sync** (plan 013). Device registers once (`POST /api/p2p/device` → + `deviceId` + `secret`, kept in `localStorage.ytpDevice`). It reports its holdings + (`POST /api/p2p/holdings`, full list at launch, deltas after save/delete). The server + accepts only cids in `p2p_content` with `status='verified'`; unknown cids come back + in `unknown` (candidates for intake). When the server still has the file it returns + up to 5 **range challenges**; a correct answer sets `trust='challenged'`, a wrong one + removes the holder. `last_verified_at` = time of the last report where the device + re-checked the file (exists, same size; full re-hash every 30 days). A full report + marks every active holder row of that device that is missing from the list as + `removed`. There is **no TTL**. +4. **Presence** (plan 014). `/ws/p2p` socket per device, authenticated with the device + secret. Online status lives only in memory. The hub also relays WebRTC signalling + between two online devices and carries server → device requests (plan 018). +5. **Availability** (plans 014/015). `GET /api/p2p/holders?v=` lists cids and + their holders: opaque peer id (never the fingerprint/profile), `online`, + `lastVerifiedAt`, `stale`, `trust`, plus `serverHas`. The UI shows e.g. + "📡 On 3 devices · 1 online now · last checked 2 d ago", with stale holders greyed. +6. **Peer download** (plan 017). WebRTC data channel (STUN only, same ICE list as watch + party), 64 KiB frames with `bufferedAmount` back-pressure, receiver writes through a + worker into OPFS while hashing; only a matching SHA-256 is committed. The new copy + is a holder at the next report. Download-then-play; no progressive peer streaming. + Used when `/api/streams` fails and the device has no copy, and from a + "Get from a device" button. +7. **Intake** (plan 016). A device can hand a file to the server + (`POST /api/p2p/intake` → ticket, `PUT` the bytes). The server writes it to the + quarantine dir, hashes it, runs `validateMedia()`, runs the scan if enabled, and only + then inserts `p2p_content` (`origin 'intake'`). If the server has no copy of that + video it adopts the file into the media cache (budget permitting). +8. **Rehydrate** (plan 018). When a video is requested, its source fails, the server + evicted its copy, and a verified holder is online with `p2pShare` on, the hub asks + that device to upload it through intake (known cid → quick accept). +9. **Retention** (plan 010). Views are counted per video per day. When the media cache + needs room it evicts in this order: copies that are neither "top" + (`views in P2P_KEEP_DAYS ≥ P2P_KEEP_MIN_VIEWS`) nor "recent" (played within + `P2P_KEEP_RECENT_DAYS`), fewest views first, then oldest; only then the qualifying + ones by LRU. The 10-minute play protection and `MEDIA_CACHE_MAX_BYTES` stay. + Evicting a server copy never deletes `p2p_content` or holder rows. + +## Security rules every plan must keep + +- No cid enters `p2p_content` unless the SERVER computed it over bytes it holds and + `validateMedia` passed. Clients can never insert or edit content rows, views or trust. +- Intake files live in `P2P_INTAKE_DIR`, never under `./public` or `MEDIA_DIR`, and + are deleted on failure. +- Device secrets: 32 random bytes, only `sha256(secret)` stored, compared with + `timingSafeEqual`. +- Holder lists never expose fingerprints, profile names or IPs; a peer id is + `sha256('peer:' + device_id).slice(0, 12)`. +- The hub relays signalling only between two authenticated, online devices, with a + per-socket message budget. +- `P2P_ENABLED=0` must leave the rest of the app working exactly as before. +- Jobs stay server-owned; never pass a request `AbortSignal` into them (CLAUDE.md). + +## Where this differs from the earlier phase 02–06 draft + +| Earlier draft | Now | +|---------------|-----| +| "Default-off" P2P subsystem; "no inventory/upload from default settings" | P2P on by default; devices report holdings and seed by default (can be turned off). | +| Mandatory scanner, "scan skip is failure" | Scanner off by default (`P2P_MALWARE_SCAN=0`); hash + `validateMedia` mandatory. | +| Short-lived online leases; "cache availability only as an expiring hint" | Persistent holder rows with `last_verified_at` and a stale marker; online status is separate. | +| Migrate all localStorage (`_ytpdata`) to IndexedDB first | Not now: the device DB holds files + cids only. `_ytpdata` stays in localStorage (lower risk). | +| Collections, invitations, scoped principals, signed manifests, TURN, renditions lineage | Deferred. Scope is one shared catalog + device secrets; add later if needed. | +| Separate `server/p2p/*` directory with migrations ledger | Flat files `server/p2p-*.js` matching the repo's style (`party.js`, `remote.js`). | diff --git a/plans/INDEX.md b/plans/INDEX.md new file mode 100644 index 0000000..90a87a3 --- /dev/null +++ b/plans/INDEX.md @@ -0,0 +1,29 @@ +# Plan Queue Index + +Speed first (001–007, independent of P2P), then peer-to-peer sharing (008–019, +see `docs/p2p-architecture.md`). Run in order; each plan's `depends_on` lists what +must be in `done/` first. Every plan was dry-run end to end on 2026-09-29 (all 19 +applied in order to a clean checkout: 13 server test files and 61 frontend tests +green, app boots with no JS errors, P2P on by default, offline boot works). + +| Seq | ID | Title | Status | Commit | Notes | +|-----|----|-------|--------|--------|-------| +| 001 | 001-perf-timing-marks-105acc | Add startup, search and play timing marks plus yt-dlp duration logs | queued | | baseline numbers for the rest | +| 002 | 002-compress-and-etag-shell-bd459c | Serve the app shell gzip/brotli-compressed with ETags | queued | | app.js 426 KB → 94 KB (br) | +| 003 | 003-self-host-fonts-89466b | Self-host the three web fonts and drop the render-blocking Google Fonts CSS | queued | | needs network once; CSP font-src fix | +| 004 | 004-coalesce-stream-resolves-a92d40 | Coalesce concurrent resolveStreams calls for the same video | queued | | | +| 005 | 005-warm-streams-on-intent-47b3d3 | Warm the stream cache for likely next plays | queued | | cold play 7.5 s → cached 1.2 s | +| 006 | 006-innertube-search-48066b | Answer searches from YouTube InnerTube directly with yt-dlp fallback | queued | | search 4–5 s → ~0.8 s | +| 007 | 007-ytdlp-worker-045800 | Keep one long-lived yt-dlp worker process instead of spawning per call | queued | | ~1 s per yt-dlp call | +| 008 | 008-p2p-schema-and-config-127966 | Add P2P tables, config flags and db helpers | queued | | P2P ON, malware scan OFF by default | +| 009 | 009-server-content-hash-186e7f | Hash every validated server copy and register it as verified content | queued | | uses plans/patches/009-* | +| 010 | 010-views-and-retention-d0c6ca | Count views and evict server copies by retention criteria before LRU | queued | | | +| 011 | 011-browser-sha256-e1793d | Add an incremental SHA-256 library for the browser and node tests | queued | | | +| 012 | 012-device-file-registry-288d55 | Add the on-device IndexedDB file registry and hash saves while downloading | queued | | browser harness | +| 013 | 013-device-identity-and-holdings-3ba493 | Register devices and report verified holdings to the server | queued | | persistent holders, no TTL | +| 014 | 014-p2p-presence-hub-ceced8 | Add the /ws/p2p presence and signalling hub and the holders endpoint | queued | | stale flag, never hidden | +| 015 | 015-availability-ui-and-settings-3b9397 | Show peer availability with stale markers and add Sharing settings | queued | | | +| 016 | 016-intake-and-server-verification-cfe031 | Let a device hand a file to the server for hashing and validation | queued | | needs ffmpeg for tests | +| 017 | 017-peer-transfer-1faaa7 | Download a verified file from another device over WebRTC | queued | | STUN only | +| 018 | 018-server-rehydrate-from-peer-4fb8bd | Restore an evicted server copy from an online holder | queued | | | +| 019 | 019-admin-p2p-panel-4dc623 | Add a P2P panel to the admin page | queued | | | diff --git a/plans/README.md b/plans/README.md new file mode 100644 index 0000000..8387d92 --- /dev/null +++ b/plans/README.md @@ -0,0 +1,22 @@ +# plans/ + +Executable plan queue (skills `plan-queue` / `run-queue`). Each file in `queue/` is +self-contained: exact files, anchors, code and copy-paste verification. + +- `INDEX.md` — the order and status of every plan. +- `queue/ active/ done/ failed/` — plan lifecycle. +- `patches/` — unified diffs referenced by plans (`git apply plans/patches/.diff`). + They were produced from code that passed its tests, and are meant to be applied in plan + order (e.g. `016-media-cache-adopt.diff` expects `009` and `010` applied). If a patch + does not apply, stop and report — never hand-edit around it. +- `harness/` — small throwaway servers + Playwright checks that exercise the browser + side of the P2P plans for real (OPFS, IndexedDB, WebSocket, WebRTC). Run the server + from `server/` (`bun ../plans/harness/-server.js`), then `node -check.mjs` from + `plans/harness/` (needs `npm i --no-save playwright` and a Chromium; the cloud image has + one at `/opt/pw-browsers/chromium-1194/chrome-linux/chrome`, override with `CHROMIUM=`). + +Environment notes: server tests need `cd server && bun install`; media / intake tests need +`ffmpeg` + `ffprobe`; plan 003 needs network access to Google Fonts once; plan 007's +pool test needs `bin/yt-dlp` (`npm run setup`) or skips. +Stop a background server by PID (`… & SRV=$!` … `kill $SRV`) — `pkill -f ` also +matches the shell running the command and kills it. diff --git a/plans/active/.gitkeep b/plans/active/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/plans/done/.gitkeep b/plans/done/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/plans/failed/.gitkeep b/plans/failed/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/plans/harness/device-db-check.mjs b/plans/harness/device-db-check.mjs new file mode 100644 index 0000000..8e64a59 --- /dev/null +++ b/plans/harness/device-db-check.mjs @@ -0,0 +1,22 @@ +// Plan 012 harness check. Needs playwright (npm i --no-save playwright) and a +// Chromium: CHROMIUM=/path/to/chrome (default: the cloud image's /opt/pw-browsers one). +import { chromium } from 'playwright'; +const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' }); +const p = await b.newPage(); +p.on('console', (m) => console.log('console:', m.text())); +await p.goto('http://localhost:8766/'); +const out = await p.evaluate(async () => { + const sha = await (await fetch('/sha')).text(); + const good = await OPFS.downloadVideo('good', '/api/download/good'); + const bad = await OPFS.downloadVideo('bad', '/api/download/bad'); + const nohash = await OPFS.downloadVideo('nohash', '/api/download/nohash'); + const list = (await OPFS.listVideos()).map((x) => x.id).sort(); + await DeviceDB.putFile({ videoId: 'good', cid: good.sha256, size: good.size, state: 'verified' }); + const rec = await DeviceDB.getFile('good'); + const byCid = await DeviceDB.getByCid(good.sha256); + await DeviceDB.deleteFile('good'); + const after = await DeviceDB.listFiles(); + return { shaOk: good.sha256 === sha, goodExpected: good.expectedSha === sha, bad, nohashSha: nohash.sha256 === sha, nohashExpected: nohash.expectedSha, list, rec: rec && rec.state, byCid: byCid && byCid.videoId, after: after.length }; +}); +console.log(JSON.stringify(out)); +await b.close(); diff --git a/plans/harness/device-db-server.js b/plans/harness/device-db-server.js new file mode 100644 index 0000000..85e4d07 --- /dev/null +++ b/plans/harness/device-db-server.js @@ -0,0 +1,17 @@ +// Plan 012 harness: serves ../../frontend plus fake /api/download/ responses +// (good hash / wrong hash / no hash). Run: bun device-db-server.js +import { createHash } from 'node:crypto'; +const FE = new URL('../../frontend/', import.meta.url).pathname; +const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256); +const sha = createHash('sha256').update(body).digest('hex'); +Bun.serve({ port: 8766, fetch(req) { + const u = new URL(req.url); + if (u.pathname === '/api/download/good') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } }); + if (u.pathname === '/api/download/bad') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': 'f'.repeat(64) } }); + if (u.pathname === '/api/download/nohash') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length) } }); + if (u.pathname === '/sha') return new Response(sha); + if (u.pathname === '/') return new Response('', { headers: { 'Content-Type': 'text/html' } }); + const f = Bun.file(FE + u.pathname.slice(1)); + return new Response(f, { headers: { 'Content-Type': 'text/javascript' } }); +} }); +console.log('ready'); diff --git a/plans/harness/intake-check.mjs b/plans/harness/intake-check.mjs new file mode 100644 index 0000000..25312a6 --- /dev/null +++ b/plans/harness/intake-check.mjs @@ -0,0 +1,19 @@ +// Plan 016 harness check: an unknown saved file → contribute → accepted. +import { chromium } from 'playwright'; +const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' }); +const p = await b.newPage(); +await p.goto('http://localhost:8769/'); +const out = await p.evaluate(async () => { + const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer()); + const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true }); + const w = await (await dir.getFileHandle('upAAAAAAAA9.mp4', { create: true })).createWritable(); + await w.write(bytes); await w.close(); + P2PClient.start({ getSettings: () => ({}) }); + const first = await P2PClient.sync(); + const unknown = await P2PClient.unknownVideos(); + const c = await P2PClient.contribute(unknown[0]); + const second = await P2PClient.sync(); + return { firstUnknown: first.unknown.length, unknown, contribute: { ok: c.ok, adopted: c.adopted, cidOk: /^[0-9a-f]{64}$/.test(c.cid) }, secondAccepted: second.accepted.length, secondUnknown: second.unknown.length }; +}); +console.log(JSON.stringify(out)); +await b.close(); diff --git a/plans/harness/intake-server.js b/plans/harness/intake-server.js new file mode 100644 index 0000000..1b69615 --- /dev/null +++ b/plans/harness/intake-server.js @@ -0,0 +1,29 @@ +// Plan 016 harness: routes + intake with real validateMedia (needs ffmpeg). +// Run from server/: bun ../plans/harness/intake-server.js +const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js'); +import { mkdtempSync, readFileSync } from 'node:fs'; +import { spawnSync } from 'node:child_process'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +const root = mkdtempSync(join(tmpdir(), 'ytp-h16-')); +process.env.DB_PATH = join(root, 'h.db'); +const dbmod = await import(process.cwd() + '/db.js'); +const p2pDb = await import(process.cwd() + '/p2p-db.js'); +const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js'); +const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js'); +const { admitFile } = await import(process.cwd() + '/p2p-admit.js'); +const { validateMedia } = await import(process.cwd() + '/media-cache.js'); +const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname; +await dbmod.initDb(); await p2pDb.initP2pSchema(); +const fx = join(root, 'fx.mp4'); +spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4', + '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]); +const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 }; +const app = new Hono(); +const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' }); +registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile }); +app.get('/fixture.mp4', () => new Response(readFileSync(fx))); +app.get('/', (c) => c.html('')); +app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } })); +Bun.serve({ port: 8769, fetch: app.fetch }); +console.log('ready'); diff --git a/plans/harness/p2p-client-check.mjs b/plans/harness/p2p-client-check.mjs new file mode 100644 index 0000000..c0aaa91 --- /dev/null +++ b/plans/harness/p2p-client-check.mjs @@ -0,0 +1,26 @@ +// Plan 013 harness check (see device-db-check.mjs for requirements). +import { chromium } from 'playwright'; +const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' }); +const p = await b.newPage(); +p.on('console', (m) => { if (m.type() === 'error') console.log('console:', m.text()); }); +await p.goto('http://localhost:8767/'); +const out = await p.evaluate(async () => { + const good = await OPFS.downloadVideo('goodAAAAAAA', '/api/download/goodAAAAAAA'); + // A legacy save: in OPFS but no DeviceDB record, unknown to the server. + const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true }); + const w = await (await dir.getFileHandle('legacyAAAAA.mp4', { create: true })).createWritable(); + await w.write(new Uint8Array(1000).fill(9)); await w.close(); + await DeviceDB.putFile({ videoId: 'goodAAAAAAA', cid: good.sha256, size: good.size, state: 'unverified' }); + await DeviceDB.putFile({ videoId: 'ghostAAAAAA', cid: 'e'.repeat(64), size: 5, state: 'verified' }); // no file + P2PClient.start({ getSettings: () => ({}) }); + const res = await P2PClient.sync(); + const recs = (await DeviceDB.listFiles()).map((r) => [r.videoId, r.state, !!r.cid]).sort(); + const holders = await (await fetch('/debug/holders')).json(); + // Turn sharing off → full withdrawal. + P2PClient.start({ getSettings: () => ({ p2pShare: false }) }); + await P2PClient.sync(); + const after = await (await fetch('/debug/holders')).json(); + return { accepted: res && res.accepted.length, unknown: res && res.unknown.length, challenges: res && res.challenges.length, recs, trust: holders.map((h) => h.trust), afterShareOff: after.length, device: !!P2PClient.device() }; +}); +console.log(JSON.stringify(out)); +await b.close(); diff --git a/plans/harness/p2p-client-server.js b/plans/harness/p2p-client-server.js new file mode 100644 index 0000000..9849b3b --- /dev/null +++ b/plans/harness/p2p-client-server.js @@ -0,0 +1,27 @@ +// Plan 013 harness: real p2p-routes + temp DB + the frontend files. +// Run from server/: bun ../plans/harness/p2p-client-server.js +const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js'); +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { createHash } from 'node:crypto'; +const root = mkdtempSync(join(tmpdir(), 'ytp-h13-')); +process.env.DB_PATH = join(root, 'h.db'); +const dbmod = await import(process.cwd() + '/db.js'); +const p2pDb = await import(process.cwd() + '/p2p-db.js'); +const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js'); +const { sha256Range } = await import(process.cwd() + '/hash.js'); +const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname; +await dbmod.initDb(); await p2pDb.initP2pSchema(); +const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256); +const sha = createHash('sha256').update(body).digest('hex'); +const file = join(root, 'good.mp4'); writeFileSync(file, body); +await p2pDb.upsertContent({ cid: sha, videoId: 'goodAAAAAAA', size: body.length, origin: 'server', now: Date.now() }); +const app = new Hono(); +registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async (c) => (c === sha ? { path: file, size: body.length } : null), sha256Range }); +app.get('/api/download/:id', () => new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } })); +app.get('/debug/holders', async (c) => c.json(await p2pDb.listHolders(sha))); +app.get('/', (c) => c.html('')); +app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } })); +Bun.serve({ port: 8767, fetch: app.fetch }); +console.log('ready'); diff --git a/plans/harness/presence-check.mjs b/plans/harness/presence-check.mjs new file mode 100644 index 0000000..e7cdc3c --- /dev/null +++ b/plans/harness/presence-check.mjs @@ -0,0 +1,21 @@ +// Plan 014 harness check: two browsers, one signal (see device-db-check.mjs for requirements). +import { chromium } from 'playwright'; +const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' }); +const pages = []; +for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); await p.goto('http://localhost:8768/'); pages.push(p); } +const peers = []; +for (const p of pages) { + peers.push(await p.evaluate(async () => { + P2PClient.start({ getSettings: () => ({}) }); + await P2PClient.sync(); + for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100)); + window.__got = []; + P2PClient.onMessage('signal', (m) => window.__got.push(m)); + return P2PClient.peer(); + })); +} +const sent = await pages[0].evaluate((to) => P2PClient.signal(to, { hi: 1 }), peers[1]); +await new Promise((r) => setTimeout(r, 500)); +const got = await pages[1].evaluate(() => window.__got); +console.log(JSON.stringify({ peers: peers.map((x) => typeof x === 'string' && x.length === 12), distinct: peers[0] !== peers[1], sent, got: got.map((m) => ({ from: m.from === peers[0], data: m.data })) })); +await b.close(); diff --git a/plans/harness/presence-server.js b/plans/harness/presence-server.js new file mode 100644 index 0000000..62574c5 --- /dev/null +++ b/plans/harness/presence-server.js @@ -0,0 +1,24 @@ +// Plan 014 harness: p2p-routes + p2p-hub on a real Bun server (websocket). +// Run from server/: bun ../plans/harness/presence-server.js +const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js'); +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +const root = mkdtempSync(join(tmpdir(), 'ytp-h14-')); +process.env.DB_PATH = join(root, 'h.db'); +const dbmod = await import(process.cwd() + '/db.js'); +const p2pDb = await import(process.cwd() + '/p2p-db.js'); +const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js'); +const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js'); +const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname; +await dbmod.initDb(); await p2pDb.initP2pSchema(); +const app = new Hono(); +registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' }); +const hub = createP2pHub({ getDevice: p2pDb.getDevice }); +app.get('/', (c) => c.html('')); +app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } })); +Bun.serve({ port: 8768, fetch(req, server) { + if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server); + return app.fetch(req, server); +}, websocket: hub.websocket }); +console.log('ready'); diff --git a/plans/harness/rehydrate-check.mjs b/plans/harness/rehydrate-check.mjs new file mode 100644 index 0000000..2d27a74 --- /dev/null +++ b/plans/harness/rehydrate-check.mjs @@ -0,0 +1,21 @@ +// Plan 018 harness check: a holder is asked to restore an evicted copy. +import { chromium } from 'playwright'; +const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' }); +const p = await b.newPage(); +await p.goto('http://localhost:8771/'); +const out = await p.evaluate(async () => { + const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer()); + const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true }); + const w = await (await dir.getFileHandle('goneAAAAAAA.mp4', { create: true })).createWritable(); + await w.write(bytes); await w.close(); + P2PClient.start({ getSettings: () => ({}) }); + const rep = await P2PClient.sync(); // hashes the file, reports it → accepted (cid is verified content) + for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100)); + const s1 = await (await fetch('/api/streams?v=goneAAAAAAA')).json(); + let dbg; + for (let i = 0; i < 50; i++) { dbg = await (await fetch('/debug')).json(); if (dbg.adopted.length) break; await new Promise((r) => setTimeout(r, 200)); } + const s2 = await (await fetch('/api/streams?v=goneAAAAAAA')).json(); + return { accepted: rep.accepted.length, first: s1, adopted: dbg.adopted.map((a) => [a.videoId, a.sha256 === dbg.cid]), second: s2 }; +}); +console.log(JSON.stringify(out)); +await b.close(); diff --git a/plans/harness/rehydrate-server.js b/plans/harness/rehydrate-server.js new file mode 100644 index 0000000..40050fd --- /dev/null +++ b/plans/harness/rehydrate-server.js @@ -0,0 +1,49 @@ +// Plan 018 harness: hub + intake + rehydrator + a stand-in /api/streams whose +// source always fails and whose server copy is gone. Port 8771. +// Run from server/: bun ../plans/harness/rehydrate-server.js (needs ffmpeg) +const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js'); +import { mkdtempSync, readFileSync, unlinkSync } from 'node:fs'; +import { spawnSync } from 'node:child_process'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { createHash } from 'node:crypto'; +const root = mkdtempSync(join(tmpdir(), 'ytp-h18-')); +process.env.DB_PATH = join(root, 'h.db'); +const dbmod = await import(process.cwd() + '/db.js'); +const p2pDb = await import(process.cwd() + '/p2p-db.js'); +const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js'); +const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js'); +const { createP2pHub, createRehydrator } = await import(process.cwd() + '/p2p-hub.js'); +const { admitFile } = await import(process.cwd() + '/p2p-admit.js'); +const { validateMedia } = await import(process.cwd() + '/media-cache.js'); +const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname; +await dbmod.initDb(); await p2pDb.initP2pSchema(); +const fx = join(root, 'fx.mp4'); +spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4', + '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]); +const bytes = readFileSync(fx); +const cid = createHash('sha256').update(bytes).digest('hex'); +await p2pDb.upsertContent({ cid, videoId: 'goneAAAAAAA', size: bytes.length, origin: 'server', now: Date.now() }); // verified earlier, file since evicted +const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 }; +const adopted = []; +const app = new Hono(); +const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' }); +const hub = createP2pHub({ getDevice: p2pDb.getDevice }); +registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, + serverHasCid: async () => adopted.length > 0, + adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256 }); unlinkSync(path); return { adopted: true }; } }); +const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => adopted.length > 0 }); +app.get('/api/streams', async (c) => { + const v = c.req.query('v'); + if (await rehydrate(v)) return c.json({ ok: false, restoring: true, error: 'restoring from a device' }, 503); + return c.json({ ok: false, error: 'source unavailable' }, 500); +}); +app.get('/debug', (c) => c.json({ adopted, cid })); +app.get('/fixture.mp4', () => new Response(bytes)); +app.get('/', (c) => c.html('')); +app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } })); +Bun.serve({ port: 8771, fetch(req, server) { + if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server); + return app.fetch(req, server); +}, websocket: hub.websocket }); +console.log('ready'); diff --git a/plans/harness/transfer-check.mjs b/plans/harness/transfer-check.mjs new file mode 100644 index 0000000..282c14e --- /dev/null +++ b/plans/harness/transfer-check.mjs @@ -0,0 +1,52 @@ +// Plan 017 harness check: device A holds a file; device B downloads it over WebRTC. +import { chromium } from 'playwright'; +const b = await chromium.launch({ + executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome', + args: ['--disable-features=WebRtcHideLocalIpsWithMdns'], +}); +const pages = []; +for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); p.on('console', (m) => { if (m.type() === 'error') console.log('console', i, m.text()); }); await p.goto('http://localhost:8770/'); pages.push(p); } +const setup = (p) => p.evaluate(async () => { + P2PClient.start({ getSettings: () => ({}) }); + await P2PClient.sync(); + for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100)); + P2PTransfer.start({ iceServers: [] }); + return P2PClient.peer(); +}); +// Holder: write a 5 MB file + registry record. +const held = await pages[0].evaluate(async () => { + const bytes = new Uint8Array(5 * 1024 * 1024 + 777).map((_, i) => (i * 31 + 7) % 256); + const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true }); + const w = await (await dir.getFileHandle('heldAAAAAAA.mp4', { create: true })).createWritable(); + await w.write(bytes); await w.close(); + const cid = Sha256.hex(bytes); + await DeviceDB.putFile({ videoId: 'heldAAAAAAA', cid, size: bytes.length, state: 'verified' }); + return { cid, size: bytes.length }; +}); +const peerA = await setup(pages[0]); +await setup(pages[1]); +const t0 = Date.now(); +const res = await pages[1].evaluate(async ({ peerA, held }) => { + const good = await P2PTransfer.download({ videoId: 'gotAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] }); + const list = (await OPFS.listVideos()).map((x) => [x.id, x.size]); + const bad = await P2PTransfer.download({ videoId: 'badAAAAAAAA', cid: 'f'.repeat(64), size: held.size, peers: [{ peer: peerA }] }); + const list2 = (await OPFS.listVideos()).map((x) => x.id); + return { good, list, bad, list2 }; +}, { peerA, held }); +// Holder's file gets corrupted (same size, one byte flipped) but its record still claims the cid. +await pages[0].evaluate(async () => { + const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos'); + const fh = await dir.getFileHandle('heldAAAAAAA.mp4'); + const bytes = new Uint8Array(await (await fh.getFile()).arrayBuffer()); + bytes[12345] ^= 0xff; + const w = await fh.createWritable(); await w.write(bytes); await w.close(); +}); +res.corrupt = await pages[1].evaluate(async ({ peerA, held }) => { + const r = await P2PTransfer.download({ videoId: 'corAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] }); + await new Promise((x) => setTimeout(x, 300)); + const names = []; + for await (const [n] of (await (await navigator.storage.getDirectory()).getDirectoryHandle('videos')).entries()) names.push(n); + return { r, names: names.sort() }; +}, { peerA, held }); +console.log(JSON.stringify({ ...res, ms: Date.now() - t0 })); +await b.close(); diff --git a/plans/harness/transfer-server.js b/plans/harness/transfer-server.js new file mode 100644 index 0000000..6dac48f --- /dev/null +++ b/plans/harness/transfer-server.js @@ -0,0 +1,24 @@ +// Plan 017 harness: p2p-routes + p2p-hub + the transfer scripts (port 8770). +// Run from server/: bun ../plans/harness/transfer-server.js +const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js'); +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +const root = mkdtempSync(join(tmpdir(), 'ytp-h14-')); +process.env.DB_PATH = join(root, 'h.db'); +const dbmod = await import(process.cwd() + '/db.js'); +const p2pDb = await import(process.cwd() + '/p2p-db.js'); +const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js'); +const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js'); +const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname; +await dbmod.initDb(); await p2pDb.initP2pSchema(); +const app = new Hono(); +registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' }); +const hub = createP2pHub({ getDevice: p2pDb.getDevice }); +app.get('/', (c) => c.html('')); +app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } })); +Bun.serve({ port: 8770, fetch(req, server) { + if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server); + return app.fetch(req, server); +}, websocket: hub.websocket }); +console.log('ready'); diff --git a/plans/patches/009-media-cache-test.diff b/plans/patches/009-media-cache-test.diff new file mode 100644 index 0000000..c67fdcb --- /dev/null +++ b/plans/patches/009-media-cache-test.diff @@ -0,0 +1,63 @@ +--- a/server/media-cache.test.js ++++ b/server/media-cache.test.js +@@ -10,6 +10,8 @@ + process.env.DB_PATH = join(root, 'test.db'); + const dbmod = await import('./db.js'); + const { createMediaCache, validateMedia, MediaSkip, HIGH, LOW, OPT_REV } = await import('./media-cache.js'); ++const { initP2pSchema } = await import('./p2p-db.js'); ++const { sha256File } = await import('./hash.js'); + + const fx = (name) => join(root, name); + const ff = (...args) => { +@@ -26,6 +28,7 @@ + + beforeAll(async () => { + await dbmod.initDb(); ++ await initP2pSchema(); // adds media_cache.sha256 + // 10 s 640x360 H.264 + AAC — the shape of a good cached copy. + ff('-f', 'lavfi', '-i', 'testsrc=size=640x360:rate=30:duration=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=10', + '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', fx('good.mp4')); +@@ -102,6 +105,7 @@ + download: async (id, out) => { calls.push(id); await sleep(30); copyFileSync(fx(state.fixture), out); return out; }, + transcode: { enabled: false }, + freeBytes: () => 100 * 1024 ** 3, ++ backfillDelayMs: -1, + ...opts, + }); + return { cache, dir, calls, state }; +@@ -110,6 +114,35 @@ + for (const r of await dbmod.listMedia()) await dbmod.deleteMedia(r.video_id); + } + ++describe('content hashes (P2P cids)', () => { ++ test('a promoted copy stores the sha256 of its mp4 and reports it once', async () => { ++ await clearDb(); ++ const seen = []; ++ const { cache, dir } = makeCache({ onReady: (i) => { seen.push(i); } }); ++ await cache.init(); ++ const row = await cache.ensureCached('hashAAAAAA1', { priority: HIGH }); ++ const want = await sha256File(join(dir, `hashAAAAAA1.${row.gen}.mp4`)); ++ expect((await dbmod.getMedia('hashAAAAAA1')).sha256).toBe(want); ++ expect(await until(() => seen.length === 1)).toBe(true); ++ expect(seen[0]).toMatchObject({ id: 'hashAAAAAA1', gen: row.gen, sha256: want, vcodec: 'h264' }); ++ expect(seen[0].meta.title).toBe('T hashAAAAAA1'); ++ }); ++ ++ test('backfill hashes ready copies that have no sha256 yet', async () => { ++ await clearDb(); ++ const seen = []; ++ const { cache } = makeCache({ onReady: (i) => { seen.push(i); } }); ++ await cache.init(); ++ await cache.ensureCached('hashAAAAAA2', { priority: HIGH }); ++ await dbmod.upsertMedia('hashAAAAAA2', { sha256: null }); ++ seen.length = 0; ++ expect(await cache.backfillHashes()).toBe(1); ++ expect((await dbmod.getMedia('hashAAAAAA2')).sha256).toMatch(/^[0-9a-f]{64}$/); ++ expect(await until(() => seen.length === 1)).toBe(true); ++ expect(await cache.backfillHashes()).toBe(0); // nothing left ++ }); ++}); ++ + describe('media cache jobs', () => { + test('fetches once, dedupes concurrent requests, then serves from disk', async () => { + await clearDb(); diff --git a/plans/patches/009-media-cache.diff b/plans/patches/009-media-cache.diff new file mode 100644 index 0000000..e521317 --- /dev/null +++ b/plans/patches/009-media-cache.diff @@ -0,0 +1,120 @@ +--- a/server/media-cache.js ++++ b/server/media-cache.js +@@ -24,6 +24,7 @@ + statSync, statfsSync, unlinkSync, + } from 'node:fs'; + import { join } from 'node:path'; ++import { sha256File } from './hash.js'; + + export const HIGH = 0; // explicit save / Broken re-download + export const LOW = 1; // auto-cache after a play +@@ -167,6 +168,9 @@ + freeBytes = () => { const s = statfsSync(dir); return s.bavail * s.bsize; }, + now = () => Date.now(), + log = console, ++ hashFile = sha256File, // server-computed SHA-256 = the file's P2P content id ++ onReady = null, // ({ id, gen, path, sha256, size, … }) after a validated copy lands ++ backfillDelayMs = 30_000, // hash pre-existing copies this long after init() + } = {}) { + const tmpDir = join(dir, '.tmp'); + const fileFor = (id, gen, kind = 'mp4') => join(dir, `${id}.${gen}.${kind}`); +@@ -236,6 +240,20 @@ + } + } + ++ // Tell the P2P layer a validated file with a server-computed hash exists. ++ // Never throws and never delays the caller. ++ function notifyReady(id, gen, sha256, probe, metaJson) { ++ if (!onReady || !sha256) return; ++ let meta = {}; ++ try { meta = JSON.parse(metaJson || '{}'); } catch { /* keep {} */ } ++ Promise.resolve() ++ .then(() => onReady({ ++ id, gen, path: fileFor(id, gen), sha256, size: probe.size, height: probe.height, ++ vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta, ++ })) ++ .catch((e) => log.warn?.(`[media] onReady ${id} failed: ${e.message}`)); ++ } ++ + async function setStatus(job, status) { + job.status = status; + await db.upsertMedia(job.id, { status, updated_at: now() }); +@@ -308,6 +326,7 @@ + '-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]); + if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr)); + ++ const sha256 = await hashFile(norm); + const prev = await db.getMedia(id); + const gen = ((prev && prev.gen) || 0) + 1; + renameSync(norm, fileFor(id, gen)); +@@ -318,13 +337,14 @@ + size: probe.size + sizeOf(fileFor(id, gen, 'm4a')), + height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, + optimized: 0, meta: JSON.stringify(metaFromInfo(info, probe.duration)), +- attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t, ++ attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t, sha256, + }; + await db.upsertMedia(id, fields); + removeFiles(id, gen); + job.status = 'ready'; + log.info?.(`[media] cached ${id} ${probe.height}p ${fmtMB(fields.size)}`); + enqueueOptimize(id); ++ notifyReady(id, gen, sha256, probe, fields.meta); + return { ...(prev || {}), video_id: id, ...fields }; + } catch (err) { + job.status = 'failed'; +@@ -427,6 +447,7 @@ + log.info?.(`[media] kept original ${id}: ${probe.vcodec} ${fmtMB(probe.size)} vs ${fmtMB(oldSize)}`); + return; + } ++ const sha256 = await hashFile(out); + const gen = row.gen + 1; + renameSync(out, fileFor(id, gen)); + const oldAudio = fileFor(id, row.gen, 'm4a'); +@@ -434,8 +455,9 @@ + try { linkSync(oldAudio, newAudio); } catch { copyFileSync(oldAudio, newAudio); } + await db.upsertMedia(id, { + gen, size: probe.size + sizeOf(newAudio), height: probe.height, vcodec: probe.vcodec, +- duration: probe.duration, optimized: OPT_REV, updated_at: now(), ++ duration: probe.duration, optimized: OPT_REV, updated_at: now(), sha256, + }); ++ notifyReady(id, gen, sha256, probe, cur.meta); + removeFiles(id, gen, OLD_GEN_GRACE_MS); + log.info?.(`[media] optimized ${id}: ${fmtMB(oldSize)} → ${fmtMB(probe.size)} (${probe.vcodec})`); + } catch (err) { +@@ -622,7 +644,34 @@ + } + const s = await db.mediaStats(); + log.info?.(`[media] ${s.count} cached (${fmtMB(s.bytes)}), ${resumed} job(s) resumed, dir ${dir}`); ++ if (backfillDelayMs >= 0) { ++ const t = setTimeout(() => backfillHashes().catch(() => {}), backfillDelayMs); ++ t.unref?.(); ++ } ++ } ++ ++ // Copies cached before hashing existed get their sha256 in the background, ++ // one at a time, so their content ids can be admitted too. ++ async function backfillHashes() { ++ let n = 0; ++ for (const r of await db.listMedia()) { ++ if (r.status !== 'ready' || r.sha256) continue; ++ const p = fileFor(r.video_id, r.gen); ++ if (!existsSync(p)) continue; ++ try { ++ const sha256 = await hashFile(p); ++ const cur = await db.getMedia(r.video_id); ++ if (!cur || cur.status !== 'ready' || cur.gen !== r.gen) continue; ++ await db.upsertMedia(r.video_id, { sha256 }); ++ notifyReady(r.video_id, r.gen, sha256, ++ { size: sizeOf(p), height: r.height, vcodec: r.vcodec, acodec: r.acodec, duration: r.duration }, r.meta); ++ n++; ++ } catch (e) { ++ log.warn?.(`[media] hash backfill ${r.video_id}: ${e.message}`); ++ } ++ } ++ return n; + } + +- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId }; ++ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes }; + } diff --git a/plans/patches/010-retention-eviction.diff b/plans/patches/010-retention-eviction.diff new file mode 100644 index 0000000..3a2b3a6 --- /dev/null +++ b/plans/patches/010-retention-eviction.diff @@ -0,0 +1,44 @@ +--- a/server/media-cache.js ++++ b/server/media-cache.js +@@ -226,7 +226,10 @@ + let { bytes } = await db.mediaStats(); + if (bytes + est > maxBytes) { + const cutoff = now() - EVICT_PROTECT_MS; +- for (const r of await db.listMediaLru()) { ++ // Retention order (plan 010) when the server provides it: copies that ++ // are neither top nor recent go first. Otherwise plain LRU. ++ const order = db.listMediaEvictionOrder ? await db.listMediaEvictionOrder() : await db.listMediaLru(); ++ for (const r of order) { + if (bytes + est <= maxBytes) break; + if (r.video_id === id || r.last_access > cutoff || jobs.has(r.video_id) || optActive === r.video_id) continue; + await evict(r.video_id, 'budget'); +--- a/server/media-cache.test.js ++++ b/server/media-cache.test.js +@@ -240,6 +240,27 @@ + expect(await cache.getReady('lruAAAAAAA3')).not.toBeNull(); + }); + ++ test('eviction follows listMediaEvictionOrder when the db provides it', async () => { ++ await clearDb(); ++ const size = readFileSync(fx('good.mp4')).length; ++ const db = { ++ getMedia: dbmod.getMedia, upsertMedia: dbmod.upsertMedia, deleteMedia: dbmod.deleteMedia, ++ listMedia: dbmod.listMedia, listMediaLru: dbmod.listMediaLru, touchMedia: dbmod.touchMedia, ++ mediaStats: dbmod.mediaStats, ++ // Retention says the MORE recently played #2 is the one to drop (e.g. no views). ++ listMediaEvictionOrder: async () => (await dbmod.listMediaLru()).reverse(), ++ }; ++ const { cache } = makeCache({ db, maxBytes: size * 2 + 60 * 250 * 1024 + 50_000, duration: 10 }); ++ await cache.init(); ++ await cache.ensureCached('retAAAAAAA1', { priority: HIGH }); ++ await cache.ensureCached('retAAAAAAA2', { priority: HIGH }); ++ await dbmod.upsertMedia('retAAAAAAA1', { last_access: 1000 }); ++ await dbmod.upsertMedia('retAAAAAAA2', { last_access: 2000 }); ++ await cache.ensureCached('retAAAAAAA3', { priority: HIGH }); ++ expect(await dbmod.getMedia('retAAAAAAA2')).toBeNull(); ++ expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull(); ++ }); ++ + test('free-disk guard skips caching', async () => { + await clearDb(); + const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 }); diff --git a/plans/patches/012-opfs-hash.diff b/plans/patches/012-opfs-hash.diff new file mode 100644 index 0000000..b2aa1b2 --- /dev/null +++ b/plans/patches/012-opfs-hash.diff @@ -0,0 +1,75 @@ +--- a/frontend/opfs-worker.js ++++ b/frontend/opfs-worker.js +@@ -13,12 +13,20 @@ + * Out messages: + * { type: 'unsupported' } → caller falls back to main thread + * { type: 'progress', received } → bytes written so far +- * { type: 'done', ext } → file stored as . ++ * { type: 'done', ext, sha256, expectedSha, size } ++ * → file stored as .; ++ * sha256 = hash of the stored bytes ++ * (P2P content id), expectedSha = the ++ * server's X-Content-SHA256 or null + * { type: 'error', error } → failed; .part cleaned up + * ========================================================================== */ + + 'use strict'; + ++// Incremental SHA-256 (frontend/sha256.js): the file is hashed while it is ++// written, so the device knows its content id without re-reading the file. ++try { importScripts('/sha256.js'); } catch { /* hashing unavailable — save still works */ } ++ + async function getVideosDir() { + const root = await navigator.storage.getDirectory(); + return root.getDirectoryHandle('videos', { create: true }); +@@ -60,6 +68,7 @@ + dir = await getVideosDir(); + const partHandle = await dir.getFileHandle(partName, { create: true }); + const access = await partHandle.createSyncAccessHandle(); ++ const hasher = self.Sha256 ? self.Sha256.create() : null; + let offset = 0; + try { + const reader = res.body.getReader(); +@@ -67,6 +76,7 @@ + const { done, value } = await reader.read(); + if (done) break; + access.write(value, { at: offset }); ++ if (hasher) hasher.update(value); + offset += value.byteLength; + self.postMessage({ type: 'progress', received: offset }); + } +@@ -81,6 +91,14 @@ + if (expected > 0 && offset !== expected) { + throw new Error(`download cut short (${offset} of ${expected} bytes)`); + } ++ // The server names the hash of what it sent (media cache copies). A ++ // mismatch means the bytes were damaged on the way — never keep them. ++ const sha256 = hasher ? hasher.hex() : null; ++ const sent = (res.headers.get('x-content-sha256') || '').trim().toLowerCase(); ++ const expectedSha = /^[0-9a-f]{64}$/.test(sent) ? sent : null; ++ if (sha256 && expectedSha && sha256 !== expectedSha) { ++ throw new Error('integrity check failed (content hash mismatch)'); ++ } + + // Finalize: .part → permanent name. Prefer the native rename, but treat + // ANY move() failure as "unavailable" and fall back to a chunked copy — +@@ -111,7 +129,7 @@ + await dir.removeEntry(partName); + } + +- self.postMessage({ type: 'done', ext }); ++ self.postMessage({ type: 'done', ext, sha256, expectedSha, size: offset }); + } catch (err) { + // Never leave a corrupt partial behind + try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ } +--- a/frontend/opfs.js ++++ b/frontend/opfs.js +@@ -136,7 +136,7 @@ + }; + worker.onmessage = (e) => { + const m = e.data || {}; +- if (m.type === 'done') finish({ ok: true }); ++ if (m.type === 'done') finish({ ok: true, sha256: m.sha256 || null, expectedSha: m.expectedSha || null, size: m.size || 0 }); + else if (m.type === 'unsupported') finish({ ok: false, fallback: true }); + else if (m.type === 'error') finish({ ok: false, error: m.error }); + // 'progress' messages are informational; ignored here diff --git a/plans/patches/013-opfs-readrange.diff b/plans/patches/013-opfs-readrange.diff new file mode 100644 index 0000000..120da5f --- /dev/null +++ b/plans/patches/013-opfs-readrange.diff @@ -0,0 +1,22 @@ +--- a/frontend/opfs.js ++++ b/frontend/opfs.js +@@ -108,6 +108,19 @@ + } + }, + ++ // Bytes [offset, offset+length) of a saved video — answers the server's ++ // P2P range challenges (p2p-client.js). null when the file is missing. ++ async readRange(videoId, offset, length) { ++ try { ++ const found = await findHandle(videoId); ++ if (!found) return null; ++ const file = await found[0].getFile(); ++ return new Uint8Array(await file.slice(offset, offset + length).arrayBuffer()); ++ } catch { ++ return null; ++ } ++ }, ++ + revokeUrl(url) { + if (url && _blobUrls.has(url)) { + URL.revokeObjectURL(url); diff --git a/plans/patches/014-p2p-client-presence.diff b/plans/patches/014-p2p-client-presence.diff new file mode 100644 index 0000000..e73efc0 --- /dev/null +++ b/plans/patches/014-p2p-client-presence.diff @@ -0,0 +1,105 @@ +--- a/frontend/p2p-client.js ++++ b/frontend/p2p-client.js +@@ -6,6 +6,9 @@ + * changed() a save/delete happened → re-report soon + * device() { deviceId, secret } or null + * authHeaders() { 'X-Device': … } for other P2P calls ++ * onMessage(type, fn) / signal(to, data) / peer() ++ * live /ws/p2p socket (plan 014): the ++ * device is "online" while it is open + * + * What it does, in order, each sync: + * 1. registers the device once (localStorage ytpDevice) +@@ -148,6 +151,74 @@ + return j; + } + ++ // ---- presence socket (/ws/p2p) — plan 014 ------------------------------------ ++ // Open while sharing OR receiving is on. Credentials go in the first message, ++ // never the URL. Reconnects with backoff (5 s … 5 min). ++ let ws = null; ++ let myPeer = null; ++ let wsRetry = 0; ++ let wsTimer = null; ++ const listeners = new Map(); // type -> Set ++ ++ const wantSocket = () => { ++ const s = hooks.getSettings(); ++ return s.p2pShare !== false || s.p2pReceive !== false; ++ }; ++ ++ function onMessage(type, fn) { ++ if (!listeners.has(type)) listeners.set(type, new Set()); ++ listeners.get(type).add(fn); ++ return () => listeners.get(type).delete(fn); ++ } ++ function emit(m) { ++ for (const fn of listeners.get(m.type) || []) { try { fn(m); } catch { /* a listener's bug is not ours */ } } ++ } ++ ++ function connect() { ++ if (ws || !wantSocket()) return; ++ const d = device(); ++ if (!d) return; ++ const proto = location.protocol === 'https:' ? 'wss:' : 'ws:'; ++ let sock; ++ try { sock = new WebSocket(`${proto}//${location.host}/ws/p2p`); } catch { return; } ++ ws = sock; ++ let ping = null; ++ sock.onopen = () => { ++ sock.send(JSON.stringify({ type: 'auth', device: d.deviceId, secret: d.secret })); ++ // The server drops sockets idle for 120 s (server.js websocketHandler). ++ ping = setInterval(() => { try { sock.send('{"type":"ping"}'); } catch { /* closing */ } }, 50_000); ++ }; ++ sock.onmessage = (e) => { ++ let m; ++ try { m = JSON.parse(e.data); } catch { return; } ++ if (m.type === 'hello') { myPeer = m.peer; wsRetry = 0; } ++ emit(m); ++ }; ++ sock.onclose = () => { ++ clearInterval(ping); ++ if (ws !== sock) return; ++ ws = null; ++ myPeer = null; ++ if (!wantSocket()) return; ++ clearTimeout(wsTimer); ++ wsTimer = setTimeout(connect, Math.min(300_000, 5000 * 2 ** wsRetry++)); ++ }; ++ } ++ ++ function disconnect() { ++ clearTimeout(wsTimer); ++ const sock = ws; ++ ws = null; ++ myPeer = null; ++ if (sock) { try { sock.close(); } catch { /* gone */ } } ++ } ++ ++ function signal(to, data) { ++ if (!ws || ws.readyState !== 1) return false; ++ ws.send(JSON.stringify({ type: 'signal', to, data })); ++ return true; ++ } ++ + async function syncOnce() { + if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null; + if (navigator.onLine === false) return null; +@@ -158,6 +229,7 @@ + await hashPending(recs); + const res = await report(recs, dev); + lastSync = Date.now(); ++ if (wantSocket()) connect(); else disconnect(); + return res; + } + +@@ -185,5 +257,8 @@ + }); + } + +- window.P2PClient = { start, changed, sync, device, authHeaders, config }; ++ window.P2PClient = { ++ start, changed, sync, device, authHeaders, config, ++ onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer), ++ }; + }()); diff --git a/plans/patches/014-p2p-hub-new.diff b/plans/patches/014-p2p-hub-new.diff new file mode 100644 index 0000000..60d207c --- /dev/null +++ b/plans/patches/014-p2p-hub-new.diff @@ -0,0 +1,213 @@ +--- /dev/null ++++ b/server/p2p-hub.js +@@ -0,0 +1,120 @@ ++/* ============================================================================ ++ * p2p-hub.js — live presence + WebRTC signalling for P2P devices, and the ++ * availability payload (docs/p2p-architecture.md flows 4–5). ++ * ++ * /ws/p2p (credentials never go in the URL — proxies log URLs) ++ * you → server {type:'auth', device, secret} first message, within 10 s ++ * server → you {type:'hello', peer} after auth ++ * you → peer {type:'signal', to:, data} relayed as ++ * server → peer {type:'signal', from:, data} ++ * server → you {type:'error', error} ++ * Server-initiated messages (plan 018) go through hub.send(deviceId, msg). ++ * ++ * "Online" lives only in memory: a server restart forgets it, a closed ++ * socket drops it. Holder ROWS are persistent (p2p_holders) — the payload ++ * reports both: `online` (now) and `lastVerifiedAt` / `stale` (history). ++ * Peers are addressed by an opaque id (peerIdOf), never by device id. ++ * ========================================================================== */ ++import { createHash, timingSafeEqual } from 'node:crypto'; ++ ++const sha = (s) => createHash('sha256').update(String(s)).digest('hex'); ++const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); }; ++export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12); ++ ++const MAX_MSG = 64 * 1024; ++const BUDGET_PER_MIN = 300; ++ ++export function createP2pHub({ getDevice, enabled = () => true, now = () => Date.now(), log = console } = {}) { ++ const online = new Map(); // deviceId -> ws ++ const byPeer = new Map(); // peer -> deviceId (online only) ++ ++ const send = (ws, m) => { try { ws.send(JSON.stringify(m)); } catch { /* gone */ } }; ++ ++ function upgrade(req, server) { ++ if (!enabled()) return new Response('p2p disabled', { status: 404 }); ++ const ok = server.upgrade(req, { data: { hub: 'p2p', authed: false, budget: [] } }); ++ return ok ? undefined : new Response('upgrade failed', { status: 400 }); ++ } ++ ++ function open(ws) { ++ const t = setTimeout(() => { if (!ws.data.authed) { try { ws.close(4401, 'auth timeout'); } catch { /* gone */ } } }, 10_000); ++ t.unref?.(); ++ } ++ ++ async function auth(ws, m) { ++ const deviceId = String(m.device || ''); ++ const secret = String(m.secret || ''); ++ const d = /^dev_[0-9a-f]{16}$/.test(deviceId) ? await getDevice(deviceId).catch(() => null) : null; ++ if (!d || !same(d.secret_hash, sha(secret))) { try { ws.close(4401, 'unknown device'); } catch { /* gone */ } return; } ++ ws.data.authed = true; ++ ws.data.deviceId = deviceId; ++ ws.data.peer = peerIdOf(deviceId); ++ const prev = online.get(deviceId); ++ if (prev && prev !== ws) { prev.data.deviceId = null; try { prev.close(4409, 'replaced'); } catch { /* gone */ } } ++ online.set(deviceId, ws); ++ byPeer.set(ws.data.peer, deviceId); ++ send(ws, { type: 'hello', peer: ws.data.peer }); ++ } ++ ++ async function message(ws, raw) { ++ const t = now(); ++ ws.data.budget = ws.data.budget.filter((x) => t - x < 60_000); ++ if (ws.data.budget.length >= BUDGET_PER_MIN) { send(ws, { type: 'error', error: 'slow down' }); return; } ++ ws.data.budget.push(t); ++ const s = typeof raw === 'string' ? raw : Buffer.from(raw).toString('utf8'); ++ if (s.length > MAX_MSG) return; ++ let m; ++ try { m = JSON.parse(s); } catch { return; } ++ if (!ws.data.authed) { if (m && m.type === 'auth' && !ws.data.authing) { ws.data.authing = true; await auth(ws, m); } return; } ++ if (m && m.type === 'signal' && typeof m.to === 'string') { ++ const target = online.get(byPeer.get(m.to)); ++ if (!target || target === ws) { send(ws, { type: 'error', error: 'peer offline', to: m.to }); return; } ++ send(target, { type: 'signal', from: ws.data.peer, data: m.data }); ++ } ++ } ++ ++ function close(ws) { ++ const { deviceId, peer } = ws.data || {}; ++ if (deviceId && online.get(deviceId) === ws) { ++ online.delete(deviceId); ++ byPeer.delete(peer); ++ } ++ } ++ ++ return { ++ upgrade, ++ websocket: { open, message, close }, ++ isOnline: (deviceId) => online.has(deviceId), ++ send: (deviceId, msg) => { const ws = online.get(deviceId); if (!ws) return false; send(ws, msg); return true; }, ++ onlineCount: () => online.size, ++ }; ++} ++ ++// GET /api/p2p/holders payload. Only devices that share are listed; holders ++// are never dropped for age — `stale` just flags an old last check. ++export async function holdersPayload({ videoId, cid, p2pDb, isOnline, staleDays, serverHas, now = Date.now() }) { ++ const contents = cid ++ ? [await p2pDb.getContent(cid)].filter((c) => c && c.status === 'verified') ++ : await p2pDb.listContentForVideo(videoId); ++ const staleMs = staleDays * 86400_000; ++ const out = []; ++ for (const c of contents) { ++ const holders = (await p2pDb.listHolders(c.cid, 50)) ++ .filter((h) => Number(h.share) === 1) ++ .map((h) => ({ ++ peer: peerIdOf(h.device_id), ++ online: isOnline(h.device_id), ++ lastVerifiedAt: Number(h.last_verified_at), ++ stale: now - Number(h.last_verified_at) > staleMs, ++ trust: h.trust, ++ })) ++ .sort((a, b) => (b.online - a.online) || (b.lastVerifiedAt - a.lastVerifiedAt)); ++ out.push({ ++ cid: c.cid, videoId: c.video_id, size: Number(c.size), height: Number(c.height), vcodec: c.vcodec, ++ serverHas: !!(await serverHas(c.cid)), ++ counts: { holders: holders.length, online: holders.filter((h) => h.online).length, fresh: holders.filter((h) => !h.stale).length }, ++ holders, ++ }); ++ } ++ return { ok: true, staleDays, cids: out }; ++} +--- /dev/null ++++ b/server/p2p-hub.test.js +@@ -0,0 +1,87 @@ ++// Presence/signalling hub and the holders payload (plan 014). ++import { test, expect, beforeAll } from 'bun:test'; ++import { mkdtempSync } from 'node:fs'; ++import { tmpdir } from 'node:os'; ++import { join } from 'node:path'; ++import { createHash } from 'node:crypto'; ++ ++const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-hub-')); ++process.env.DB_PATH = join(root, 'test.db'); ++const dbmod = await import('./db.js'); ++const p2pDb = await import('./p2p-db.js'); ++const { createP2pHub, holdersPayload, peerIdOf } = await import('./p2p-hub.js'); ++ ++const sha = (s) => createHash('sha256').update(s).digest('hex'); ++const SECRET_A = '1'.repeat(64); ++const SECRET_B = '2'.repeat(64); ++const CID = 'a'.repeat(64); ++const DAY = 86400_000; ++const NOW = Date.UTC(2026, 8, 29, 12); ++ ++function fakeWs(data) { ++ return { data, sent: [], closed: null, send(s) { this.sent.push(JSON.parse(s)); }, close(code) { this.closed = code; } }; ++} ++ ++beforeAll(async () => { ++ await dbmod.initDb(); ++ await p2pDb.initP2pSchema(); ++ await p2pDb.createDevice({ deviceId: 'dev_000000000000000a', secretHash: sha(SECRET_A), now: NOW }); ++ await p2pDb.createDevice({ deviceId: 'dev_000000000000000b', secretHash: sha(SECRET_B), now: NOW }); ++ await p2pDb.createDevice({ deviceId: 'dev_000000000000000c', secretHash: sha('x'), now: NOW }); ++ await p2pDb.touchDevice('dev_000000000000000c', { now: NOW, share: false }); ++ await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1234, height: 720, vcodec: 'h264', origin: 'server', now: NOW }); ++ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000a', now: NOW - 1 * DAY }); ++ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000b', now: NOW - 30 * DAY }); // stale, still listed ++ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000c', now: NOW }); // share off → hidden ++}); ++ ++test('auth on open, hello with opaque peer id, bad secret is closed', async () => { ++ const hub = createP2pHub({ getDevice: p2pDb.getDevice }); ++ const a = fakeWs({ budget: [] }); ++ hub.websocket.open(a); ++ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: 'x', data: {} })); // ignored before auth ++ expect(a.sent.length).toBe(0); ++ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A })); ++ expect(a.sent[0]).toEqual({ type: 'hello', peer: peerIdOf('dev_000000000000000a') }); ++ expect(hub.isOnline('dev_000000000000000a')).toBe(true); ++ const bad = fakeWs({ budget: [] }); ++ hub.websocket.open(bad); ++ await hub.websocket.message(bad, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_A })); ++ expect(bad.closed).toBe(4401); ++ expect(hub.isOnline('dev_000000000000000b')).toBe(false); ++}); ++ ++test('signals are relayed between online peers only; close drops presence', async () => { ++ const hub = createP2pHub({ getDevice: p2pDb.getDevice }); ++ const a = fakeWs({ budget: [] }); ++ const b = fakeWs({ budget: [] }); ++ hub.websocket.open(a); ++ hub.websocket.open(b); ++ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A })); ++ await hub.websocket.message(b, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_B })); ++ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: { sdp: 'x' } })); ++ expect(b.sent.at(-1)).toEqual({ type: 'signal', from: a.data.peer, data: { sdp: 'x' } }); ++ hub.websocket.close(b); ++ expect(hub.isOnline('dev_000000000000000b')).toBe(false); ++ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: {} })); ++ expect(a.sent.at(-1)).toMatchObject({ type: 'error', error: 'peer offline' }); ++ expect(hub.send('dev_000000000000000a', { type: 'ping' })).toBe(true); ++ expect(hub.send('dev_000000000000000b', { type: 'ping' })).toBe(false); ++}); ++ ++test('holders payload: persistent rows, online flag, stale marker, share-off hidden', async () => { ++ const online = new Set(['dev_000000000000000b']); ++ const p = await holdersPayload({ ++ videoId: 'dQw4w9WgXcQ', p2pDb, isOnline: (d) => online.has(d), staleDays: 7, ++ serverHas: async () => false, now: NOW, ++ }); ++ expect(p.staleDays).toBe(7); ++ expect(p.cids.length).toBe(1); ++ const c = p.cids[0]; ++ expect(c).toMatchObject({ cid: CID, size: 1234, height: 720, serverHas: false, counts: { holders: 2, online: 1, fresh: 1 } }); ++ expect(c.holders.map((h) => [h.peer, h.online, h.stale])).toEqual([ ++ [peerIdOf('dev_000000000000000b'), true, true], // online first even though stale ++ [peerIdOf('dev_000000000000000a'), false, false], ++ ]); ++ expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids ++}); diff --git a/plans/patches/015-p2p-core-new.diff b/plans/patches/015-p2p-core-new.diff new file mode 100644 index 0000000..eaa3284 --- /dev/null +++ b/plans/patches/015-p2p-core-new.diff @@ -0,0 +1,107 @@ +--- /dev/null ++++ b/frontend/p2p-core.js +@@ -0,0 +1,55 @@ ++/* ============================================================================ ++ * p2p-core.js — pure helpers for peer-to-peer UI (window.P2PCore / node) ++ * ++ * formatAvailability(payload, now) turns GET /api/p2p/holders into the line ++ * under the now-playing title, e.g. ++ * "📡 On 3 devices · 1 online now · last checked 2d ago" ++ * Holders are persistent (docs/p2p-architecture.md): they are never hidden ++ * for age; when every holder is older than the server's staleDays the line ++ * says so and is marked stale. ++ * ========================================================================== */ ++(function (root) { ++ 'use strict'; ++ ++ function ago(ms, now) { ++ const s = Math.max(0, Math.round((now - ms) / 1000)); ++ if (s < 90) return 'just now'; ++ const m = Math.round(s / 60); ++ if (m < 90) return m + 'm ago'; ++ const h = Math.round(m / 60); ++ if (h < 48) return h + 'h ago'; ++ return Math.round(h / 24) + 'd ago'; ++ } ++ ++ function formatAvailability(payload, now) { ++ if (!payload || !payload.ok || !Array.isArray(payload.cids)) return null; ++ const holders = []; ++ for (const c of payload.cids) for (const h of c.holders || []) holders.push(h); ++ if (!holders.length) return null; ++ const online = holders.filter((h) => h.online).length; ++ const fresh = holders.filter((h) => !h.stale).length; ++ const newest = Math.max(...holders.map((h) => Number(h.lastVerifiedAt) || 0)); ++ const n = holders.length; ++ const allStale = fresh === 0; ++ const text = `📡 On ${n} device${n === 1 ? '' : 's'} · ${online ? online + ' online now' : 'none online'}` ++ + ` · last checked ${ago(newest, now)}` + (allStale ? ' (not checked recently)' : ''); ++ const title = holders.map((h) => ++ `${h.peer} · ${h.online ? 'online' : 'offline'} · checked ${ago(Number(h.lastVerifiedAt) || 0, now)}` ++ + `${h.stale ? ' (stale)' : ''}${h.trust === 'challenged' ? ' · spot-checked' : ''}`).join('\n'); ++ return { text, title, stale: allStale, online, holders: n }; ++ } ++ ++ // Holders worth trying for a download: online first, then freshest check. ++ function pickPeers(payload, cid) { ++ if (!payload || !Array.isArray(payload.cids)) return []; ++ const c = payload.cids.find((x) => x.cid === cid) || payload.cids[0]; ++ if (!c) return []; ++ return (c.holders || []).filter((h) => h.online) ++ .sort((a, b) => (b.trust === 'challenged') - (a.trust === 'challenged') || b.lastVerifiedAt - a.lastVerifiedAt) ++ .map((h) => ({ peer: h.peer, cid: c.cid, size: c.size })); ++ } ++ ++ const P2PCore = { ago, formatAvailability, pickPeers }; ++ if (typeof module !== 'undefined' && module.exports) module.exports = P2PCore; ++ else root.P2PCore = P2PCore; ++})(typeof globalThis !== 'undefined' ? globalThis : this); +--- /dev/null ++++ b/frontend/p2p-core.test.js +@@ -0,0 +1,46 @@ ++'use strict'; ++const { test } = require('node:test'); ++const assert = require('node:assert'); ++const C = require('./p2p-core'); ++ ++const NOW = Date.UTC(2026, 8, 29, 12); ++const H = 3600_000; ++const payload = (holders, extra = {}) => ({ ok: true, staleDays: 7, cids: [{ cid: 'a'.repeat(64), size: 10, holders, ...extra }] }); ++ ++test('ago buckets', () => { ++ assert.strictEqual(C.ago(NOW - 30_000, NOW), 'just now'); ++ assert.strictEqual(C.ago(NOW - 20 * 60_000, NOW), '20m ago'); ++ assert.strictEqual(C.ago(NOW - 5 * H, NOW), '5h ago'); ++ assert.strictEqual(C.ago(NOW - 72 * H, NOW), '3d ago'); ++}); ++ ++test('no holders → nothing to show', () => { ++ assert.strictEqual(C.formatAvailability(payload([]), NOW), null); ++ assert.strictEqual(C.formatAvailability(null, NOW), null); ++}); ++ ++test('mixed holders: counts, newest check, not stale', () => { ++ const f = C.formatAvailability(payload([ ++ { peer: 'p1', online: true, lastVerifiedAt: NOW - 30 * 24 * H, stale: true, trust: 'reported' }, ++ { peer: 'p2', online: false, lastVerifiedAt: NOW - 48 * H, stale: false, trust: 'challenged' }, ++ ]), NOW); ++ assert.strictEqual(f.text, '📡 On 2 devices · 1 online now · last checked 2d ago'); ++ assert.strictEqual(f.stale, false); ++ assert.match(f.title, /p1 · online · checked 30d ago \(stale\)/); ++ assert.match(f.title, /p2 · offline · checked 2d ago · spot-checked/); ++}); ++ ++test('every holder stale → still listed, flagged', () => { ++ const f = C.formatAvailability(payload([{ peer: 'p1', online: false, lastVerifiedAt: NOW - 20 * 24 * H, stale: true }]), NOW); ++ assert.strictEqual(f.text, '📡 On 1 device · none online · last checked 20d ago (not checked recently)'); ++ assert.strictEqual(f.stale, true); ++}); ++ ++test('pickPeers: online only, spot-checked first', () => { ++ const p = payload([ ++ { peer: 'off', online: false, lastVerifiedAt: NOW, trust: 'challenged' }, ++ { peer: 'on1', online: true, lastVerifiedAt: NOW, trust: 'reported' }, ++ { peer: 'on2', online: true, lastVerifiedAt: NOW - H, trust: 'challenged' }, ++ ]); ++ assert.deepStrictEqual(C.pickPeers(p).map((x) => x.peer), ['on2', 'on1']); ++}); diff --git a/plans/patches/016-client-intake.diff b/plans/patches/016-client-intake.diff new file mode 100644 index 0000000..70919d3 --- /dev/null +++ b/plans/patches/016-client-intake.diff @@ -0,0 +1,91 @@ +--- a/frontend/p2p-client.js ++++ b/frontend/p2p-client.js +@@ -6,6 +6,9 @@ + * changed() a save/delete happened → re-report soon + * device() { deviceId, secret } or null + * authHeaders() { 'X-Device': … } for other P2P calls ++ * contribute(videoId) / unknownVideos() ++ * hand a saved file the server doesn't ++ * know yet to /api/p2p/intake (plan 016) + * onMessage(type, fn) / signal(to, data) / peer() + * live /ws/p2p socket (plan 014): the + * device is "online" while it is open +@@ -33,6 +36,7 @@ + let again = false; + let timer = null; + let lastSync = 0; ++ let lastUnknown = []; // cids the server did not recognise at the last report + + function device() { + try { +@@ -130,6 +134,7 @@ + if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; } + const j = await r.json(); + if (!j || !j.ok) return null; ++ lastUnknown = Array.isArray(j.unknown) ? j.unknown : []; + const accepted = new Set(j.accepted || []); + for (const rec of recs) { + if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); } +@@ -151,6 +156,33 @@ + return j; + } + ++ // ---- intake (plan 016) --------------------------------------------------------- ++ // Send one saved video to the server so it can hash + validate it and add its ++ // cid to the catalog. Uploads the whole file: only on the user's request ++ // ("Verify & share") or when the server asks for a copy (plan 018). ++ async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) { ++ const dev = await ensureDevice(); ++ const rec = await window.DeviceDB.getFile(videoId); ++ const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null; ++ if (!file) return { ok: false, error: 'not saved on this device' }; ++ const h = { 'X-Device': dev.deviceId + '.' + dev.secret }; ++ const t = await (await fetch('/api/p2p/intake', { ++ method: 'POST', headers: { ...h, 'Content-Type': 'application/json' }, ++ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }), ++ })).json().catch(() => ({ ok: false, error: 'server unreachable' })); ++ if (!t.ok || t.known) { if (t.known) changed(); return t; } ++ const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file })) ++ .json().catch(() => ({ ok: false, error: 'upload failed' })); ++ if (r.ok) changed(); ++ return r; ++ } ++ ++ async function unknownVideos() { ++ if (!lastUnknown.length || !window.DeviceDB) return []; ++ const recs = await window.DeviceDB.listFiles(); ++ return recs.filter((r) => r.cid && lastUnknown.includes(r.cid)).map((r) => r.videoId); ++ } ++ + // ---- presence socket (/ws/p2p) — plan 014 ------------------------------------ + // Open while sharing OR receiving is on. Credentials go in the first message, + // never the URL. Reconnects with backoff (5 s … 5 min). +@@ -258,7 +290,7 @@ + } + + window.P2PClient = { +- start, changed, sync, device, authHeaders, config, ++ start, changed, sync, device, authHeaders, config, contribute, unknownVideos, + onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer), + }; + }()); +--- a/frontend/opfs.js ++++ b/frontend/opfs.js +@@ -108,6 +108,17 @@ + } + }, + ++ // The saved File itself (disk-backed, not read into memory) — used as an ++ // upload body by P2P intake. null when not saved. ++ async getFileObject(videoId) { ++ try { ++ const found = await findHandle(videoId); ++ return found ? await found[0].getFile() : null; ++ } catch { ++ return null; ++ } ++ }, ++ + // Bytes [offset, offset+length) of a saved video — answers the server's + // P2P range challenges (p2p-client.js). null when the file is missing. + async readRange(videoId, offset, length) { diff --git a/plans/patches/016-media-cache-adopt.diff b/plans/patches/016-media-cache-adopt.diff new file mode 100644 index 0000000..68c2a8f --- /dev/null +++ b/plans/patches/016-media-cache-adopt.diff @@ -0,0 +1,81 @@ +--- a/server/media-cache.js ++++ b/server/media-cache.js +@@ -522,6 +522,42 @@ + } + + // The "Broken" button: drop the copy and fetch it again at high priority. ++ // Promote a file that came from somewhere else (P2P intake / rehydrate from ++ // a device) as this video's server copy. The caller has already hashed it ++ // and run validateMedia(). The bytes are kept EXACTLY (no remux — the cid ++ // must stay true); only the audio sidecar is derived. Never replaces an ++ // existing ready copy or races a running fetch. ++ async function adoptFile(id, src, { sha256, probe, meta = {} }) { ++ if (!isMediaId(id)) throw new Error('adopt: bad video id'); ++ const cur = await db.getMedia(id); ++ if (cur && cur.status === 'ready' && readyFilesExist(cur)) return { adopted: false, reason: 'already cached' }; ++ if (jobs.has(id)) return { adopted: false, reason: 'fetch in progress' }; ++ await makeRoom(id, probe.size); ++ const prefix = `${id}-${now()}-adopt`; ++ const m4a = join(tmpDir, `${prefix}.m4a`); ++ try { ++ const ex = await run(ffmpeg, ['-v', 'error', '-nostdin', '-y', '-i', src, ++ '-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]); ++ if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr)); ++ const gen = ((cur && cur.gen) || 0) + 1; ++ try { renameSync(src, fileFor(id, gen)); } catch { copyFileSync(src, fileFor(id, gen)); unlinkSync(src); } ++ renameSync(m4a, fileFor(id, gen, 'm4a')); ++ const t = now(); ++ await db.upsertMedia(id, { ++ status: 'ready', gen, size: probe.size + sizeOf(fileFor(id, gen, 'm4a')), ++ height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, ++ optimized: OPT_REV, // re-encoding would change the bytes other devices hold ++ meta: JSON.stringify(meta), priority: HIGH, auto: 0, attempts: 0, error: null, retry_at: 0, ++ created_at: (cur && cur.created_at) || t, updated_at: t, last_access: t, sha256, ++ }); ++ removeFiles(id, gen); ++ log.info?.(`[media] adopted ${id} ${probe.height}p ${fmtMB(probe.size)} from P2P`); ++ return { adopted: true, gen }; ++ } finally { ++ sweepTmp(prefix); ++ } ++ } ++ + async function redownload(id) { + if (!isMediaId(id)) throw new MediaSkip('invalid video id'); + const existing = jobs.get(id); +@@ -676,5 +712,5 @@ + return n; + } + +- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes }; ++ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes, adoptFile }; + } +--- a/server/media-cache.test.js ++++ b/server/media-cache.test.js +@@ -261,6 +261,26 @@ + expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull(); + }); + ++ test('adoptFile promotes a validated outside file byte-for-byte, once', async () => { ++ await clearDb(); ++ const { cache, dir, calls } = makeCache(); ++ await cache.init(); ++ const src = join(root, 'adopt-src.mp4'); ++ copyFileSync(fx('good-fs.mp4'), src); ++ const bytes = readFileSync(src); ++ const probe = await validateMedia(src, 0); ++ const r = await cache.adoptFile('adoptAAAAA1', src, { sha256: 'e'.repeat(64), probe, meta: { title: 'from a device' } }); ++ expect(r.adopted).toBe(true); ++ const row = await cache.getReady('adoptAAAAA1'); ++ expect(row.sha256).toBe('e'.repeat(64)); ++ expect(row.optimized).toBe(OPT_REV); ++ expect(readFileSync(join(dir, `adoptAAAAA1.${row.gen}.mp4`)).equals(bytes)).toBe(true); ++ expect(existsSync(join(dir, `adoptAAAAA1.${row.gen}.m4a`))).toBe(true); ++ expect(calls.length).toBe(0); // nothing fetched from the source ++ copyFileSync(fx('good-fs.mp4'), src); ++ expect((await cache.adoptFile('adoptAAAAA1', src, { sha256: 'f'.repeat(64), probe })).adopted).toBe(false); ++ }); ++ + test('free-disk guard skips caching', async () => { + await clearDb(); + const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 }); diff --git a/plans/patches/016-p2p-intake-new.diff b/plans/patches/016-p2p-intake-new.diff new file mode 100644 index 0000000..2a5894a --- /dev/null +++ b/plans/patches/016-p2p-intake-new.diff @@ -0,0 +1,230 @@ +--- /dev/null ++++ b/server/p2p-intake.js +@@ -0,0 +1,121 @@ ++/* ============================================================================ ++ * p2p-intake.js — a device hands a file to the server for verification ++ * (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did ++ * not come from the server's own fetch can become verified content. ++ * ++ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? } ++ * → { ok, known:true } cid already verified — nothing to send ++ * → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min ++ * PUT /api/p2p/intake/:ticket (device) raw file body ++ * → { ok, cid, adopted } admitted (+ adopted into the media cache) ++ * → 4xx { ok:false, error } rejected; nothing kept ++ * ++ * Order is fixed: bytes land in P2P_INTAKE_DIR (never served) → the SERVER ++ * hashes them while writing → a claimed cid must match → validateMedia() → ++ * admitFile() (malware scan only when P2P_MALWARE_SCAN=1) → holder row for ++ * the uploader → optional adoption into the media cache. Any failure deletes ++ * the file. ++ * ========================================================================== */ ++import { createHash, randomBytes } from 'node:crypto'; ++import { createWriteStream, mkdirSync, unlinkSync } from 'node:fs'; ++import { join } from 'node:path'; ++ ++const CID_RE = /^[0-9a-f]{64}$/; ++const ID_RE = /^[A-Za-z0-9_-]{6,64}$/; ++const TICKET_TTL = 30 * 60_000; ++const MAX_ACTIVE = 2; ++// Display text from the device — untrusted: control chars/brackets stripped, capped. ++const clean = (v, max) => String(v || '').replace(/[\u0000-\u001f\u007f<>]+/g, ' ').replace(/\s+/g, ' ').trim().slice(0, max); ++ ++export function registerIntakeRoutes(app, deps) { ++ const { ++ cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null, ++ now = () => Date.now(), log = console, ++ } = deps; ++ const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy } ++ let active = 0; ++ mkdirSync(cfg.intakeDir, { recursive: true }); ++ ++ const sweep = () => { const t = now(); for (const [k, v] of tickets) if (!v.busy && v.expiresAt < t) tickets.delete(k); }; ++ ++ app.post('/api/p2p/intake', gate, requireDevice, async (c) => { ++ const d = c.get('device'); ++ const body = await c.req.json().catch(() => ({})); ++ const videoId = String(body.videoId || ''); ++ const cid = body.cid ? String(body.cid).toLowerCase() : null; ++ const size = Number(body.size); ++ if (!ID_RE.test(videoId) || (cid && !CID_RE.test(cid))) return c.json({ ok: false, error: 'bad videoId or cid' }, 400); ++ if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413); ++ if (cid) { ++ const known = await p2pDb.getContent(cid); ++ if (known && known.status === 'verified') return c.json({ ok: true, known: true }); ++ if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410); ++ } ++ sweep(); ++ for (const v of tickets.values()) if (v.deviceId === d.device_id) return c.json({ ok: false, error: 'an upload from this device is already open' }, 429); ++ const ticket = randomBytes(16).toString('hex'); ++ const expiresAt = now() + TICKET_TTL; ++ const meta = { title: clean(body.title, 300), channel: clean(body.channel, 200) }; ++ tickets.set(ticket, { deviceId: d.device_id, videoId, cid, size, meta, expiresAt, busy: false }); ++ return c.json({ ok: true, ticket, url: `/api/p2p/intake/${ticket}`, expiresAt }); ++ }); ++ ++ app.put('/api/p2p/intake/:ticket', gate, requireDevice, async (c) => { ++ const d = c.get('device'); ++ const ticket = c.req.param('ticket'); ++ const tk = tickets.get(ticket); ++ if (!tk || tk.deviceId !== d.device_id || tk.expiresAt < now()) return c.json({ ok: false, error: 'unknown or expired ticket' }, 404); ++ if (tk.busy) return c.json({ ok: false, error: 'upload already running' }, 409); ++ if (active >= MAX_ACTIVE) return c.json({ ok: false, error: 'server busy, retry later' }, 503); ++ tk.busy = true; ++ active++; ++ const path = join(cfg.intakeDir, ticket + '.mp4'); ++ let keep = false; ++ try { ++ const h = createHash('sha256'); ++ let got = 0; ++ const out = createWriteStream(path); ++ const reader = c.req.raw.body ? c.req.raw.body.getReader() : null; ++ if (!reader) throw Object.assign(new Error('empty body'), { status: 400 }); ++ try { ++ for (;;) { ++ const { done, value } = await reader.read(); ++ if (done) break; ++ got += value.byteLength; ++ if (got > tk.size) throw Object.assign(new Error('more bytes than announced'), { status: 413 }); ++ h.update(value); ++ if (!out.write(value)) await new Promise((r) => out.once('drain', r)); ++ } ++ } finally { ++ await new Promise((r) => out.end(r)); ++ } ++ if (got !== tk.size) throw Object.assign(new Error(`size mismatch (${got} of ${tk.size})`), { status: 400 }); ++ const cid = h.digest('hex'); ++ if (tk.cid && cid !== tk.cid) throw Object.assign(new Error('content hash mismatch'), { status: 400 }); ++ let probe; ++ try { probe = await validateMedia(path, 0, { codecs: ['h264', 'hevc'] }); } ++ catch (e) { throw Object.assign(new Error(e.message), { status: 422 }); } ++ const adm = await admitFile( ++ { path, cid, videoId: tk.videoId, size: got, height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta: tk.meta, origin: 'intake' }, ++ { cfg, upsertContent: p2pDb.upsertContent, log }, ++ ); ++ if (!adm.ok) throw Object.assign(new Error('not admitted: ' + adm.reason), { status: 422 }); ++ await p2pDb.upsertHolder({ cid, deviceId: d.device_id, trust: 'challenged', now: now() }); ++ let adopted = false; ++ if (adopt) { ++ try { adopted = !!(await adopt(tk.videoId, path, { sha256: cid, probe, meta: tk.meta })).adopted; keep = adopted; } ++ catch (e) { log.warn?.(`[p2p] adopt ${tk.videoId} failed: ${e.message}`); } ++ } ++ log.info?.(`[p2p] intake ${tk.videoId} ${cid.slice(0, 12)} admitted${adopted ? ' + adopted' : ''}`); ++ return c.json({ ok: true, cid, adopted }); ++ } catch (e) { ++ return c.json({ ok: false, error: e.message }, e.status || 500); ++ } finally { ++ tickets.delete(ticket); ++ active--; ++ if (!keep) { try { unlinkSync(path); } catch { /* never written */ } } ++ } ++ }); ++ ++ return { openTickets: () => tickets.size, active: () => active }; ++} +--- /dev/null ++++ b/server/p2p-intake.test.js +@@ -0,0 +1,103 @@ ++// Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg. ++import { test, expect, beforeAll } from 'bun:test'; ++import { mkdtempSync, readFileSync, readdirSync } from 'node:fs'; ++import { spawnSync } from 'node:child_process'; ++import { tmpdir } from 'node:os'; ++import { join } from 'node:path'; ++import { createHash } from 'node:crypto'; ++import { Hono } from 'hono'; ++ ++const root = mkdtempSync(join(tmpdir(), 'ytp-intake-')); ++process.env.DB_PATH = join(root, 'test.db'); ++const dbmod = await import('./db.js'); ++const p2pDb = await import('./p2p-db.js'); ++const { registerP2pRoutes } = await import('./p2p-routes.js'); ++const { registerIntakeRoutes } = await import('./p2p-intake.js'); ++const { admitFile } = await import('./p2p-admit.js'); ++const { validateMedia } = await import('./media-cache.js'); ++ ++const quiet = { warn() {}, info() {} }; ++const intakeDir = join(root, 'intake'); ++const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir, intakeMaxBytes: 50 * 1024 * 1024 }; ++let app; ++let dev; ++let good; ++let goodCid; ++const adopted = []; ++ ++const req = (path, { method = 'GET', body, raw } = {}) => app.request(path, { ++ method, ++ headers: { ...(raw ? {} : { 'Content-Type': 'application/json' }), 'X-Device': dev.deviceId + '.' + dev.secret }, ++ body: raw || (body ? JSON.stringify(body) : undefined), ++}); ++ ++beforeAll(async () => { ++ await dbmod.initDb(); ++ await p2pDb.initP2pSchema(); ++ const f = join(root, 'good.mp4'); ++ const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4', ++ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', f]); ++ if (r.status !== 0) throw new Error('ffmpeg fixture failed: ' + r.stderr); ++ good = readFileSync(f); ++ goodCid = createHash('sha256').update(good).digest('hex'); ++ app = new Hono(); ++ const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet }); ++ registerIntakeRoutes(app, { ++ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet, ++ adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256, meta: info.meta }); return { adopted: false }; }, ++ }); ++ dev = await (await app.request('/api/p2p/device', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' })).json(); ++}); ++ ++test('a valid file is hashed by the server, validated, admitted and its uploader becomes a holder', async () => { ++ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, title: 'Song x', channel: 'Choir' } })).json(); ++ expect(t.ok).toBe(true); ++ const r = await req(t.url, { method: 'PUT', raw: good }); ++ expect(r.status).toBe(200); ++ expect(await r.json()).toEqual({ ok: true, cid: goodCid, adopted: false }); ++ expect((await p2pDb.getContent(goodCid))).toMatchObject({ origin: 'intake', status: 'verified', scan: 'skipped', video_id: 'upAAAAAAAA1' }); ++ expect((await p2pDb.listHolders(goodCid))[0]).toMatchObject({ device_id: dev.deviceId, trust: 'challenged' }); ++ expect(adopted).toEqual([{ videoId: 'upAAAAAAAA1', sha256: goodCid, meta: { title: 'Song b x /b', channel: 'Choir' } }]); ++ expect(JSON.parse((await p2pDb.getContent(goodCid)).meta).title).toBe('Song b x /b'); ++ expect(readdirSync(intakeDir)).toEqual([]); // not adopted → deleted ++ // Already verified → nothing to upload next time. ++ expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true }); ++}); ++ ++test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => { ++ const cases = [ ++ [{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/], ++ [{ bytes: good.subarray(0, Math.floor(good.length * 0.6)) }, 422, /validation/], ++ [{ bytes: Buffer.alloc(200 * 1024, 7) }, 422, /validation/], ++ ]; ++ for (const [c, status, msg] of cases) { ++ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA2', cid: c.cid, size: c.bytes.length } })).json(); ++ const r = await req(t.url, { method: 'PUT', raw: c.bytes }); ++ expect(r.status).toBe(status); ++ expect((await r.json()).error).toMatch(msg); ++ } ++ expect(readdirSync(intakeDir)).toEqual([]); ++ expect((await p2pDb.listContentForVideo('upAAAAAAAA2')).length).toBe(0); ++}); ++ ++test('size limits and tickets are enforced', async () => { ++ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: cfg.intakeMaxBytes + 1 } })).status).toBe(413); ++ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: 10 } })).json(); ++ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA4', size: 10 } })).status).toBe(429); // one open per device ++ const r = await req(t.url, { method: 'PUT', raw: Buffer.alloc(11) }); ++ expect(r.status).toBe(413); ++ expect((await req(t.url, { method: 'PUT', raw: Buffer.alloc(10) })).status).toBe(404); // ticket consumed ++}); ++ ++test('malware scan on: an infected verdict is never admitted', async () => { ++ const cfg2 = { ...cfg, malwareScan: true, scanCmd: 'false' }; // exit 1 = infected ++ const app2 = new Hono(); ++ const p2p = registerP2pRoutes(app2, { cfg: cfg2, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet }); ++ registerIntakeRoutes(app2, { cfg: cfg2, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet }); ++ const h = { 'X-Device': dev.deviceId + '.' + dev.secret }; ++ const t = await (await app2.request('/api/p2p/intake', { method: 'POST', headers: { ...h, 'Content-Type': 'application/json' }, body: JSON.stringify({ videoId: 'upAAAAAAAA5', size: good.length }) })).json(); ++ await p2pDb.revokeContent(goodCid); // make sure it is not simply "already known" ++ const r = await app2.request(t.url, { method: 'PUT', headers: h, body: good }); ++ expect(r.status).toBe(422); ++ expect((await r.json()).error).toMatch(/scan infected/); ++}); diff --git a/plans/patches/017-p2p-transfer-new.diff b/plans/patches/017-p2p-transfer-new.diff new file mode 100644 index 0000000..f5e349f --- /dev/null +++ b/plans/patches/017-p2p-transfer-new.diff @@ -0,0 +1,280 @@ +--- /dev/null ++++ b/frontend/p2p-transfer.js +@@ -0,0 +1,184 @@ ++/* ============================================================================ ++ * p2p-transfer.js — download a verified file from another device over a ++ * WebRTC data channel (docs/p2p-architecture.md flow 6). window.P2PTransfer. ++ * ++ * start({ canShare }) serve requests from other devices (called once) ++ * download({ videoId, cid, size, peers, onProgress }) → { ok, error? } ++ * ++ * Signalling rides P2PClient.signal()/onMessage('signal') over /ws/p2p: ++ * {k:'offer', xid, cid, sdp} · {k:'answer', xid, sdp} · {k:'ice', xid, cand} ++ * {k:'deny', xid, reason} ++ * Data channel "file" (ordered): requester sends {t:'get'}; sender answers ++ * {t:'meta', size}, 64 KiB binary frames (back-pressured on bufferedAmount), ++ * then {t:'end'}. The receiver writes through p2p-recv-worker.js, which only ++ * commits the file when its SHA-256 equals the cid. STUN only — devices ++ * behind strict NATs won't connect (same as watch-party voice); the next ++ * holder is tried. ++ * ========================================================================== */ ++(function () { ++ 'use strict'; ++ ++ const ICE = [{ urls: 'stun:stun.l.google.com:19302' }, { urls: 'stun:stun1.l.google.com:19302' }]; ++ const FRAME = 64 * 1024; ++ const HIGH_WATER = 4 * 1024 * 1024; ++ const OPEN_TIMEOUT = 20_000; ++ const IDLE_TIMEOUT = 30_000; ++ const MAX_SERVING = 1; ++ ++ let iceServers = ICE; ++ let canShare = () => true; ++ const sessions = new Map(); // xid -> { pc, onSignal } ++ let serving = 0; ++ ++ const newXid = () => Math.random().toString(36).slice(2) + Date.now().toString(36); ++ ++ function onSignal(m) { ++ const d = m && m.data; ++ if (!d || !d.xid) return; ++ const s = sessions.get(d.xid); ++ if (s) { s.onSignal(d, m.from); return; } ++ if (d.k === 'offer') serve(d, m.from).catch(() => {}); ++ } ++ ++ // ---- sender ------------------------------------------------------------------- ++ async function serve(offer, from) { ++ const deny = (reason) => window.P2PClient.signal(from, { k: 'deny', xid: offer.xid, reason }); ++ if (!canShare()) return deny('sharing off'); ++ if (serving >= MAX_SERVING) return deny('busy'); ++ const rec = await window.DeviceDB.getByCid(offer.cid); ++ const file = rec && window.OPFS.getFileObject ? await window.OPFS.getFileObject(rec.videoId) : null; ++ if (!rec || !file || file.size !== rec.size) return deny('not here'); ++ serving++; ++ const pc = new RTCPeerConnection({ iceServers }); ++ let done = false; ++ const finish = () => { ++ if (done) return; ++ done = true; ++ serving--; ++ sessions.delete(offer.xid); ++ try { pc.close(); } catch { /* closed */ } ++ }; ++ sessions.set(offer.xid, { ++ pc, ++ onSignal: (d) => { if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {}); }, ++ }); ++ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(from, { k: 'ice', xid: offer.xid, cand: e.candidate.toJSON() }); }; ++ pc.onconnectionstatechange = () => { if (['failed', 'closed', 'disconnected'].includes(pc.connectionState)) finish(); }; ++ setTimeout(() => { if (pc.connectionState !== 'connected') finish(); }, OPEN_TIMEOUT); ++ pc.ondatachannel = (e) => { ++ const dc = e.channel; ++ dc.bufferedAmountLowThreshold = 1024 * 1024; ++ dc.onmessage = async (ev) => { ++ let msg; ++ try { msg = JSON.parse(ev.data); } catch { return; } ++ if (msg.t !== 'get') return; ++ try { ++ dc.send(JSON.stringify({ t: 'meta', size: file.size })); ++ for (let pos = 0; pos < file.size && !done; pos += 4 * FRAME) { ++ const buf = await file.slice(pos, pos + 4 * FRAME).arrayBuffer(); ++ for (let i = 0; i < buf.byteLength; i += FRAME) { ++ if (dc.bufferedAmount > HIGH_WATER) { ++ await new Promise((r) => { dc.onbufferedamountlow = () => { dc.onbufferedamountlow = null; r(); }; }); ++ } ++ dc.send(buf.slice(i, i + FRAME)); ++ } ++ } ++ dc.send(JSON.stringify({ t: 'end' })); ++ } catch { finish(); } ++ }; ++ dc.onclose = finish; ++ }; ++ await pc.setRemoteDescription({ type: 'offer', sdp: offer.sdp }); ++ const answer = await pc.createAnswer(); ++ await pc.setLocalDescription(answer); ++ window.P2PClient.signal(from, { k: 'answer', xid: offer.xid, sdp: answer.sdp }); ++ } ++ ++ // ---- receiver ----------------------------------------------------------------- ++ function tryPeer({ videoId, cid, size, peer, onProgress }) { ++ return new Promise((resolve) => { ++ const xid = newXid(); ++ const pc = new RTCPeerConnection({ iceServers }); ++ const worker = new Worker('/p2p-recv-worker.js'); ++ let settled = false; ++ let idle = null; ++ let expected = size; ++ const end = (res) => { ++ if (settled) return; ++ settled = true; ++ clearTimeout(idle); ++ clearTimeout(openTimer); ++ sessions.delete(xid); ++ try { pc.close(); } catch { /* closed */ } ++ if (!res.ok) worker.postMessage({ op: 'abort' }); ++ setTimeout(() => worker.terminate(), res.ok ? 0 : 2000); ++ resolve(res); ++ }; ++ const bump = () => { clearTimeout(idle); idle = setTimeout(() => end({ ok: false, error: 'peer went quiet' }), IDLE_TIMEOUT); }; ++ const openTimer = setTimeout(() => end({ ok: false, error: 'could not connect' }), OPEN_TIMEOUT); ++ ++ worker.onmessage = (e) => { ++ const m = e.data || {}; ++ if (m.op === 'progress' && onProgress) onProgress(m.received, expected); ++ if (m.op === 'done') end(m.ok ? { ok: true, sha256: m.sha256, size: m.size } : { ok: false, error: m.error }); ++ }; ++ worker.postMessage({ op: 'open', videoId }); ++ ++ sessions.set(xid, { ++ pc, ++ onSignal: (d) => { ++ if (d.k === 'answer') pc.setRemoteDescription({ type: 'answer', sdp: d.sdp }).catch(() => end({ ok: false, error: 'bad answer' })); ++ else if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {}); ++ else if (d.k === 'deny') end({ ok: false, error: 'peer declined: ' + d.reason }); ++ }, ++ }); ++ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(peer, { k: 'ice', xid, cand: e.candidate.toJSON() }); }; ++ const dc = pc.createDataChannel('file', { ordered: true }); ++ dc.binaryType = 'arraybuffer'; ++ dc.onopen = () => { clearTimeout(openTimer); bump(); dc.send(JSON.stringify({ t: 'get' })); }; ++ dc.onmessage = (e) => { ++ bump(); ++ if (typeof e.data === 'string') { ++ let m; ++ try { m = JSON.parse(e.data); } catch { return; } ++ if (m.t === 'meta') { ++ if (m.size !== size) end({ ok: false, error: 'peer has a different file size' }); ++ expected = m.size; ++ } else if (m.t === 'end') { ++ clearTimeout(idle); ++ worker.postMessage({ op: 'finish', cid, size }); ++ } ++ return; ++ } ++ worker.postMessage({ op: 'chunk', buf: e.data }, [e.data]); ++ }; ++ dc.onclose = () => { if (!settled) setTimeout(() => end({ ok: false, error: 'channel closed' }), 5000); }; ++ (async () => { ++ try { ++ const offer = await pc.createOffer(); ++ await pc.setLocalDescription(offer); ++ if (!window.P2PClient.signal(peer, { k: 'offer', xid, cid, sdp: offer.sdp })) end({ ok: false, error: 'not connected to the P2P hub' }); ++ } catch (err) { end({ ok: false, error: err.message }); } ++ })(); ++ }); ++ } ++ ++ // Try each online holder in turn until one delivers a verified file. ++ async function download({ videoId, cid, size, peers, onProgress }) { ++ if (!window.P2PClient || !window.P2PClient.isConnected()) return { ok: false, error: 'not connected to the P2P hub' }; ++ let last = { ok: false, error: 'no online device has this video' }; ++ for (const p of peers || []) { ++ last = await tryPeer({ videoId, cid, size, peer: p.peer, onProgress }); ++ if (last.ok) return last; ++ } ++ return last; ++ } ++ ++ function start(opts = {}) { ++ if (opts.canShare) canShare = opts.canShare; ++ if (opts.iceServers) iceServers = opts.iceServers; ++ window.P2PClient.onMessage('signal', onSignal); ++ } ++ ++ window.P2PTransfer = { start, download }; ++}()); +--- /dev/null ++++ b/frontend/p2p-recv-worker.js +@@ -0,0 +1,90 @@ ++/* ============================================================================ ++ * p2p-recv-worker.js — writes a file arriving from another device into OPFS ++ * while hashing it; commits it ONLY when the SHA-256 equals the expected ++ * content id (docs/p2p-architecture.md flow 6). ++ * ++ * In: { op:'open', videoId } start videos/.p2p.part ++ * { op:'chunk', buf } ArrayBuffer (transferred) ++ * { op:'finish', cid, size } verify + rename to .mp4 ++ * { op:'abort' } drop the partial file ++ * Out: { op:'opened' } | { op:'progress', received } | ++ * { op:'done', ok:true, sha256, size } | { op:'done', ok:false, error } ++ * The ".part" suffix keeps OPFS.listVideos() from ever listing a partial file. ++ * ========================================================================== */ ++'use strict'; ++importScripts('/sha256.js'); ++ ++let dir = null; ++let handle = null; ++let access = null; ++let partName = ''; ++let finalName = ''; ++let hasher = null; ++let offset = 0; ++let lastProgress = 0; ++ ++async function cleanup() { ++ try { if (access) access.close(); } catch { /* closed */ } ++ access = null; ++ try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ } ++} ++ ++// Messages are handled strictly one after another: an async handler would ++// otherwise let 'chunk' or 'abort' run while 'open' is still awaiting OPFS. ++let chain = Promise.resolve(); ++self.onmessage = (e) => { chain = chain.then(() => onOp(e.data || {})); }; ++ ++async function onOp(m) { ++ try { ++ if (m.op === 'open') { ++ const root = await navigator.storage.getDirectory(); ++ dir = await root.getDirectoryHandle('videos', { create: true }); ++ partName = `${m.videoId}.p2p.part`; ++ finalName = `${m.videoId}.mp4`; ++ handle = await dir.getFileHandle(partName, { create: true }); ++ access = await handle.createSyncAccessHandle(); ++ access.truncate(0); ++ hasher = self.Sha256.create(); ++ offset = 0; ++ self.postMessage({ op: 'opened' }); ++ } else if (m.op === 'chunk') { ++ const u8 = new Uint8Array(m.buf); ++ access.write(u8, { at: offset }); ++ hasher.update(u8); ++ offset += u8.byteLength; ++ if (offset - lastProgress > 1024 * 1024) { lastProgress = offset; self.postMessage({ op: 'progress', received: offset }); } ++ } else if (m.op === 'finish') { ++ access.truncate(offset); ++ access.flush(); ++ access.close(); ++ access = null; ++ const sha256 = hasher.hex(); ++ if (offset !== m.size) throw new Error(`size mismatch (${offset} of ${m.size})`); ++ if (sha256 !== m.cid) throw new Error('content hash mismatch'); ++ try { await dir.removeEntry(finalName); } catch { /* no previous copy */ } ++ let renamed = false; ++ if (typeof handle.move === 'function') { try { await handle.move(finalName); renamed = true; } catch { /* copy below */ } } ++ if (!renamed) { ++ const out = await (await dir.getFileHandle(finalName, { create: true })).createSyncAccessHandle(); ++ try { ++ const file = await handle.getFile(); ++ for (let pos = 0; pos < file.size; pos += 8 * 1024 * 1024) { ++ const buf = new Uint8Array(await file.slice(pos, pos + 8 * 1024 * 1024).arrayBuffer()); ++ out.write(buf, { at: pos }); ++ } ++ out.truncate(file.size); ++ out.flush(); ++ } finally { out.close(); } ++ await dir.removeEntry(partName); ++ } ++ partName = ''; ++ self.postMessage({ op: 'done', ok: true, sha256, size: offset }); ++ } else if (m.op === 'abort') { ++ await cleanup(); ++ self.postMessage({ op: 'done', ok: false, error: 'aborted' }); ++ } ++ } catch (err) { ++ await cleanup(); ++ self.postMessage({ op: 'done', ok: false, error: err && err.message ? err.message : String(err) }); ++ } ++} diff --git a/plans/patches/018-client-restore.diff b/plans/patches/018-client-restore.diff new file mode 100644 index 0000000..6713507 --- /dev/null +++ b/plans/patches/018-client-restore.diff @@ -0,0 +1,40 @@ +--- a/frontend/p2p-client.js ++++ b/frontend/p2p-client.js +@@ -160,7 +160,7 @@ + // Send one saved video to the server so it can hash + validate it and add its + // cid to the catalog. Uploads the whole file: only on the user's request + // ("Verify & share") or when the server asks for a copy (plan 018). +- async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) { ++ async function contribute(videoId, { cid = null, title = '', channel = '', restore = false } = {}) { + const dev = await ensureDevice(); + const rec = await window.DeviceDB.getFile(videoId); + const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null; +@@ -168,7 +168,7 @@ + const h = { 'X-Device': dev.deviceId + '.' + dev.secret }; + const t = await (await fetch('/api/p2p/intake', { + method: 'POST', headers: { ...h, 'Content-Type': 'application/json' }, +- body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }), ++ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel, restore }), + })).json().catch(() => ({ ok: false, error: 'server unreachable' })); + if (!t.ok || t.known) { if (t.known) changed(); return t; } + const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file })) +@@ -280,6 +280,19 @@ + timer = setTimeout(sync, 5000); + } + ++ // Plan 018: the server lost its copy and the source is gone — it asks one ++ // holder to send the file back. Only while sharing is on, one at a time, ++ // and only for the exact cid this device holds. ++ let restoring = false; ++ onMessage('upload-request', async (m) => { ++ if (restoring || hooks.getSettings().p2pShare === false) return; ++ const rec = window.DeviceDB ? await window.DeviceDB.getFile(m.videoId) : null; ++ if (!rec || rec.cid !== m.cid) return; ++ restoring = true; ++ try { await contribute(m.videoId, { cid: m.cid, restore: true }); } catch { /* next request retries */ } ++ finally { restoring = false; } ++ }); ++ + function start(h) { + hooks = { ...hooks, ...(h || {}) }; + const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1)); diff --git a/plans/patches/018-server-rehydrate.diff b/plans/patches/018-server-rehydrate.diff new file mode 100644 index 0000000..ae1b326 --- /dev/null +++ b/plans/patches/018-server-rehydrate.diff @@ -0,0 +1,125 @@ +--- a/server/p2p-hub.js ++++ b/server/p2p-hub.js +@@ -118,3 +118,28 @@ + } + return { ok: true, staleDays, cids: out }; + } ++ ++// Flow 8 (plan 018): the source is gone and the server evicted its copy, so ++// ask ONE online holder that shares to upload it through intake ++// ({type:'upload-request', videoId, cid}). At most one request per video per ++// 10 minutes; returns true when a device was asked (or recently was). ++export function createRehydrator({ p2pDb, hub, hasServerCopy, enabled = () => true, now = () => Date.now() }) { ++ const asked = new Map(); // videoId -> ms ++ return async function rehydrate(videoId) { ++ if (!enabled()) return false; ++ if (await hasServerCopy(videoId)) return false; ++ const last = asked.get(videoId); ++ if (last && now() - last < 10 * 60_000) return true; ++ for (const c of await p2pDb.listContentForVideo(videoId)) { ++ for (const h of await p2pDb.listHolders(c.cid, 50)) { ++ if (Number(h.share) !== 1 || !hub.isOnline(h.device_id)) continue; ++ if (hub.send(h.device_id, { type: 'upload-request', videoId, cid: c.cid })) { ++ asked.set(videoId, now()); ++ if (asked.size > 5000) asked.delete(asked.keys().next().value); ++ return true; ++ } ++ } ++ } ++ return false; ++ }; ++} +--- a/server/p2p-hub.test.js ++++ b/server/p2p-hub.test.js +@@ -85,3 +85,21 @@ + ]); + expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids + }); ++ ++test('rehydrator asks one online sharing holder, once per 10 min, only when the server has no copy', async () => { ++ const { createRehydrator } = await import('./p2p-hub.js'); ++ const sent = []; ++ let t = NOW; ++ let serverHas = false; ++ const hub = { isOnline: (d) => d === 'dev_000000000000000a' || d === 'dev_000000000000000c', send: (d, m) => { sent.push([d, m]); return true; } }; ++ const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => serverHas, now: () => t }); ++ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true); ++ // c is online but has sharing off; a is online and shares. ++ expect(sent).toEqual([['dev_000000000000000a', { type: 'upload-request', videoId: 'dQw4w9WgXcQ', cid: CID }]]); ++ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true); // recently asked — no second message ++ expect(sent.length).toBe(1); ++ t += 11 * 60_000; ++ serverHas = true; ++ expect(await rehydrate('dQw4w9WgXcQ')).toBe(false); // server copy is back ++ expect(await rehydrate('unknownVid1')).toBe(false); // nobody holds it ++}); +--- a/server/p2p-intake.js ++++ b/server/p2p-intake.js +@@ -3,8 +3,10 @@ + * (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did + * not come from the server's own fetch can become verified content. + * +- * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? } ++ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel?, restore? } + * → { ok, known:true } cid already verified — nothing to send ++ * (unless restore:true and the server ++ * lost its copy — plan 018) + * → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min + * PUT /api/p2p/intake/:ticket (device) raw file body + * → { ok, cid, adopted } admitted (+ adopted into the media cache) +@@ -30,6 +32,7 @@ + export function registerIntakeRoutes(app, deps) { + const { + cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null, ++ serverHasCid = async () => true, // (cid) → does the server still hold these bytes? + now = () => Date.now(), log = console, + } = deps; + const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy } +@@ -48,7 +51,8 @@ + if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413); + if (cid) { + const known = await p2pDb.getContent(cid); +- if (known && known.status === 'verified') return c.json({ ok: true, known: true }); ++ const restore = body.restore === true && known && known.status === 'verified' && !(await serverHasCid(cid)); ++ if (known && known.status === 'verified' && !restore) return c.json({ ok: true, known: true }); + if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410); + } + sweep(); +--- a/server/p2p-intake.test.js ++++ b/server/p2p-intake.test.js +@@ -1,6 +1,6 @@ + // Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg. + import { test, expect, beforeAll } from 'bun:test'; +-import { mkdtempSync, readFileSync, readdirSync } from 'node:fs'; ++import { mkdtempSync, readFileSync, readdirSync, unlinkSync } from 'node:fs'; + import { spawnSync } from 'node:child_process'; + import { tmpdir } from 'node:os'; + import { join } from 'node:path'; +@@ -64,6 +64,27 @@ + expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true }); + }); + ++test('restore: a known cid is uploaded again only when the server lost its copy', async () => { ++ let serverHas = true; ++ const app3 = new Hono(); ++ const p2p = registerP2pRoutes(app3, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet }); ++ const got = []; ++ registerIntakeRoutes(app3, { ++ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet, ++ serverHasCid: async () => serverHas, ++ adopt: async (videoId, path, info) => { got.push(info.sha256); unlinkSync(path); return { adopted: true }; }, // a real adopt moves the file ++ }); ++ const h = { 'X-Device': dev.deviceId + '.' + dev.secret, 'Content-Type': 'application/json' }; ++ const open = async () => (await app3.request('/api/p2p/intake', { method: 'POST', headers: h, body: JSON.stringify({ videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, restore: true }) })).json(); ++ expect(await open()).toEqual({ ok: true, known: true }); // server still has it ++ serverHas = false; ++ const t = await open(); ++ expect(t.ticket).toMatch(/^[0-9a-f]{32}$/); ++ const r = await (await app3.request(t.url, { method: 'PUT', headers: { 'X-Device': h['X-Device'] }, body: good })).json(); ++ expect(r).toEqual({ ok: true, cid: goodCid, adopted: true }); ++ expect(got).toEqual([goodCid]); ++}); ++ + test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => { + const cases = [ + [{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/], diff --git a/plans/queue/.gitkeep b/plans/queue/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/plans/queue/001-perf-timing-marks-105acc.md b/plans/queue/001-perf-timing-marks-105acc.md new file mode 100644 index 0000000..4528dd9 --- /dev/null +++ b/plans/queue/001-perf-timing-marks-105acc.md @@ -0,0 +1,123 @@ +--- +id: 001-perf-timing-marks-105acc +title: Add startup, search and play timing marks plus yt-dlp duration logs +created: 2026-09-29 +depends_on: [] +est_files: 2 +--- + +# 001 — Add startup, search and play timing marks plus yt-dlp duration logs + +## Objective + +Every later speed plan needs a before/after number. After this plan: +- The browser records `performance.measure` entries `ytp:boot`, `ytp:search`, + `ytp:tap-to-play`, and `window.__ytpPerf()` returns their latest values in ms. +- The server logs one line per yt-dlp call: `[ytdlp] ms ok|fail`. + +Measured baseline (prod, 2026-09-29): search 4.1–5.0 s, first play `/api/streams` 7.5 s, +second play 1.2 s, `/api/version` 1.25 s. + +## Context the executor must NOT rediscover + +`server/server.js:143-165` — the only place yt-dlp is spawned: + +```js +function runYtdlp(args, { signal } = {}) { + return new Promise((resolve, reject) => { + const child = spawn(YTDLP, args, { stdio: ['ignore', 'pipe', 'pipe'] }); + ... + child.on('error', (e) => reject(new Error('yt-dlp not found: ' + e.message))); + child.on('close', (code) => { + if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code)); + else resolve(out); + }); + }); +} +``` + +`frontend/app.js`: +- `async function boot()` starts at ~line 9749 (`wirePlayerEvents();` is its first line). + The first `render()` call inside boot happens after the `hasShareParam` block. +- `async function runSearchQuery(q, { instant = null } = {})` at ~line 8583; the + success path ends with `RecentSearches.cacheResults(q, results);`. +- `Player.loadVideo(videoObj, …)` at ~line 1650; first statement is + `if (!this._handoff) Transition.cancel();`. +- The master element `playing` listener at ~line 2287: + ```js + el.addEventListener('playing', () => { + if (!masterIs(el)) return; + showSpinner(false); + ``` + +## Steps + +1. `server/server.js` — in `runYtdlp`, directly after the `const child = spawn(...)` line add: + ```js + const t0 = Date.now(); + const kind = String(args.find((a) => /^ytsearch|^https?:/.test(String(a))) || args[0] || '') + .replace(/^ytsearch\d*:.*/, 'search').replace(/^https?:\/\/[^/]+\/watch.*/, 'video').slice(0, 40); + ``` + and replace the `child.on('close', …)` handler body with: + ```js + child.on('close', (code) => { + console.log(`[ytdlp] ${kind} ${Date.now() - t0}ms ${code === 0 ? 'ok' : 'fail'}`); + if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code)); + else resolve(out); + }); + ``` +2. `frontend/app.js` — near the top of the file, directly after the line + `const APP_VERSION = '1.0.0';` (~line 24), add: + ```js + // Timing marks for the speed work (plans/). performance.measure entries are + // visible in DevTools → Performance; __ytpPerf() prints the latest ones. + function perfMark(name) { try { performance.mark(name); } catch { /* old browser */ } } + function perfMeasure(name, start) { + try { performance.measure(name, start); } catch { /* start mark missing */ } + } + window.__ytpPerf = () => { + const out = {}; + try { for (const m of performance.getEntriesByType('measure')) if (m.name.startsWith('ytp:')) out[m.name] = Math.round(m.duration); } catch { /* none */ } + return out; + }; + ``` +3. `frontend/app.js` `boot()` — first line of the function body: `perfMark('ytp:boot-start');`. + Immediately after the FIRST `render();` call inside `boot()` add + `perfMeasure('ytp:boot', 'ytp:boot-start');`. +4. `frontend/app.js` `runSearchQuery` — after `const mySeq = ++searchSeq;` add + `perfMark('ytp:search-start');`. After `RecentSearches.cacheResults(q, results);` add + `perfMeasure('ytp:search', 'ytp:search-start');`. +5. `frontend/app.js` `Player.loadVideo` — first line of the body: `perfMark('ytp:tap');`. +6. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` add: + ```js + if (performance.getEntriesByName('ytp:tap').length) { + perfMeasure('ytp:tap-to-play', 'ytp:tap'); + try { performance.clearMarks('ytp:tap'); } catch { /* ignore */ } + } + ``` + +## Out of scope / do NOT touch + +- No reporting endpoint, no UI. Don't change any behaviour, only add marks/logs. +- Do not touch `runYtdlpResilient` or the fallback-client logic. + +## Verification + +```bash +cd /home/user/ytplayer && node --check frontend/app.js && cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK +cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3 +grep -c "perfMark\|perfMeasure" frontend/app.js +``` + +Expected: no syntax errors, `SERVER_OK`, tests `fail 0`, grep count ≥ 8. + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff` (unified) of all changes. +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything + skipped and why. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/002-compress-and-etag-shell-bd459c.md b/plans/queue/002-compress-and-etag-shell-bd459c.md new file mode 100644 index 0000000..2f59d56 --- /dev/null +++ b/plans/queue/002-compress-and-etag-shell-bd459c.md @@ -0,0 +1,156 @@ +--- +id: 002-compress-and-etag-shell-bd459c +title: Serve the app shell gzip/brotli-compressed with ETags +created: 2026-09-29 +depends_on: [001-perf-timing-marks-105acc] +est_files: 2 +--- + +# 002 — Serve the app shell gzip/brotli-compressed with ETags + +## Objective + +Measured on prod: `app.js` is sent as 426 244 bytes with no `content-encoding` +and no `ETag` (2.8 s download at ~150 KB/s). gzip -9 makes it 117 527 bytes; +the whole shell drops from 618 KB to ~158 KB. After this plan: +- Every static text file under `./public` (`.js .css .html .json .webmanifest .svg`) + is served compressed (`br` preferred, else `gzip`) when the client accepts it, + with `Vary: Accept-Encoding`. +- Every static file carries a strong `ETag` and `If-None-Match` returns `304`. +- `index.html` (build-stamped at request time) and `/sw.js` (BUILD_TAG-injected) are + also compressed + ETagged using the text they actually send. +- `Cache-Control` values stay EXACTLY as they are today (`no-cache` for the shell, + `no-store` for sw.js) — CLAUDE.md "Update-flow architecture" depends on that. + +## Context the executor must NOT rediscover + +`server/server.js:1895-1910` today: + +```js +function indexHtml(c) { + if (_indexSource === null) { + try { _indexSource = readFileSync('./public/index.html', 'utf8'); } + catch { return c.text('index.html not found', 404); } + } + return c.html(_indexSource.replace('__BUILD_TAG__', BUILD_TAG), 200, { 'Cache-Control': 'no-cache' }); +} +app.get('/', indexHtml); +app.get('/index.html', indexHtml); +... +app.use('/*', serveStatic({ root: './public', onFound: (_path, c) => { c.header('Cache-Control', 'no-cache'); } })); +// SPA fallback — return index.html for any unmatched path +app.get('/*', indexHtml); +``` + +`/sw.js` handler at ~line 1860 ends with `return c.text(src, 200, { ...no-store headers })`. + +Bun provides `Bun.gzipSync(buf, { level: 9 })`. Brotli: `import { brotliCompressSync, constants } from 'node:zlib'` +(works in Bun). `createHash` is already imported from `node:crypto` at the top. + +**Never compress** `/api/play`, `/api/media/*`, `/api/download/*` (Range/binary) — +they are not served by `serveStatic`, so a static-only middleware cannot touch them. + +## Steps + +1. `server/server.js` — add near the top imports: + `import { brotliCompressSync, constants as zlibConstants } from 'node:zlib';` +2. `server/server.js` — directly ABOVE `function indexHtml(c) {` add this helper block: + ```js + // Compressed + ETagged static text. The shell is ~620 KB raw / ~160 KB gzip + // and every byte crosses the slow VPS→homelab link, so compress once per + // file content and keep it in memory. ETag = sha256 of the RAW bytes, so a + // `no-cache` revalidation costs a 304 instead of the whole file. + const COMPRESSIBLE = /\.(js|css|html|json|webmanifest|svg|txt)$/i; + const compressedCache = new Map(); // key -> { etag, raw, gz, br, type } + function compressedEntry(key, raw, type) { + let e = compressedCache.get(key); + const etag = '"' + createHash('sha256').update(raw).digest('hex').slice(0, 32) + '"'; + if (e && e.etag === etag) return e; + e = { + etag, raw, type, + gz: Bun.gzipSync(raw, { level: 9 }), + br: brotliCompressSync(raw, { params: { [zlibConstants.BROTLI_PARAM_QUALITY]: 11 } }), + }; + compressedCache.set(key, e); + return e; + } + function sendCompressed(c, e, cacheControl) { + const headers = { 'Content-Type': e.type, 'Cache-Control': cacheControl, ETag: e.etag, Vary: 'Accept-Encoding' }; + const inm = c.req.header('if-none-match') || ''; + if (inm.split(',').map((s) => s.trim()).includes(e.etag)) return new Response(null, { status: 304, headers }); + const ae = c.req.header('accept-encoding') || ''; + if (/\bbr\b/.test(ae)) return new Response(e.br, { headers: { ...headers, 'Content-Encoding': 'br' } }); + if (/\bgzip\b/.test(ae)) return new Response(e.gz, { headers: { ...headers, 'Content-Encoding': 'gzip' } }); + return new Response(e.raw, { headers }); + } + const MIME = { js: 'text/javascript; charset=utf-8', css: 'text/css; charset=utf-8', html: 'text/html; charset=utf-8', + json: 'application/json', webmanifest: 'application/manifest+json', svg: 'image/svg+xml', txt: 'text/plain; charset=utf-8' }; + ``` +3. `server/server.js` — change `indexHtml` so its return line becomes: + ```js + const html = _indexSource.replace('__BUILD_TAG__', BUILD_TAG); + return sendCompressed(c, compressedEntry('index.html', Buffer.from(html), MIME.html), 'no-cache'); + ``` +4. `server/server.js` — in the `/sw.js` handler, replace its final return, which is exactly: + ```js + return c.text(src, 200, { + 'Content-Type': 'application/javascript; charset=utf-8', + 'Cache-Control': 'no-store, no-cache, must-revalidate', + }); + ``` + with: + ```js + return sendCompressed(c, compressedEntry('sw.js', Buffer.from(src), MIME.js), 'no-store, no-cache, must-revalidate'); + ``` + (`text/javascript` is a valid service-worker MIME type.) Dry-run result: app.js 426 244 → 93 717 bytes (br). +5. `server/server.js` — directly BEFORE the `app.use('/*', serveStatic(...))` line add: + ```js + app.get('/*', async (c, next) => { + const p = decodeURIComponent(new URL(c.req.url).pathname); + if (!COMPRESSIBLE.test(p) || p.includes('..') || p.startsWith('/api/')) return next(); + const file = Bun.file('./public' + p); + if (!(await file.exists())) return next(); + const raw = Buffer.from(await file.arrayBuffer()); + const ext = p.slice(p.lastIndexOf('.') + 1).toLowerCase(); + return sendCompressed(c, compressedEntry(p, raw, MIME[ext] || 'application/octet-stream'), 'no-cache'); + }); + ``` + (`/sw.js`, `/` and `/index.html` are registered earlier and win; this only covers the rest.) + +## Out of scope / do NOT touch + +- `frontend/sw.js`, `frontend/sw-update.js`: no changes. The SW fetches shell files with + `cache: 'reload'` and `?__ytpfresh=` — query strings don't affect the pathname match, fine. +- Do not change any `Cache-Control` value. Do not add `hono/compress` globally (it would + hit Range media responses). +- Do not touch `computeBuildTag`. + +## Verification + +```bash +cd /home/user/ytplayer/server && [ -e public ] || ln -s ../frontend public +PORT=3999 bun server.js > /tmp/ytp002.log 2>&1 & SRV=$!; sleep 4 +curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip, br' http://localhost:3999/app.js | grep -iE 'content-encoding|etag|cache-control|vary' +ET=$(curl -s -D - -o /dev/null http://localhost:3999/app.js | grep -i '^etag' | cut -d' ' -f2 | tr -d '\r') +curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" http://localhost:3999/app.js +curl -s -H 'Accept-Encoding: gzip' --compressed http://localhost:3999/ | grep -c 'ytp-build' +curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip' http://localhost:3999/sw.js | grep -iE 'content-encoding|cache-control' +curl -s --compressed http://localhost:3999/sw.js | grep -c "__BUILD_TAG__ !== 'undefined'" ; true +kill $SRV; true +cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3 +``` + +Expected: `content-encoding: br`, an `etag`, `cache-control: no-cache`, `vary: Accept-Encoding`; +the If-None-Match request prints `304`; index grep prints `1`; sw.js shows `content-encoding: gzip` +and its original no-store cache-control; the last grep prints `0` (tag was injected); tests `fail 0`. + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff` (unified) of all changes. +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything + skipped and why. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/003-self-host-fonts-89466b.md b/plans/queue/003-self-host-fonts-89466b.md new file mode 100644 index 0000000..1346fe7 --- /dev/null +++ b/plans/queue/003-self-host-fonts-89466b.md @@ -0,0 +1,131 @@ +--- +id: 003-self-host-fonts-89466b +title: Self-host the three web fonts and drop the render-blocking Google Fonts CSS +created: 2026-09-29 +depends_on: [002-compress-and-etag-shell-bd459c] +est_files: 5 +--- + +# 003 — Self-host the three web fonts + +## Objective + +`frontend/index.html` loads a render-blocking stylesheet from `fonts.googleapis.com` +(3 families, 10 weights), which costs two extra origins (DNS+TLS each) before first +paint and only works offline through the SW's `ytplayer-fonts` runtime cache. After +this plan the fonts are files under `frontend/fonts/`, declared in +`frontend/fonts/fonts.css`, precached with the shell, and the display face is +preloaded. Visual result must be identical. + +## Context the executor must NOT rediscover + +`frontend/index.html:20-26` today: + +```html + + + + +``` + +CSS tokens (`frontend/styles.css:27-29`) reference the family names +`"Bricolage Grotesque"`, `"Hanken Grotesk"`, `"JetBrains Mono"` — keep those names. + +`frontend/sw.js:55-71` — the `SHELL` array (precache list). Every new shell file must be +listed there or it will be missing offline. `frontend/sw.js:165` keeps a runtime rule for +the Google hosts — leave it (harmless, and old clients may still request them). + +Google serves one variable `woff2` per family per subset when asked with a modern UA. +`fitLyricLines` already re-fits on `document.fonts` load (CLAUDE.md), so swap is safe. + +## Steps + +1. Create `scripts/fetch-fonts.js` (Node ≥18, no deps): + ```js + // Downloads the app's Google fonts once (latin + latin-ext subsets) into + // frontend/fonts/ and writes frontend/fonts/fonts.css pointing at them. + // Re-run only when the font list changes. + const fs = require('node:fs'); + const path = require('node:path'); + const CSS_URL = 'https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600..800&family=Hanken+Grotesk:wght@400..700&family=JetBrains+Mono:wght@400..700&display=swap'; + const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'; + const OUT = path.join(__dirname, '..', 'frontend', 'fonts'); + (async () => { + fs.mkdirSync(OUT, { recursive: true }); + const css = await (await fetch(CSS_URL, { headers: { 'User-Agent': UA } })).text(); + // parts[k] ends with the "/* */" comment of the block in parts[k+1]. + const parts = css.split('@font-face'); + let out = '/* Generated by scripts/fetch-fonts.js — do not edit by hand. */\n'; + let n = 0; + for (let k = 1; k < parts.length; k++) { + const subset = (parts[k - 1].match(/\/\*\s*([\w-]+)\s*\*\/\s*$/) || [])[1]; + const b = parts[k].slice(0, parts[k].indexOf('}') + 1); // just this block + if (subset !== 'latin' && subset !== 'latin-ext') continue; + const url = (b.match(/url\((https:[^)]+\.woff2)\)/) || [])[1]; + const fam = (b.match(/font-family:\s*'([^']+)'/) || [])[1]; + if (!url || !fam) continue; + const file = `${fam.replace(/\s+/g, '')}-${subset}.woff2`; + const buf = Buffer.from(await (await fetch(url)).arrayBuffer()); + fs.writeFileSync(path.join(OUT, file), buf); + out += '@font-face' + b.replace(url, file).trimEnd() + '\n'; + n++; + } + fs.writeFileSync(path.join(OUT, 'fonts.css'), out); + console.log(`wrote ${n} font files + fonts.css`); + })().catch((e) => { console.error(e); process.exit(1); }); + ``` + Run it: `node scripts/fetch-fonts.js`. Expect `wrote 6 font files + fonts.css` + (3 families × 2 subsets). If the network is blocked, STOP and report — do not hand-write fonts. +2. Open `frontend/fonts/fonts.css` and confirm every block has `font-display: swap;`; if a + block lacks it, add it. +3. `frontend/index.html` — replace the 5 lines from `` + through the closing `/>` of the Google stylesheet link with: + ```html + + + + ``` + (Use the exact filenames the script produced — check with `ls frontend/fonts`.) +4. `frontend/sw.js` `SHELL` array — after `'/styles.css',` add `'/fonts/fonts.css',` and one + `'/fonts/.woff2',` line per generated woff2 file. +5. `frontend/index.html` line 16 — the Content-Security-Policy `` has + `font-src https://fonts.gstatic.com data:` which would BLOCK self-hosted fonts. Change that part to + `font-src 'self' https://fonts.gstatic.com data:` (leave the rest of the policy unchanged; + `style-src` already allows `'self'`). `frontend/admin.html` has no Google Fonts link — leave it alone. +6. `package.json` scripts — add `"fetch-fonts": "node scripts/fetch-fonts.js"`. + +## Out of scope / do NOT touch + +- Do not rename the font families or edit `styles.css`. +- Do not remove the fonts rule in `sw.js` fetch handler or `UTILITY_CACHES`. +- Do not subset further (e.g. glyph-level subsetting) — out of scope. + +## Verification + +```bash +cd /home/user/ytplayer && ls -la frontend/fonts +grep -c 'href="https://fonts.googleapis' frontend/index.html +grep -c "font-src 'self'" frontend/index.html +for f in $(ls frontend/fonts/*.woff2); do grep -c "/fonts/$(basename $f)" frontend/sw.js; done +node --test frontend/*.test.js 2>&1 | tail -3 +cd server && [ -e public ] || ln -s ../frontend public; PORT=3998 bun server.js >/tmp/ytp003.log 2>&1 & SRV=$!; sleep 4 +curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:3998/fonts/fonts.css +curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3998/fonts/$(ls ../frontend/fonts | grep woff2 | head -1) +kill $SRV; true +``` + +Expected: 6 `.woff2` + `fonts.css`; google link grep `0`; CSP grep `1`; each sw.js grep `1`; tests `fail 0`; +`200 text/css…` and `200`. + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff --stat` plus unified diff of text files (not the woff2 binaries). +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/004-coalesce-stream-resolves-a92d40.md b/plans/queue/004-coalesce-stream-resolves-a92d40.md new file mode 100644 index 0000000..4a11dbb --- /dev/null +++ b/plans/queue/004-coalesce-stream-resolves-a92d40.md @@ -0,0 +1,85 @@ +--- +id: 004-coalesce-stream-resolves-a92d40 +title: Coalesce concurrent resolveStreams calls for the same video +created: 2026-09-29 +depends_on: [001-perf-timing-marks-105acc] +est_files: 1 +--- + +# 004 — Coalesce concurrent resolveStreams calls + +## Objective + +`resolveStreams(videoId)` checks `streamCache` but has no in-flight map, so two +requests for the same id that arrive before the first finishes (a warm-up + the +real play, two devices, the media cache's `getInfo` + `/api/streams`) each spawn a +~6 s `yt-dlp -J`. After this plan, concurrent callers share ONE promise; a failure +is not cached (the next call retries). + +## Context the executor must NOT rediscover + +`server/server.js:457-494`: + +```js +async function resolveStreams(videoId) { + const now = Date.now(); + const cached = streamCache.get(videoId); + if (cached && now < cached.expiresAt) return cached; + + const out = await runYtdlpResilient(['-J', '--no-warnings', `https://www.youtube.com/watch?v=${videoId}`]); + const info = JSON.parse(out); + ... + streamCache.set(videoId, entry); + return entry; +} +``` + +`streamCache` and `STREAM_CACHE_MAX` are declared just above it. + +## Steps + +1. `server/server.js` — rename the existing function `resolveStreams` to + `resolveStreamsUncached` (definition only; body unchanged). +2. Directly after that function add: + ```js + // One yt-dlp -J per video at a time: concurrent callers (warm-up + play, + // two devices, the media cache's getInfo) share the in-flight promise. + const inflightStreams = new Map(); // videoId -> Promise + function resolveStreams(videoId) { + const cached = streamCache.get(videoId); + if (cached && Date.now() < cached.expiresAt) return Promise.resolve(cached); + let p = inflightStreams.get(videoId); + if (!p) { + p = resolveStreamsUncached(videoId).finally(() => inflightStreams.delete(videoId)); + inflightStreams.set(videoId, p); + } + return p; + } + ``` + All existing callers keep calling `resolveStreams` (it still returns a Promise). +3. Create `server/streams-inflight.test.js`? — NO. Keep it in-file; instead verify with the + script below. + +## Out of scope / do NOT touch + +- Cache TTL logic, `runYtdlpResilient`, format filtering. + +## Verification + +```bash +cd /home/user/ytplayer/server && grep -n "function resolveStreams\|function resolveStreamsUncached\|inflightStreams" server.js +bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK +bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)" +``` + +Expected: both functions + map present, `SERVER_OK`, all server test files `0 fail`. + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff` (unified) of all changes. +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/005-warm-streams-on-intent-47b3d3.md b/plans/queue/005-warm-streams-on-intent-47b3d3.md new file mode 100644 index 0000000..6841884 --- /dev/null +++ b/plans/queue/005-warm-streams-on-intent-47b3d3.md @@ -0,0 +1,111 @@ +--- +id: 005-warm-streams-on-intent-47b3d3 +title: Warm the stream cache for likely next plays +created: 2026-09-29 +depends_on: [004-coalesce-stream-resolves-a92d40] +est_files: 2 +--- + +# 005 — Warm the stream cache for likely next plays + +## Objective + +A cold `/api/streams` costs ~6 s of yt-dlp; a cached one ~0 s (1.2 s total on prod, +all network). Start that work before the user taps: +- new `GET /api/streams/warm?v=` → `204` immediately, resolves in the background + (shares the in-flight promise from plan 004), at most 2 warm resolves at a time, + skipped when the server already holds a ready media copy. +- the client warms: the top 3 search results after a search renders; a card on + `pointerdown` (touch/mouse down fires ~100–300 ms before `click`); the next 2 queue + items when a song starts playing. + +## Context the executor must NOT rediscover + +- `server/server.js` `resolveStreams(videoId)` returns a Promise and dedupes (plan 004). +- `media.getReady(videoId)` (server/media-cache.js) resolves the ready row or null. +- `server/server.js:532` — `app.get('/api/streams', async (c) => {` — register the new + route directly ABOVE it (Hono matches `/api/streams/warm` separately anyway). +- Client `frontend/app.js`: + - `const YT_ID_RE = /^[A-Za-z0-9_-]{11}$/;` at ~line 1202 (declared later in the file + than the helper you add — fine, it is only read at call time). + - `runSearchQuery` success path (~line 8603): `searchResults = results; … renderList(); RecentSearches.cacheResults(q, results);` + - `renderCard(v, index, list)` at ~line 7725; it has `card.addEventListener('click', (e) => {` at ~line 7781. + - globals `queue` (array of video objects) and `queueIndex` (~line 353). + - master `playing` listener at ~line 2287 (`el.addEventListener('playing', () => { if (!masterIs(el)) return; …`). + - `WEB` constant is true for the PWA; `cachedIds` is a Set of ids saved on this device. + +## Steps + +1. `server/server.js` — above `app.get('/api/streams', …)` add: + ```js + // GET /api/streams/warm?v= — fire-and-forget: resolve streams into + // streamCache so the real /api/streams a moment later is instant. Bounded + // so a scrolling user can't queue dozens of yt-dlp processes. + const WARM_MAX = 2; + let warmActive = 0; + app.get('/api/streams/warm', async (c) => { + const id = (c.req.query('v') || '').trim(); + if (!/^[A-Za-z0-9_-]{11}$/.test(id)) return c.body(null, 204); + if (warmActive >= WARM_MAX) return c.body(null, 204); + try { if (await media.getReady(id)) return c.body(null, 204); } catch { /* fall through */ } + warmActive++; + resolveStreams(id).catch(() => {}).finally(() => { warmActive--; }); + return c.body(null, 204); + }); + ``` +2. `frontend/app.js` — directly after the `const API = { … };` object (~line 320) add: + ```js + // Pre-resolve streams for videos the user is likely to play next (server + // /api/streams/warm). Each id is warmed at most once per 20 min per tab. + const warmedAt = new Map(); + function warmStreams(ids) { + if (!WEB || navigator.onLine === false) return; + const now = Date.now(); + for (const id of ids) { + if (!id || !/^[A-Za-z0-9_-]{11}$/.test(id) || cachedIds.has(id)) continue; + if (now - (warmedAt.get(id) || 0) < 20 * 60_000) continue; + warmedAt.set(id, now); + fetch(`/api/streams/warm?v=${encodeURIComponent(id)}`, { priority: 'low' }).catch(() => {}); + } + } + ``` +3. `frontend/app.js` `runSearchQuery` — after `RecentSearches.cacheResults(q, results);` add + `warmStreams(results.slice(0, 3).map((r) => r.id));` +4. `frontend/app.js` `renderCard` — directly BEFORE `card.addEventListener('click', (e) => {` add: + ```js + card.addEventListener('pointerdown', () => warmStreams([v.id]), { passive: true }); + ``` +5. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` (and after + the plan-001 perf lines if present) add: + ```js + if (Array.isArray(queue) && queueIndex >= 0) warmStreams(queue.slice(queueIndex + 1, queueIndex + 3).map((x) => x && x.id)); + ``` + +## Out of scope / do NOT touch + +- Do not warm on hover/scroll, do not warm uploads (`upl_…`) — they need no yt-dlp. +- Do not change `/api/streams` itself. + +## Verification + +```bash +cd /home/user/ytplayer && node --check frontend/app.js && echo APP_OK +cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK +[ -e public ] || ln -s ../frontend public; PORT=3997 bun server.js >/tmp/ytp005.log 2>&1 & SRV=$!; sleep 4 +curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=bad' +curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=dQw4w9WgXcQ' +kill $SRV; true +cd .. && node --test frontend/*.test.js 2>&1 | tail -3 +``` + +Expected: `APP_OK`, `SERVER_OK`, `204`, `204`, tests `fail 0`. + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff` (unified) of all changes. +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/006-innertube-search-48066b.md b/plans/queue/006-innertube-search-48066b.md new file mode 100644 index 0000000..d2c5e38 --- /dev/null +++ b/plans/queue/006-innertube-search-48066b.md @@ -0,0 +1,212 @@ +--- +id: 006-innertube-search-48066b +title: Answer searches from YouTube InnerTube directly with yt-dlp fallback +created: 2026-09-29 +depends_on: [001-perf-timing-marks-105acc] +est_files: 4 +--- + +# 006 — InnerTube search with yt-dlp fallback + +## Objective + +`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process +(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search +API returns the same data in a few hundred ms. After this plan `/api/search` tries +InnerTube first (6 s timeout), maps results to the existing card shape, and falls +back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos. +Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri +bridge contract are untouched. + +## Context the executor must NOT rediscover + +A trimmed real response is committed at `server/fixtures/innertube-search.json` +(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`, +and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29): + +``` +contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[] + .itemSectionRenderer.contents[].videoRenderer: + videoId -> id + title.runs[0].text -> title + ownerText.runs[0].text -> channel + ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId -> channelId (UC…) + ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl -> '/@handle' or '/channel/UC…' + lengthText.simpleText "5:43" | "1:59:47" | absent (live) -> duration seconds (0 if absent) +``` + +Request that works (no key needed): +``` +POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false +Content-Type: application/json +{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":""} +``` +The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable. +Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first +cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the +fallback is for. Do not "fix" the 403 by adding cookies/keys. + +Existing card shape — `server/server.js:293-305` `slimEntry`: +```js +return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` }; +``` +`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`. + +Current route `server/server.js:365-395` (inside it): +```js + let mine = []; + try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ } + try { + const out = await runYtdlpResilient([ + `ytsearch${SEARCH_LIMIT}:${q}`, + '--dump-json', '--flat-playlist', + '--no-warnings', '--ignore-errors', + ]); + const results = [...mine, ...parseCards(out)]; +``` + +Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script +runs each file separately joined by `&&`. + +## Steps + +1. Create `server/innertube.js`: + ```js + /* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn). + * parseSearch() is pure (tested against fixtures/innertube-search.json); + * search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */ + const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' }; + + export function lengthToSeconds(s) { + if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0; + return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0); + } + + export function parseSearch(json) { + const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents + ?.sectionListRenderer?.contents; + if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape'); + const out = []; + for (const s of sections) { + for (const it of s?.itemSectionRenderer?.contents || []) { + const v = it && it.videoRenderer; + if (!v || typeof v.videoId !== 'string') continue; + const owner = v.ownerText?.runs?.[0] || {}; + const be = owner.navigationEndpoint?.browseEndpoint || {}; + const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : ''); + out.push({ + id: v.videoId, + title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)', + channel: owner.text || '', + channelId: be.browseId || '', + channelUrl: path ? `https://www.youtube.com${path}` : '', + duration: lengthToSeconds(v.lengthText?.simpleText), + thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`, + }); + } + } + return out; + } + + export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) { + const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ context: { client: CLIENT }, query: q }), + signal: AbortSignal.timeout(timeoutMs), + }); + if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`); + return parseSearch(await res.json()); + } + ``` +2. Create `server/innertube.test.js`: + ```js + import { test, expect } from 'bun:test'; + import { readFileSync } from 'node:fs'; + import { parseSearch, lengthToSeconds, search } from './innertube.js'; + const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url))); + + test('parses video renderers into slim cards', () => { + const r = parseSearch(fixture); + expect(r.length).toBe(4); + expect(r[0]).toEqual({ + id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship', + channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg', + duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg', + }); + expect(r[1].duration).toBe(7187); + expect(r[3].duration).toBe(0); // live, no lengthText + }); + test('lengthToSeconds', () => { + expect(lengthToSeconds('5:43')).toBe(343); + expect(lengthToSeconds('1:59:47')).toBe(7187); + expect(lengthToSeconds('LIVE')).toBe(0); + expect(lengthToSeconds(undefined)).toBe(0); + }); + test('unexpected shape throws (caller falls back to yt-dlp)', () => { + expect(() => parseSearch({})).toThrow(); + }); + test('search() throws on HTTP error', async () => { + const fetchImpl = async () => new Response('no', { status: 429 }); + await expect(search('x', { fetchImpl })).rejects.toThrow('429'); + }); + ``` + If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's + `lengthText` and use that value (the fixture is the source of truth). +3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`. +4. `server/server.js` — add import next to the other local imports: + `import * as innertube from './innertube.js';` +5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];` + head with: + ```js + try { + let yt = []; + try { + yt = await innertube.search(q); + } catch (e) { + console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`); + } + if (!yt.length) { + const out = await runYtdlpResilient([ + `ytsearch${SEARCH_LIMIT}:${q}`, + '--dump-json', '--flat-playlist', + '--no-warnings', '--ignore-errors', + ]); + yt = parseCards(out); + } + const results = [...mine, ...yt]; + ``` + Everything after (`searchCache.set`, return, catch) stays as is. +6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')` + skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented + line `# SEARCH_INNERTUBE: "0" # force yt-dlp search` next to the other commented env vars. + +## Out of scope / do NOT touch + +- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging. +- Card shape, `searchCache`, `app.js`. + +## Verification + +```bash +cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4 +bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)" +[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4 +time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo +kill $SRV; true +grep -c "innertube failed" /tmp/ytp006.log +``` + +Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns +`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container +has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure). + +## Report format (executor: follow exactly) + +Output ONLY the following, no other prose: + +1. `git diff` (unified) of all changes. +2. Raw output of the Verification commands. +3. `Findings:` — max 10 lines. + +Do not commit. Do not push. Do not touch files outside the Steps. diff --git a/plans/queue/007-ytdlp-worker-045800.md b/plans/queue/007-ytdlp-worker-045800.md new file mode 100644 index 0000000..2f97276 --- /dev/null +++ b/plans/queue/007-ytdlp-worker-045800.md @@ -0,0 +1,338 @@ +--- +id: 007-ytdlp-worker-045800 +title: Keep one long-lived yt-dlp worker process instead of spawning per call +created: 2026-09-29 +depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b] +est_files: 5 +--- + +# 007 — Long-lived yt-dlp worker pool + +## Objective + +Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with +yt-dlp 2026.08.19 (same host, same network): + +| call | per-call spawn | warm worker | +|------|----------------|-------------| +| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s | +| `-J