josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Queue executable speed and peer-to-peer sharing plans with tested patches and harnesses

Browse Source
This commit is contained in:
Claude
2026-09-29 19:03:12 +00:00
parent a0ffc1b493
commit c17bd4c9ac
54 changed files with 6369 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
plans/queue/.gitkeep Normal file
View File

123
plans/queue/001-perf-timing-marks-105acc.md Normal file
View File

@@ -0,0 +1,123 @@
---
id: 001-perf-timing-marks-105acc
title: Add startup, search and play timing marks plus yt-dlp duration logs
created: 2026-09-29
depends_on: []
est_files: 2
---
# 001 — Add startup, search and play timing marks plus yt-dlp duration logs
## Objective
Every later speed plan needs a before/after number. After this plan:
- The browser records `performance.measure` entries `ytp:boot`, `ytp:search`,
`ytp:tap-to-play`, and `window.__ytpPerf()` returns their latest values in ms.
- The server logs one line per yt-dlp call: `[ytdlp] <kind> <ms>ms ok|fail`.
Measured baseline (prod, 2026-09-29): search 4.1–5.0 s, first play `/api/streams` 7.5 s,
second play 1.2 s, `/api/version` 1.25 s.
## Context the executor must NOT rediscover
`server/server.js:143-165` — the only place yt-dlp is spawned:
```js
function runYtdlp(args, { signal } = {}) {
return new Promise((resolve, reject) => {
const child = spawn(YTDLP, args, { stdio: ['ignore', 'pipe', 'pipe'] });
...
child.on('error', (e) => reject(new Error('yt-dlp not found: ' + e.message)));
child.on('close', (code) => {
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
});
}
```
`frontend/app.js`:
- `async function boot()` starts at ~line 9749 (`wirePlayerEvents();` is its first line).
The first `render()` call inside boot happens after the `hasShareParam` block.
- `async function runSearchQuery(q, { instant = null } = {})` at ~line 8583; the
success path ends with `RecentSearches.cacheResults(q, results);`.
- `Player.loadVideo(videoObj, …)` at ~line 1650; first statement is
`if (!this._handoff) Transition.cancel();`.
- The master element `playing` listener at ~line 2287:
```js
el.addEventListener('playing', () => {
if (!masterIs(el)) return;
showSpinner(false);
```
## Steps
1. `server/server.js` — in `runYtdlp`, directly after the `const child = spawn(...)` line add:
```js
const t0 = Date.now();
const kind = String(args.find((a) => /^ytsearch|^https?:/.test(String(a))) || args[0] || '')
.replace(/^ytsearch\d*:.*/, 'search').replace(/^https?:\/\/[^/]+\/watch.*/, 'video').slice(0, 40);
```
and replace the `child.on('close', …)` handler body with:
```js
child.on('close', (code) => {
console.log(`[ytdlp] ${kind} ${Date.now() - t0}ms ${code === 0 ? 'ok' : 'fail'}`);
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
```
2. `frontend/app.js` — near the top of the file, directly after the line
`const APP_VERSION = '1.0.0';` (~line 24), add:
```js
// Timing marks for the speed work (plans/). performance.measure entries are
// visible in DevTools → Performance; __ytpPerf() prints the latest ones.
function perfMark(name) { try { performance.mark(name); } catch { /* old browser */ } }
function perfMeasure(name, start) {
try { performance.measure(name, start); } catch { /* start mark missing */ }
}
window.__ytpPerf = () => {
const out = {};
try { for (const m of performance.getEntriesByType('measure')) if (m.name.startsWith('ytp:')) out[m.name] = Math.round(m.duration); } catch { /* none */ }
return out;
};
```
3. `frontend/app.js` `boot()` — first line of the function body: `perfMark('ytp:boot-start');`.
Immediately after the FIRST `render();` call inside `boot()` add
`perfMeasure('ytp:boot', 'ytp:boot-start');`.
4. `frontend/app.js` `runSearchQuery` — after `const mySeq = ++searchSeq;` add
`perfMark('ytp:search-start');`. After `RecentSearches.cacheResults(q, results);` add
`perfMeasure('ytp:search', 'ytp:search-start');`.
5. `frontend/app.js` `Player.loadVideo` — first line of the body: `perfMark('ytp:tap');`.
6. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` add:
```js
if (performance.getEntriesByName('ytp:tap').length) {
perfMeasure('ytp:tap-to-play', 'ytp:tap');
try { performance.clearMarks('ytp:tap'); } catch { /* ignore */ }
}
```
## Out of scope / do NOT touch
- No reporting endpoint, no UI. Don't change any behaviour, only add marks/logs.
- Do not touch `runYtdlpResilient` or the fallback-client logic.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
grep -c "perfMark\|perfMeasure" frontend/app.js
```
Expected: no syntax errors, `SERVER_OK`, tests `fail 0`, grep count ≥ 8.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

156
plans/queue/002-compress-and-etag-shell-bd459c.md Normal file
View File

@@ -0,0 +1,156 @@
---
id: 002-compress-and-etag-shell-bd459c
title: Serve the app shell gzip/brotli-compressed with ETags
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 2
---
# 002 — Serve the app shell gzip/brotli-compressed with ETags
## Objective
Measured on prod: `app.js` is sent as 426 244 bytes with no `content-encoding`
and no `ETag` (2.8 s download at ~150 KB/s). gzip -9 makes it 117 527 bytes;
the whole shell drops from 618 KB to ~158 KB. After this plan:
- Every static text file under `./public` (`.js .css .html .json .webmanifest .svg`)
is served compressed (`br` preferred, else `gzip`) when the client accepts it,
with `Vary: Accept-Encoding`.
- Every static file carries a strong `ETag` and `If-None-Match` returns `304`.
- `index.html` (build-stamped at request time) and `/sw.js` (BUILD_TAG-injected) are
also compressed + ETagged using the text they actually send.
- `Cache-Control` values stay EXACTLY as they are today (`no-cache` for the shell,
`no-store` for sw.js) — CLAUDE.md "Update-flow architecture" depends on that.
## Context the executor must NOT rediscover
`server/server.js:1895-1910` today:
```js
function indexHtml(c) {
if (_indexSource === null) {
try { _indexSource = readFileSync('./public/index.html', 'utf8'); }
catch { return c.text('index.html not found', 404); }
}
return c.html(_indexSource.replace('__BUILD_TAG__', BUILD_TAG), 200, { 'Cache-Control': 'no-cache' });
}
app.get('/', indexHtml);
app.get('/index.html', indexHtml);
...
app.use('/*', serveStatic({ root: './public', onFound: (_path, c) => { c.header('Cache-Control', 'no-cache'); } }));
// SPA fallback — return index.html for any unmatched path
app.get('/*', indexHtml);
```
`/sw.js` handler at ~line 1860 ends with `return c.text(src, 200, { ...no-store headers })`.
Bun provides `Bun.gzipSync(buf, { level: 9 })`. Brotli: `import { brotliCompressSync, constants } from 'node:zlib'`
(works in Bun). `createHash` is already imported from `node:crypto` at the top.
**Never compress** `/api/play`, `/api/media/*`, `/api/download/*` (Range/binary) —
they are not served by `serveStatic`, so a static-only middleware cannot touch them.
## Steps
1. `server/server.js` — add near the top imports:
`import { brotliCompressSync, constants as zlibConstants } from 'node:zlib';`
2. `server/server.js` — directly ABOVE `function indexHtml(c) {` add this helper block:
```js
// Compressed + ETagged static text. The shell is ~620 KB raw / ~160 KB gzip
// and every byte crosses the slow VPS→homelab link, so compress once per
// file content and keep it in memory. ETag = sha256 of the RAW bytes, so a
// `no-cache` revalidation costs a 304 instead of the whole file.
const COMPRESSIBLE = /\.(js|css|html|json|webmanifest|svg|txt)$/i;
const compressedCache = new Map(); // key -> { etag, raw, gz, br, type }
function compressedEntry(key, raw, type) {
let e = compressedCache.get(key);
const etag = '"' + createHash('sha256').update(raw).digest('hex').slice(0, 32) + '"';
if (e && e.etag === etag) return e;
e = {
etag, raw, type,
gz: Bun.gzipSync(raw, { level: 9 }),
br: brotliCompressSync(raw, { params: { [zlibConstants.BROTLI_PARAM_QUALITY]: 11 } }),
};
compressedCache.set(key, e);
return e;
}
function sendCompressed(c, e, cacheControl) {
const headers = { 'Content-Type': e.type, 'Cache-Control': cacheControl, ETag: e.etag, Vary: 'Accept-Encoding' };
const inm = c.req.header('if-none-match') || '';
if (inm.split(',').map((s) => s.trim()).includes(e.etag)) return new Response(null, { status: 304, headers });
const ae = c.req.header('accept-encoding') || '';
if (/\bbr\b/.test(ae)) return new Response(e.br, { headers: { ...headers, 'Content-Encoding': 'br' } });
if (/\bgzip\b/.test(ae)) return new Response(e.gz, { headers: { ...headers, 'Content-Encoding': 'gzip' } });
return new Response(e.raw, { headers });
}
const MIME = { js: 'text/javascript; charset=utf-8', css: 'text/css; charset=utf-8', html: 'text/html; charset=utf-8',
json: 'application/json', webmanifest: 'application/manifest+json', svg: 'image/svg+xml', txt: 'text/plain; charset=utf-8' };
```
3. `server/server.js` — change `indexHtml` so its return line becomes:
```js
const html = _indexSource.replace('__BUILD_TAG__', BUILD_TAG);
return sendCompressed(c, compressedEntry('index.html', Buffer.from(html), MIME.html), 'no-cache');
```
4. `server/server.js` — in the `/sw.js` handler, replace its final return, which is exactly:
```js
return c.text(src, 200, {
'Content-Type': 'application/javascript; charset=utf-8',
'Cache-Control': 'no-store, no-cache, must-revalidate',
});
```
with:
```js
return sendCompressed(c, compressedEntry('sw.js', Buffer.from(src), MIME.js), 'no-store, no-cache, must-revalidate');
```
(`text/javascript` is a valid service-worker MIME type.) Dry-run result: app.js 426 244 → 93 717 bytes (br).
5. `server/server.js` — directly BEFORE the `app.use('/*', serveStatic(...))` line add:
```js
app.get('/*', async (c, next) => {
const p = decodeURIComponent(new URL(c.req.url).pathname);
if (!COMPRESSIBLE.test(p) || p.includes('..') || p.startsWith('/api/')) return next();
const file = Bun.file('./public' + p);
if (!(await file.exists())) return next();
const raw = Buffer.from(await file.arrayBuffer());
const ext = p.slice(p.lastIndexOf('.') + 1).toLowerCase();
return sendCompressed(c, compressedEntry(p, raw, MIME[ext] || 'application/octet-stream'), 'no-cache');
});
```
(`/sw.js`, `/` and `/index.html` are registered earlier and win; this only covers the rest.)
## Out of scope / do NOT touch
- `frontend/sw.js`, `frontend/sw-update.js`: no changes. The SW fetches shell files with
`cache: 'reload'` and `?__ytpfresh=` — query strings don't affect the pathname match, fine.
- Do not change any `Cache-Control` value. Do not add `hono/compress` globally (it would
hit Range media responses).
- Do not touch `computeBuildTag`.
## Verification
```bash
cd /home/user/ytplayer/server && [ -e public ] || ln -s ../frontend public
PORT=3999 bun server.js > /tmp/ytp002.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip, br' http://localhost:3999/app.js | grep -iE 'content-encoding|etag|cache-control|vary'
ET=$(curl -s -D - -o /dev/null http://localhost:3999/app.js | grep -i '^etag' | cut -d' ' -f2 | tr -d '\r')
curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" http://localhost:3999/app.js
curl -s -H 'Accept-Encoding: gzip' --compressed http://localhost:3999/ | grep -c 'ytp-build'
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip' http://localhost:3999/sw.js | grep -iE 'content-encoding|cache-control'
curl -s --compressed http://localhost:3999/sw.js | grep -c "__BUILD_TAG__ !== 'undefined'" ; true
kill $SRV; true
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `content-encoding: br`, an `etag`, `cache-control: no-cache`, `vary: Accept-Encoding`;
the If-None-Match request prints `304`; index grep prints `1`; sw.js shows `content-encoding: gzip`
and its original no-store cache-control; the last grep prints `0` (tag was injected); tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

131
plans/queue/003-self-host-fonts-89466b.md Normal file
View File

@@ -0,0 +1,131 @@
---
id: 003-self-host-fonts-89466b
title: Self-host the three web fonts and drop the render-blocking Google Fonts CSS
created: 2026-09-29
depends_on: [002-compress-and-etag-shell-bd459c]
est_files: 5
---
# 003 — Self-host the three web fonts
## Objective
`frontend/index.html` loads a render-blocking stylesheet from `fonts.googleapis.com`
(3 families, 10 weights), which costs two extra origins (DNS+TLS each) before first
paint and only works offline through the SW's `ytplayer-fonts` runtime cache. After
this plan the fonts are files under `frontend/fonts/`, declared in
`frontend/fonts/fonts.css`, precached with the shell, and the display face is
preloaded. Visual result must be identical.
## Context the executor must NOT rediscover
`frontend/index.html:20-26` today:
```html
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600;12..96,700;12..96,800&family=Hanken+Grotesk:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="styles.css" />
```
CSS tokens (`frontend/styles.css:27-29`) reference the family names
`"Bricolage Grotesque"`, `"Hanken Grotesk"`, `"JetBrains Mono"` — keep those names.
`frontend/sw.js:55-71` — the `SHELL` array (precache list). Every new shell file must be
listed there or it will be missing offline. `frontend/sw.js:165` keeps a runtime rule for
the Google hosts — leave it (harmless, and old clients may still request them).
Google serves one variable `woff2` per family per subset when asked with a modern UA.
`fitLyricLines` already re-fits on `document.fonts` load (CLAUDE.md), so swap is safe.
## Steps
1. Create `scripts/fetch-fonts.js` (Node ≥18, no deps):
```js
// Downloads the app's Google fonts once (latin + latin-ext subsets) into
// frontend/fonts/ and writes frontend/fonts/fonts.css pointing at them.
// Re-run only when the font list changes.
const fs = require('node:fs');
const path = require('node:path');
const CSS_URL = 'https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600..800&family=Hanken+Grotesk:wght@400..700&family=JetBrains+Mono:wght@400..700&display=swap';
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36';
const OUT = path.join(__dirname, '..', 'frontend', 'fonts');
(async () => {
fs.mkdirSync(OUT, { recursive: true });
const css = await (await fetch(CSS_URL, { headers: { 'User-Agent': UA } })).text();
// parts[k] ends with the "/* <subset> */" comment of the block in parts[k+1].
const parts = css.split('@font-face');
let out = '/* Generated by scripts/fetch-fonts.js — do not edit by hand. */\n';
let n = 0;
for (let k = 1; k < parts.length; k++) {
const subset = (parts[k - 1].match(/\/\*\s*([\w-]+)\s*\*\/\s*$/) || [])[1];
const b = parts[k].slice(0, parts[k].indexOf('}') + 1); // just this block
if (subset !== 'latin' && subset !== 'latin-ext') continue;
const url = (b.match(/url\((https:[^)]+\.woff2)\)/) || [])[1];
const fam = (b.match(/font-family:\s*'([^']+)'/) || [])[1];
if (!url || !fam) continue;
const file = `${fam.replace(/\s+/g, '')}-${subset}.woff2`;
const buf = Buffer.from(await (await fetch(url)).arrayBuffer());
fs.writeFileSync(path.join(OUT, file), buf);
out += '@font-face' + b.replace(url, file).trimEnd() + '\n';
n++;
}
fs.writeFileSync(path.join(OUT, 'fonts.css'), out);
console.log(`wrote ${n} font files + fonts.css`);
})().catch((e) => { console.error(e); process.exit(1); });
```
Run it: `node scripts/fetch-fonts.js`. Expect `wrote 6 font files + fonts.css`
(3 families × 2 subsets). If the network is blocked, STOP and report — do not hand-write fonts.
2. Open `frontend/fonts/fonts.css` and confirm every block has `font-display: swap;`; if a
block lacks it, add it.
3. `frontend/index.html` — replace the 5 lines from `<link rel="preconnect" href="https://fonts.googleapis.com" />`
through the closing `/>` of the Google stylesheet link with:
```html
<link rel="preload" href="fonts/BricolageGrotesque-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="preload" href="fonts/HankenGrotesk-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="stylesheet" href="fonts/fonts.css" />
```
(Use the exact filenames the script produced — check with `ls frontend/fonts`.)
4. `frontend/sw.js` `SHELL` array — after `'/styles.css',` add `'/fonts/fonts.css',` and one
`'/fonts/<file>.woff2',` line per generated woff2 file.
5. `frontend/index.html` line 16 — the Content-Security-Policy `<meta>` has
`font-src https://fonts.gstatic.com data:` which would BLOCK self-hosted fonts. Change that part to
`font-src 'self' https://fonts.gstatic.com data:` (leave the rest of the policy unchanged;
`style-src` already allows `'self'`). `frontend/admin.html` has no Google Fonts link — leave it alone.
6. `package.json` scripts — add `"fetch-fonts": "node scripts/fetch-fonts.js"`.
## Out of scope / do NOT touch
- Do not rename the font families or edit `styles.css`.
- Do not remove the fonts rule in `sw.js` fetch handler or `UTILITY_CACHES`.
- Do not subset further (e.g. glyph-level subsetting) — out of scope.
## Verification
```bash
cd /home/user/ytplayer && ls -la frontend/fonts
grep -c 'href="https://fonts.googleapis' frontend/index.html
grep -c "font-src 'self'" frontend/index.html
for f in $(ls frontend/fonts/*.woff2); do grep -c "/fonts/$(basename $f)" frontend/sw.js; done
node --test frontend/*.test.js 2>&1 | tail -3
cd server && [ -e public ] || ln -s ../frontend public; PORT=3998 bun server.js >/tmp/ytp003.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:3998/fonts/fonts.css
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3998/fonts/$(ls ../frontend/fonts | grep woff2 | head -1)
kill $SRV; true
```
Expected: 6 `.woff2` + `fonts.css`; google link grep `0`; CSP grep `1`; each sw.js grep `1`; tests `fail 0`;
`200 text/css…` and `200`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` plus unified diff of text files (not the woff2 binaries).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

85
plans/queue/004-coalesce-stream-resolves-a92d40.md Normal file
View File

@@ -0,0 +1,85 @@
---
id: 004-coalesce-stream-resolves-a92d40
title: Coalesce concurrent resolveStreams calls for the same video
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 1
---
# 004 — Coalesce concurrent resolveStreams calls
## Objective
`resolveStreams(videoId)` checks `streamCache` but has no in-flight map, so two
requests for the same id that arrive before the first finishes (a warm-up + the
real play, two devices, the media cache's `getInfo` + `/api/streams`) each spawn a
~6 s `yt-dlp -J`. After this plan, concurrent callers share ONE promise; a failure
is not cached (the next call retries).
## Context the executor must NOT rediscover
`server/server.js:457-494`:
```js
async function resolveStreams(videoId) {
const now = Date.now();
const cached = streamCache.get(videoId);
if (cached && now < cached.expiresAt) return cached;
const out = await runYtdlpResilient(['-J', '--no-warnings', `https://www.youtube.com/watch?v=${videoId}`]);
const info = JSON.parse(out);
...
streamCache.set(videoId, entry);
return entry;
}
```
`streamCache` and `STREAM_CACHE_MAX` are declared just above it.
## Steps
1. `server/server.js` — rename the existing function `resolveStreams` to
`resolveStreamsUncached` (definition only; body unchanged).
2. Directly after that function add:
```js
// One yt-dlp -J per video at a time: concurrent callers (warm-up + play,
// two devices, the media cache's getInfo) share the in-flight promise.
const inflightStreams = new Map(); // videoId -> Promise<entry>
function resolveStreams(videoId) {
const cached = streamCache.get(videoId);
if (cached && Date.now() < cached.expiresAt) return Promise.resolve(cached);
let p = inflightStreams.get(videoId);
if (!p) {
p = resolveStreamsUncached(videoId).finally(() => inflightStreams.delete(videoId));
inflightStreams.set(videoId, p);
}
return p;
}
```
All existing callers keep calling `resolveStreams` (it still returns a Promise).
3. Create `server/streams-inflight.test.js`? — NO. Keep it in-file; instead verify with the
script below.
## Out of scope / do NOT touch
- Cache TTL logic, `runYtdlpResilient`, format filtering.
## Verification
```bash
cd /home/user/ytplayer/server && grep -n "function resolveStreams\|function resolveStreamsUncached\|inflightStreams" server.js
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
```
Expected: both functions + map present, `SERVER_OK`, all server test files `0 fail`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

111
plans/queue/005-warm-streams-on-intent-47b3d3.md Normal file
View File

@@ -0,0 +1,111 @@
---
id: 005-warm-streams-on-intent-47b3d3
title: Warm the stream cache for likely next plays
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40]
est_files: 2
---
# 005 — Warm the stream cache for likely next plays
## Objective
A cold `/api/streams` costs ~6 s of yt-dlp; a cached one ~0 s (1.2 s total on prod,
all network). Start that work before the user taps:
- new `GET /api/streams/warm?v=<id>` → `204` immediately, resolves in the background
(shares the in-flight promise from plan 004), at most 2 warm resolves at a time,
skipped when the server already holds a ready media copy.
- the client warms: the top 3 search results after a search renders; a card on
`pointerdown` (touch/mouse down fires ~100–300 ms before `click`); the next 2 queue
items when a song starts playing.
## Context the executor must NOT rediscover
- `server/server.js` `resolveStreams(videoId)` returns a Promise and dedupes (plan 004).
- `media.getReady(videoId)` (server/media-cache.js) resolves the ready row or null.
- `server/server.js:532` — `app.get('/api/streams', async (c) => {` — register the new
route directly ABOVE it (Hono matches `/api/streams/warm` separately anyway).
- Client `frontend/app.js`:
- `const YT_ID_RE = /^[A-Za-z0-9_-]{11}$/;` at ~line 1202 (declared later in the file
than the helper you add — fine, it is only read at call time).
- `runSearchQuery` success path (~line 8603): `searchResults = results; … renderList(); RecentSearches.cacheResults(q, results);`
- `renderCard(v, index, list)` at ~line 7725; it has `card.addEventListener('click', (e) => {` at ~line 7781.
- globals `queue` (array of video objects) and `queueIndex` (~line 353).
- master `playing` listener at ~line 2287 (`el.addEventListener('playing', () => { if (!masterIs(el)) return; …`).
- `WEB` constant is true for the PWA; `cachedIds` is a Set of ids saved on this device.
## Steps
1. `server/server.js` — above `app.get('/api/streams', …)` add:
```js
// GET /api/streams/warm?v=<id> — fire-and-forget: resolve streams into
// streamCache so the real /api/streams a moment later is instant. Bounded
// so a scrolling user can't queue dozens of yt-dlp processes.
const WARM_MAX = 2;
let warmActive = 0;
app.get('/api/streams/warm', async (c) => {
const id = (c.req.query('v') || '').trim();
if (!/^[A-Za-z0-9_-]{11}$/.test(id)) return c.body(null, 204);
if (warmActive >= WARM_MAX) return c.body(null, 204);
try { if (await media.getReady(id)) return c.body(null, 204); } catch { /* fall through */ }
warmActive++;
resolveStreams(id).catch(() => {}).finally(() => { warmActive--; });
return c.body(null, 204);
});
```
2. `frontend/app.js` — directly after the `const API = { … };` object (~line 320) add:
```js
// Pre-resolve streams for videos the user is likely to play next (server
// /api/streams/warm). Each id is warmed at most once per 20 min per tab.
const warmedAt = new Map();
function warmStreams(ids) {
if (!WEB || navigator.onLine === false) return;
const now = Date.now();
for (const id of ids) {
if (!id || !/^[A-Za-z0-9_-]{11}$/.test(id) || cachedIds.has(id)) continue;
if (now - (warmedAt.get(id) || 0) < 20 * 60_000) continue;
warmedAt.set(id, now);
fetch(`/api/streams/warm?v=${encodeURIComponent(id)}`, { priority: 'low' }).catch(() => {});
}
}
```
3. `frontend/app.js` `runSearchQuery` — after `RecentSearches.cacheResults(q, results);` add
`warmStreams(results.slice(0, 3).map((r) => r.id));`
4. `frontend/app.js` `renderCard` — directly BEFORE `card.addEventListener('click', (e) => {` add:
```js
card.addEventListener('pointerdown', () => warmStreams([v.id]), { passive: true });
```
5. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` (and after
the plan-001 perf lines if present) add:
```js
if (Array.isArray(queue) && queueIndex >= 0) warmStreams(queue.slice(queueIndex + 1, queueIndex + 3).map((x) => x && x.id));
```
## Out of scope / do NOT touch
- Do not warm on hover/scroll, do not warm uploads (`upl_…`) — they need no yt-dlp.
- Do not change `/api/streams` itself.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && echo APP_OK
cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
[ -e public ] || ln -s ../frontend public; PORT=3997 bun server.js >/tmp/ytp005.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=bad'
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=dQw4w9WgXcQ'
kill $SRV; true
cd .. && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `APP_OK`, `SERVER_OK`, `204`, `204`, tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

212
plans/queue/006-innertube-search-48066b.md Normal file
View File

@@ -0,0 +1,212 @@
---
id: 006-innertube-search-48066b
title: Answer searches from YouTube InnerTube directly with yt-dlp fallback
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 4
---
# 006 — InnerTube search with yt-dlp fallback
## Objective
`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process
(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search
API returns the same data in a few hundred ms. After this plan `/api/search` tries
InnerTube first (6 s timeout), maps results to the existing card shape, and falls
back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos.
Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri
bridge contract are untouched.
## Context the executor must NOT rediscover
A trimmed real response is committed at `server/fixtures/innertube-search.json`
(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`,
and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29):
```
contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
.itemSectionRenderer.contents[].videoRenderer:
videoId -> id
title.runs[0].text -> title
ownerText.runs[0].text -> channel
ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId -> channelId (UC…)
ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl -> '/@handle' or '/channel/UC…'
lengthText.simpleText "5:43" | "1:59:47" | absent (live) -> duration seconds (0 if absent)
```
Request that works (no key needed):
```
POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
Content-Type: application/json
{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}
```
The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable.
Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first
cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the
fallback is for. Do not "fix" the 403 by adding cookies/keys.
Existing card shape — `server/server.js:293-305` `slimEntry`:
```js
return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };
```
`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`.
Current route `server/server.js:365-395` (inside it):
```js
let mine = [];
try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
try {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
const results = [...mine, ...parseCards(out)];
```
Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script
runs each file separately joined by `&&`.
## Steps
1. Create `server/innertube.js`:
```js
/* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
* parseSearch() is pure (tested against fixtures/innertube-search.json);
* search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
export function lengthToSeconds(s) {
if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
}
export function parseSearch(json) {
const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
?.sectionListRenderer?.contents;
if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
const out = [];
for (const s of sections) {
for (const it of s?.itemSectionRenderer?.contents || []) {
const v = it && it.videoRenderer;
if (!v || typeof v.videoId !== 'string') continue;
const owner = v.ownerText?.runs?.[0] || {};
const be = owner.navigationEndpoint?.browseEndpoint || {};
const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
out.push({
id: v.videoId,
title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
channel: owner.text || '',
channelId: be.browseId || '',
channelUrl: path ? `https://www.youtube.com${path}` : '',
duration: lengthToSeconds(v.lengthText?.simpleText),
thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
});
}
}
return out;
}
export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ context: { client: CLIENT }, query: q }),
signal: AbortSignal.timeout(timeoutMs),
});
if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
return parseSearch(await res.json());
}
```
2. Create `server/innertube.test.js`:
```js
import { test, expect } from 'bun:test';
import { readFileSync } from 'node:fs';
import { parseSearch, lengthToSeconds, search } from './innertube.js';
const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
test('parses video renderers into slim cards', () => {
const r = parseSearch(fixture);
expect(r.length).toBe(4);
expect(r[0]).toEqual({
id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
});
expect(r[1].duration).toBe(7187);
expect(r[3].duration).toBe(0); // live, no lengthText
});
test('lengthToSeconds', () => {
expect(lengthToSeconds('5:43')).toBe(343);
expect(lengthToSeconds('1:59:47')).toBe(7187);
expect(lengthToSeconds('LIVE')).toBe(0);
expect(lengthToSeconds(undefined)).toBe(0);
});
test('unexpected shape throws (caller falls back to yt-dlp)', () => {
expect(() => parseSearch({})).toThrow();
});
test('search() throws on HTTP error', async () => {
const fetchImpl = async () => new Response('no', { status: 429 });
await expect(search('x', { fetchImpl })).rejects.toThrow('429');
});
```
If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's
`lengthText` and use that value (the fixture is the source of truth).
3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`.
4. `server/server.js` — add import next to the other local imports:
`import * as innertube from './innertube.js';`
5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];`
head with:
```js
try {
let yt = [];
try {
yt = await innertube.search(q);
} catch (e) {
console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
}
if (!yt.length) {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
yt = parseCards(out);
}
const results = [...mine, ...yt];
```
Everything after (`searchCache.set`, return, catch) stays as is.
6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')`
skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented
line `# SEARCH_INNERTUBE: "0" # force yt-dlp search` next to the other commented env vars.
## Out of scope / do NOT touch
- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging.
- Card shape, `searchCache`, `app.js`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
kill $SRV; true
grep -c "innertube failed" /tmp/ytp006.log
```
Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns
`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container
has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

338
plans/queue/007-ytdlp-worker-045800.md Normal file
View File

@@ -0,0 +1,338 @@
---
id: 007-ytdlp-worker-045800
title: Keep one long-lived yt-dlp worker process instead of spawning per call
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b]
est_files: 5
---
# 007 — Long-lived yt-dlp worker pool
## Objective
Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with
yt-dlp 2026.08.19 (same host, same network):
| call | per-call spawn | warm worker |
|------|----------------|-------------|
| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s |
| `-J <video>` | 3.32 s | 2.29 s |
After this plan, the "read-only" yt-dlp calls (search fallback, channel, playlist
expand, `-J` stream resolve) go to a pool of 2 long-lived Python workers that import
`yt_dlp` once. Downloads (which write files and use an AbortSignal) keep spawning.
Any pool infrastructure problem falls back to spawning transparently. A bot-check answer
recycles the workers and retries that call with a fresh spawn; each worker is replaced after
100 requests. `YTDLP_WORKER=0` disables the pool.
Dry-run notes (2026-09-29): the worker calls yt-dlp's own CLI entry (`yt_dlp._real_main`) — a
first version using `parse_options()` + `YoutubeDL` directly behaved differently from the CLI.
During testing the container's IP got bot-checked by YouTube for plain CLI calls too, so the
pooled `-J` path could only be measured before that (2.29 s vs 3.32 s). Measure on prod with
plan 001's `[ytdlp] pooled …` log lines and keep `YTDLP_WORKER=0` as the escape hatch.
## Context the executor must NOT rediscover
- In Docker, yt-dlp is the release **zipapp** at `/usr/local/bin/yt-dlp`; Python can import
`yt_dlp` from it by putting that path on `sys.path`. `python3` is installed in the image
(Dockerfile apt line). `/etc/yt-dlp.conf` (the `--js-runtimes bun:…` line) is honoured
because `yt_dlp.parse_options()` reads config files like the CLI does.
- The Dockerfile copies `server/` to `/app`, so a file at `server/ytdlp-worker.py` ships
automatically.
- `server/server.js:143` — `function runYtdlp(args, { signal } = {})` spawns yt-dlp
(after plan 001 it also logs `[ytdlp] <kind> <ms>ms ok|fail`).
- `server/server.js:189` — `async function runYtdlpResilient(args, opts = {})` calls
`runYtdlp(withCookies(args), opts)` and, on a bot check, retries with
`runYtdlp(withCookies(['--extractor-args', …, ...args]), opts)`. `opts` is passed through.
- Read-only call sites to mark `{ pooled: true }` (line numbers from before plans 004/006;
search by the text):
1. `/api/search` fallback: `runYtdlpResilient([\`ytsearch${SEARCH_LIMIT}:${q}\`, …])`
2. `/api/channel`: `runYtdlpResilient([` … `'--playlist-end', String(CHANNEL_LIMIT),`
3. `resolveStreamsUncached`: `runYtdlpResilient(['-J', '--no-warnings', \`https://www.youtube.com/watch?v=${videoId}\`])`
4. `/api/playlist/expand` (~line 1712): `const out = await runYtdlpResilient([`
Do NOT mark the calls at ~861, ~932, ~988 (downloads) or `server/notes.js:362`.
- `const YTDLP = process.env.YTDLP_PATH || 'yt-dlp';` near the top of server.js.
- Expected noise on a machine WITHOUT yt-dlp (local dev before `npm run setup`): three
`ModuleNotFoundError: No module named 'yt_dlp'` lines, then
`[ytdlp-pool] disabled after 3 failed starts …` — the server keeps working via spawn. Not a bug.
## Steps
1. Create `server/ytdlp-worker.py` with exactly:
```python
#!/usr/bin/env python3
"""ytdlp-worker.py - one long-lived yt-dlp process answering many requests.
Spawning yt-dlp per call pays Python start-up + extractor import every time.
This worker imports yt_dlp ONCE and runs each request's argv in-process
through the CLI's own entry point (yt_dlp._real_main), so config files such
as /etc/yt-dlp.conf and everything else the command line sets up apply.
Protocol (newline-delimited JSON):
stdin : {"id": <int>, "args": [<yt-dlp argv>...]}
stdout: {"id": <int>, "code": <int>, "out": "<captured stdout>", "err": "<stderr tail>"}
Only for calls whose result is printed to stdout (-J, --dump-json, --version).
Usage: python3 ytdlp-worker.py <path-to-yt-dlp-zipapp-or-empty>
"""
import io, json, sys, contextlib
if len(sys.argv) > 1 and sys.argv[1]:
sys.path.insert(0, sys.argv[1]) # the yt-dlp release binary is a zipapp
import yt_dlp # noqa: E402
proto_out = sys.stdout
sys.stdout = io.StringIO() # nothing may leak onto the protocol pipe
def run(args):
out, err = io.StringIO(), io.StringIO()
code = 0
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
try:
# The CLI's own entry point (not parse_options + YoutubeDL): it also
# sets up what the command line does (JS challenge solving, plugins,
# post-processing defaults). A bare YoutubeDL got "Sign in to
# confirm you're not a bot" where the CLI succeeded.
ret = yt_dlp._real_main(args)
code = ret[0] if isinstance(ret, tuple) else (ret or 0)
except SystemExit as e:
code = e.code if isinstance(e.code, int) else 1
except Exception as e: # report, never die
err.write(f"ERROR: {e}\n")
code = 1
return code, out.getvalue(), err.getvalue()[-8000:]
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
code, out, err = run([str(a) for a in req.get("args", [])])
resp = {"id": req.get("id"), "code": code, "out": out, "err": err}
except Exception as e:
resp = {"id": None, "code": 1, "out": "", "err": f"worker: {e}"}
proto_out.write(json.dumps(resp) + "\n")
proto_out.flush()
```
2. Create `server/ytdlp-pool.js` with exactly:
```js
/* ytdlp-pool.js — a small pool of long-lived yt-dlp workers (ytdlp-worker.py).
*
* run(args) resolves stdout exactly like a spawned `yt-dlp <args>` would, or
* rejects with the captured stderr. Each worker handles one request at a time;
* extra requests queue. A worker that exits or exceeds the timeout is killed
* and replaced. Errors carrying `poolInfra: true` mean "the pool could not run
* this" — the caller falls back to a normal per-call spawn. Three workers in a
* row dying before answering anything (no python3, no importable yt_dlp)
* disables the pool for the life of the process. Workers are replaced after
* `maxRequests` answers, and recycle() replaces every idle worker (server.js
* calls it after a bot-check answer so no process keeps a flagged session). */
import { spawn } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const WORKER = join(dirname(fileURLToPath(import.meta.url)), 'ytdlp-worker.py');
const infra = (msg) => Object.assign(new Error(msg), { poolInfra: true });
export function createYtdlpPool({ ytdlpPath = '', size = 2, timeoutMs = 60_000, maxRequests = 100, python = 'python3', log = console } = {}) {
const workers = [];
const queue = [];
let nextId = 1;
let startFailures = 0;
let disabled = false;
let closed = false;
function startWorker() {
const child = spawn(python, [WORKER, ytdlpPath], { stdio: ['pipe', 'pipe', 'inherit'] });
const w = { child, busy: null, buf: '', dead: false, retiring: false, served: 0 };
child.stdout.setEncoding('utf8');
child.stdout.on('data', (d) => {
w.buf += d;
let nl;
while ((nl = w.buf.indexOf('\n')) >= 0) {
const line = w.buf.slice(0, nl);
w.buf = w.buf.slice(nl + 1);
let msg;
try { msg = JSON.parse(line); } catch { continue; }
const job = w.busy;
if (!job || msg.id !== job.id) continue;
clearTimeout(job.timer);
w.busy = null;
w.served++;
startFailures = 0;
if (msg.code === 0) job.resolve(msg.out);
else job.reject(new Error((msg.err || '').trim() || 'yt-dlp exited with code ' + msg.code));
if (w.served >= maxRequests) retire(w);
pump();
}
});
const onDead = (why) => {
if (w.dead) return;
w.dead = true;
const i = workers.indexOf(w);
if (i >= 0) workers.splice(i, 1);
if (w.busy) { clearTimeout(w.busy.timer); w.busy.reject(infra('yt-dlp worker ' + why)); w.busy = null; }
if (closed) return;
if (!w.served && ++startFailures >= 3) {
disabled = true;
log.warn?.(`[ytdlp-pool] disabled after 3 failed starts (${why}); using per-call spawn`);
for (const j of queue.splice(0)) j.reject(infra('pool disabled'));
return;
}
if (!w.retiring) log.warn?.(`[ytdlp-pool] worker ${why}; restarting`);
const t = setTimeout(() => { if (!closed && !disabled) { workers.push(startWorker()); pump(); } }, 1000);
t.unref?.();
};
child.on('exit', (code) => onDead('exited (' + code + ')'));
child.on('error', (e) => onDead('failed to start: ' + e.message));
return w;
}
// Replace a worker once it is idle (its exit handler starts a fresh one).
function retire(w) {
if (w.retiring || w.dead) return;
w.retiring = true;
try { w.child.kill(); } catch { /* gone */ }
}
for (let i = 0; i < size; i++) workers.push(startWorker());
function pump() {
for (const w of workers) {
if (w.dead || w.retiring || w.busy || !queue.length) continue;
const job = queue.shift();
w.busy = job;
job.timer = setTimeout(() => { try { w.child.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
try { w.child.stdin.write(JSON.stringify({ id: job.id, args: job.args }) + '\n'); }
catch (e) { clearTimeout(job.timer); w.busy = null; job.reject(infra(e.message)); }
}
}
function run(args) {
if (closed || disabled) return Promise.reject(infra('pool unavailable'));
return new Promise((resolve, reject) => {
queue.push({ id: nextId++, args: args.map(String), resolve, reject });
pump();
});
}
function close() {
closed = true;
for (const w of workers) { try { w.child.kill(); } catch { /* gone */ } }
for (const j of queue.splice(0)) j.reject(infra('pool closed'));
}
function recycle() {
for (const w of workers) if (!w.busy) retire(w);
}
return { run, close, recycle, get disabled() { return disabled; } };
}
```
3. Create `server/ytdlp-pool.test.js` with exactly:
```js
import { test, expect } from 'bun:test';
import { existsSync } from 'node:fs';
import { createYtdlpPool } from './ytdlp-pool.js';
// The worker imports yt_dlp from the release zipapp. Skip when none is around.
const YTDLP = [process.env.YTDLP_PATH, '../bin/yt-dlp', Bun.which('yt-dlp')].find((p) => p && existsSync(p)) || '';
const quiet = { warn() {}, info() {} };
test.skipIf(!YTDLP)('runs requests and reports yt-dlp errors without dying', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, log: quiet });
try {
const v = await pool.run(['--version']);
expect(v.trim()).toMatch(/^\d{4}\.\d{2}\.\d{2}/);
const err = await pool.run(['--definitely-not-a-flag']).catch((e) => e);
expect(err).toBeInstanceOf(Error);
expect(err.poolInfra).toBeFalsy();
expect((await pool.run(['--version'])).trim()).toBe(v.trim());
} finally { pool.close(); }
}, 30000);
test('a broken python disables the pool and rejects as infra', async () => {
const pool = createYtdlpPool({ python: '/nonexistent/python3', size: 1, log: quiet });
const err = await pool.run(['--version']).catch((e) => e);
expect(err.poolInfra).toBe(true);
await new Promise((r) => setTimeout(r, 3500));
expect(pool.disabled).toBe(true);
expect((await pool.run(['--version']).catch((e) => e)).poolInfra).toBe(true);
pool.close();
}, 15000);
test.skipIf(!YTDLP)('workers are replaced after maxRequests and by recycle(), without failing requests', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, maxRequests: 1, log: quiet });
try {
const a = await pool.run(['--version']);
const b = await pool.run(['--version']); // served by a fresh worker
expect(b).toBe(a);
pool.recycle();
expect(await pool.run(['--version'])).toBe(a);
expect(pool.disabled).toBe(false);
} finally { pool.close(); }
}, 30000);
```
4. `server/package.json` "test" script — append ` && bun test ./ytdlp-pool.test.js`.
5. `server/server.js`:
a. Add import: `import { createYtdlpPool } from './ytdlp-pool.js';`
b. Rename the existing `function runYtdlp(args, { signal } = {})` to
`function runYtdlpSpawn(args, { signal } = {})` (body unchanged).
c. Directly after it add:
```js
// Read-only calls (-J, --dump-json) go to long-lived workers that import
// yt_dlp once (~1 s saved per call). Downloads keep spawning. Pool trouble
// (not a yt-dlp error) falls back to a spawn. YTDLP_WORKER=0 disables it.
const ytdlpPool = process.env.YTDLP_WORKER === '0' ? null : createYtdlpPool({
ytdlpPath: Bun.which(YTDLP) || YTDLP,
size: Math.max(1, Number(process.env.YTDLP_WORKERS) || 2),
});
function runYtdlp(args, opts = {}) {
if (!opts.pooled || !ytdlpPool || opts.signal) return runYtdlpSpawn(args, opts);
const t0 = Date.now();
return ytdlpPool.run(args).then(
(out) => { console.log(`[ytdlp] pooled ${Date.now() - t0}ms ok`); return out; },
(err) => {
if (err.poolInfra) return runYtdlpSpawn(args, opts);
// A bot check can stick to a long-lived process: replace the workers and
// answer this call the old way, from a fresh process.
if (BOT_CHECK_RE.test(err.message)) { ytdlpPool.recycle(); return runYtdlpSpawn(args, opts); }
console.log(`[ytdlp] pooled ${Date.now() - t0}ms fail`);
throw err;
},
);
}
```
d. At the 4 read-only call sites listed in Context, add a second argument `{ pooled: true }`
to `runYtdlpResilient(...)`. Example: `runYtdlpResilient(['-J', '--no-warnings', url], { pooled: true })`.
6. `docker-compose.yml` — next to the commented yt-dlp env lines add:
`# YTDLP_WORKER: "0" # disable the long-lived yt-dlp worker pool`
`# YTDLP_WORKERS: "2" # pool size`
## Out of scope / do NOT touch
- Download paths (~861, ~932, ~988), `server/notes.js`, `withSaveSlot`, fallback-client logic.
- Dockerfile (python3 is already installed; the worker file ships with `server/`).
## Verification
```bash
cd /home/user/ytplayer && npm run setup >/dev/null 2>&1; ls -la bin/yt-dlp
cd server && bun test ./ytdlp-pool.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail|skip)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "pooled: true" server.js
```
Expected: pool tests `3 pass` (or `1 pass 2 skip` if `bin/yt-dlp` could not be downloaded —
say so in Findings); every file `0 fail`; `SERVER_OK`; grep prints `4`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

427
plans/queue/008-p2p-schema-and-config-127966.md Normal file
View File

@@ -0,0 +1,427 @@
---
id: 008-p2p-schema-and-config-127966
title: Add P2P tables, config flags and db helpers
created: 2026-09-29
depends_on: []
est_files: 5
---
# 008 — P2P tables, config flags and db helpers
## Objective
Lay the storage foundation for peer-to-peer sharing described in
`docs/p2p-architecture.md` (READ IT FIRST — it is short). After this plan:
- `server/p2p-config.js` exports `P2P` (config) and `loadP2pConfig(env)`.
Defaults: **P2P enabled**, **malware scan disabled**, stale after 7 days.
- `server/p2p-db.js` creates `p2p_content`, `p2p_devices`, `p2p_holders`,
`video_views` and adds `media_cache.sha256`, with query helpers.
- The server calls `initP2pSchema()` at boot. Nothing else changes behaviour yet.
## Context the executor must NOT rediscover
- `server/db.js` exports `db` (libsql client) and `initDb()`. Its `MEDIA_COLS` set
(~line 308) whitelists columns `upsertMedia()` may write:
```js
const MEDIA_COLS = new Set([
'status', 'gen', 'size', 'height', 'vcodec', 'acodec', 'duration', 'optimized',
'meta', 'priority', 'auto', 'attempts', 'error', 'retry_at', 'created_at',
'updated_at', 'last_access', 'hits',
]);
```
- `server/server.js` `main()` (~line 1916): `await initDb();` then `await media.init();`.
- Server test conventions: `bun:test`, a temp `DB_PATH` set BEFORE importing `./db.js`
(see `server/notes.test.js:1-12`), and each file runs in its own `bun test` process via the
`server/package.json` "test" script (db.js is a singleton).
## Steps
1. Create `server/p2p-config.js` with exactly:
```js
/* p2p-config.js — peer-to-peer settings (see docs/p2p-architecture.md).
* P2P is ON unless P2P_ENABLED=0. The malware scan is OFF unless
* P2P_MALWARE_SCAN=1. Hashing + validateMedia are never optional. */
import { dirname, join } from 'node:path';
const num = (v, d) => (Number.isFinite(Number(v)) && String(v).trim() !== '' ? Number(v) : d);
export function loadP2pConfig(env = process.env) {
const dbDir = dirname(env.DB_PATH || './data/ytplayer.db');
return {
enabled: env.P2P_ENABLED !== '0',
malwareScan: env.P2P_MALWARE_SCAN === '1',
scanCmd: (env.P2P_SCAN_CMD || 'clamscan --no-summary --infected').trim(),
staleDays: num(env.P2P_STALE_DAYS, 7),
keepMinViews: num(env.P2P_KEEP_MIN_VIEWS, 3),
keepDays: num(env.P2P_KEEP_DAYS, 30),
keepRecentDays: num(env.P2P_KEEP_RECENT_DAYS, 14),
intakeDir: env.P2P_INTAKE_DIR || join(dbDir, 'p2p-intake'),
intakeMaxBytes: num(env.P2P_INTAKE_MAX_BYTES, 3 * 1024 ** 3),
};
}
export const P2P = loadP2pConfig();
```
2. Create `server/p2p-db.js` — copy VERBATIM from the "p2p-db.js" appendix at the end of this plan.
3. Create `server/p2p-db.test.js` — copy VERBATIM from the "p2p-db.test.js" appendix.
4. `server/db.js` — add `'sha256'` to the end of `MEDIA_COLS` (after `'hits'`).
5. `server/package.json` "test" script — append ` && bun test ./p2p-db.test.js`.
6. `server/server.js` — add `import { initP2pSchema } from './p2p-db.js';` next to the other
local imports, and in `main()` directly after `await initDb();` add `await initP2pSchema();`.
7. `docker-compose.yml` — in the `ytplayer` service `environment:` block, after the
`LYRICS_WORKER_TOKEN` line, add:
```yaml
# Peer-to-peer sharing (docs/p2p-architecture.md). ON by default.
P2P_ENABLED: "${P2P_ENABLED:-1}"
# Malware scan before a file's hash is admitted. OFF by default; needs an
# image built with INSTALL_CLAMAV=1. Hashing + media validation always run.
P2P_MALWARE_SCAN: "${P2P_MALWARE_SCAN:-0}"
# P2P_STALE_DAYS: "7" # holder shown as stale after this many days unchecked
# P2P_KEEP_MIN_VIEWS: "3" # server keeps copies with ≥ this many views…
# P2P_KEEP_DAYS: "30" # …in this many days
# P2P_KEEP_RECENT_DAYS: "14" # …or played this recently
```
## Out of scope / do NOT touch
- No routes, no media-cache changes, no frontend changes (later plans).
- Do not edit `initDb()`'s SQL; the new column is added by `initP2pSchema()`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./p2p-db.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
```
Expected: `4 pass 0 fail`; every file `0 fail`; `SERVER_OK`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files in full).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — p2p-db.js
```js
/* ============================================================================
* p2p-db.js — tables + queries for peer-to-peer sharing
* (docs/p2p-architecture.md). Shares the libsql client from db.js.
*
* p2p_content one row per verified file (cid = sha256 of the bytes); never
* deleted, only revoked — the catalog grows over time
* p2p_devices registered devices (secret stored as sha256)
* p2p_holders which device holds which cid; PERSISTENT (no TTL) with
* last_verified_at — the UI decides what is "stale"
* video_views per-video per-day view counts (retention criteria)
* All timestamps are ms epochs.
* ========================================================================== */
import { db } from './db.js';
export async function initP2pSchema() {
await db.executeMultiple(`
CREATE TABLE IF NOT EXISTS p2p_content (
cid TEXT PRIMARY KEY,
video_id TEXT NOT NULL,
size INTEGER NOT NULL,
height INTEGER NOT NULL DEFAULT 0,
vcodec TEXT,
acodec TEXT,
duration REAL NOT NULL DEFAULT 0,
meta TEXT NOT NULL DEFAULT '{}',
origin TEXT NOT NULL, -- server | intake
status TEXT NOT NULL DEFAULT 'verified', -- verified | revoked
scan TEXT NOT NULL DEFAULT 'skipped', -- skipped | clean
created_at INTEGER NOT NULL,
verified_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_p2p_content_video ON p2p_content (video_id, created_at DESC);
CREATE TABLE IF NOT EXISTS p2p_devices (
device_id TEXT PRIMARY KEY,
secret_hash TEXT NOT NULL,
fingerprint TEXT,
profile TEXT,
share INTEGER NOT NULL DEFAULT 1,
created_at INTEGER NOT NULL,
last_seen_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS p2p_holders (
cid TEXT NOT NULL,
device_id TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active', -- active | removed
trust TEXT NOT NULL DEFAULT 'reported', -- reported | challenged
first_reported_at INTEGER NOT NULL,
last_verified_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (cid, device_id)
);
CREATE INDEX IF NOT EXISTS idx_p2p_holders_device ON p2p_holders (device_id, status);
CREATE TABLE IF NOT EXISTS video_views (
video_id TEXT NOT NULL,
day TEXT NOT NULL,
n INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (video_id, day)
);
`);
// media_cache.sha256 — the cid of the current <id>.<gen>.mp4 (plan 009).
try { await db.execute('ALTER TABLE media_cache ADD COLUMN sha256 TEXT'); }
catch (e) { if (!/duplicate column/i.test(String(e.message))) throw e; }
}
const rowsOf = (r) => r.rows.map((row) => {
const o = {};
r.columns.forEach((c, i) => { o[c] = row[i]; });
return o;
});
// ---- content ----------------------------------------------------------------
export async function upsertContent(c) {
await db.execute({
sql: `INSERT INTO p2p_content (cid, video_id, size, height, vcodec, acodec, duration, meta, origin, status, scan, created_at, verified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'verified', ?, ?, ?)
ON CONFLICT(cid) DO UPDATE SET verified_at = excluded.verified_at, scan = excluded.scan`,
args: [c.cid, c.videoId, c.size, c.height || 0, c.vcodec || null, c.acodec || null, c.duration || 0,
JSON.stringify(c.meta || {}), c.origin, c.scan || 'skipped', c.now, c.now],
});
}
export async function getContent(cid) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_content WHERE cid = ?', args: [cid] });
return rowsOf(r)[0] || null;
}
export async function listContentForVideo(videoId) {
const r = await db.execute({
sql: "SELECT * FROM p2p_content WHERE video_id = ? AND status = 'verified' ORDER BY created_at DESC LIMIT 10",
args: [videoId],
});
return rowsOf(r);
}
export async function knownCids(cids) {
if (!cids.length) return new Set();
const out = new Set();
for (let i = 0; i < cids.length; i += 200) {
const part = cids.slice(i, i + 200);
const r = await db.execute({
sql: `SELECT cid FROM p2p_content WHERE status = 'verified' AND cid IN (${part.map(() => '?').join(',')})`,
args: part,
});
for (const row of r.rows) out.add(row[0]);
}
return out;
}
export async function revokeContent(cid) {
await db.execute({ sql: "UPDATE p2p_content SET status = 'revoked' WHERE cid = ?", args: [cid] });
}
// ---- devices ------------------------------------------------------------------
export async function createDevice({ deviceId, secretHash, fingerprint, profile, now }) {
await db.execute({
sql: `INSERT INTO p2p_devices (device_id, secret_hash, fingerprint, profile, share, created_at, last_seen_at)
VALUES (?, ?, ?, ?, 1, ?, ?)`,
args: [deviceId, secretHash, fingerprint || null, profile || null, now, now],
});
}
export async function getDevice(deviceId) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_devices WHERE device_id = ?', args: [deviceId] });
return rowsOf(r)[0] || null;
}
export async function touchDevice(deviceId, { now, share, profile } = {}) {
await db.execute({
sql: `UPDATE p2p_devices SET last_seen_at = ?,
share = COALESCE(?, share), profile = COALESCE(?, profile)
WHERE device_id = ?`,
args: [now, share === undefined ? null : (share ? 1 : 0), profile || null, deviceId],
});
}
// ---- holders (persistent; never expired by time) ------------------------------
export async function upsertHolder({ cid, deviceId, trust = 'reported', now }) {
await db.execute({
sql: `INSERT INTO p2p_holders (cid, device_id, status, trust, first_reported_at, last_verified_at)
VALUES (?, ?, 'active', ?, ?, ?)
ON CONFLICT(cid, device_id) DO UPDATE SET
status = 'active', removed_at = NULL, last_verified_at = excluded.last_verified_at,
trust = CASE WHEN p2p_holders.trust = 'challenged' OR excluded.trust = 'challenged'
THEN 'challenged' ELSE 'reported' END`,
args: [cid, deviceId, trust, now, now],
});
}
export async function setHolderTrust({ cid, deviceId, trust, now }) {
await db.execute({
sql: 'UPDATE p2p_holders SET trust = ?, last_verified_at = ? WHERE cid = ? AND device_id = ?',
args: [trust, now, cid, deviceId],
});
}
export async function removeHolder({ cid, deviceId, now }) {
await db.execute({
sql: "UPDATE p2p_holders SET status = 'removed', removed_at = ? WHERE cid = ? AND device_id = ? AND status = 'active'",
args: [now, cid, deviceId],
});
}
// A full report: every active holding of this device NOT in `keep` is removed.
export async function removeHoldersExcept({ deviceId, keep, now }) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
const keepSet = new Set(keep);
let removed = 0;
for (const row of r.rows) {
if (keepSet.has(row[0])) continue;
await removeHolder({ cid: row[0], deviceId, now });
removed++;
}
return removed;
}
export async function activeHoldingsOf(deviceId) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
return r.rows.map((row) => row[0]);
}
// Holders of one cid, joined with the device's share flag. Newest check first.
export async function listHolders(cid, limit = 50) {
const r = await db.execute({
sql: `SELECT h.device_id, h.trust, h.first_reported_at, h.last_verified_at, d.share
FROM p2p_holders h JOIN p2p_devices d ON d.device_id = h.device_id
WHERE h.cid = ? AND h.status = 'active'
ORDER BY h.last_verified_at DESC LIMIT ?`,
args: [cid, limit],
});
return rowsOf(r);
}
// ---- views + stats --------------------------------------------------------------
export function dayKey(ms) {
return new Date(ms).toISOString().slice(0, 10);
}
export async function addView(videoId, now) {
await db.execute({
sql: `INSERT INTO video_views (video_id, day, n) VALUES (?, ?, 1)
ON CONFLICT(video_id, day) DO UPDATE SET n = n + 1`,
args: [videoId, dayKey(now)],
});
}
export async function viewsSince(videoId, sinceMs) {
const r = await db.execute({
sql: 'SELECT COALESCE(SUM(n), 0) FROM video_views WHERE video_id = ? AND day >= ?',
args: [videoId, dayKey(sinceMs)],
});
return Number(r.rows[0][0]) || 0;
}
export async function p2pStats() {
const one = async (sql) => Number((await db.execute(sql)).rows[0][0]) || 0;
return {
content: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'verified'"),
revoked: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'revoked'"),
devices: await one('SELECT COUNT(*) FROM p2p_devices'),
holders: await one("SELECT COUNT(*) FROM p2p_holders WHERE status = 'active'"),
heldCids: await one("SELECT COUNT(DISTINCT cid) FROM p2p_holders WHERE status = 'active'"),
};
}
```
## Appendix — p2p-db.test.js
```js
// P2P tables against a real temp libsql DB (docs/p2p-architecture.md).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const { loadP2pConfig } = await import('./p2p-config.js');
const CID = 'a'.repeat(64);
const CID2 = 'b'.repeat(64);
const T0 = Date.UTC(2026, 8, 29, 12);
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
await P.initP2pSchema(); // idempotent (ALTER TABLE duplicate column is ignored)
});
test('config defaults: P2P on, malware scan off', () => {
const c = loadP2pConfig({});
expect(c.enabled).toBe(true);
expect(c.malwareScan).toBe(false);
expect(c.staleDays).toBe(7);
expect(loadP2pConfig({ P2P_ENABLED: '0', P2P_MALWARE_SCAN: '1' })).toMatchObject({ enabled: false, malwareScan: true });
});
test('content upsert/get/known/revoke', async () => {
await P.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1000, height: 720, vcodec: 'h264', acodec: 'aac', duration: 212, meta: { title: 'x' }, origin: 'server', now: T0 });
const c = await P.getContent(CID);
expect(c.video_id).toBe('dQw4w9WgXcQ');
expect(c.status).toBe('verified');
expect([...(await P.knownCids([CID, CID2]))]).toEqual([CID]);
expect((await P.listContentForVideo('dQw4w9WgXcQ')).length).toBe(1);
await P.upsertContent({ cid: CID2, videoId: 'dQw4w9WgXcQ', size: 5, origin: 'intake', now: T0 });
await P.revokeContent(CID2);
expect([...(await P.knownCids([CID2]))]).toEqual([]);
});
test('holders persist, never expire, and a full report removes missing ones', async () => {
await P.createDevice({ deviceId: 'dev_1', secretHash: 'h', fingerprint: 'fp', now: T0 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 });
// 90 days later with no new report: still listed (UI marks it stale).
let hs = await P.listHolders(CID);
expect(hs.length).toBe(1);
expect(hs[0].last_verified_at).toBe(T0);
await P.setHolderTrust({ cid: CID, deviceId: 'dev_1', trust: 'challenged', now: T0 + 1000 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', trust: 'reported', now: T0 + 2000 });
hs = await P.listHolders(CID);
expect(hs[0].trust).toBe('challenged'); // a later plain report never downgrades trust
expect(hs[0].last_verified_at).toBe(T0 + 2000);
expect(await P.removeHoldersExcept({ deviceId: 'dev_1', keep: [], now: T0 + 3000 })).toBe(1);
expect((await P.listHolders(CID)).length).toBe(0);
expect(await P.activeHoldingsOf('dev_1')).toEqual([]);
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 + 4000 }); // re-added
expect(await P.activeHoldingsOf('dev_1')).toEqual([CID]);
});
test('views per day and window sums', async () => {
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0 - 40 * 86400_000);
expect(await P.viewsSince('vid00000001', T0 - 30 * 86400_000)).toBe(2);
expect(await P.viewsSince('vid00000001', T0 - 50 * 86400_000)).toBe(3);
const s = await P.p2pStats();
expect(s.content).toBe(1);
expect(s.devices).toBe(1);
});
```

273
plans/queue/009-server-content-hash-186e7f.md Normal file
View File

@@ -0,0 +1,273 @@
---
id: 009-server-content-hash-186e7f
title: Hash every validated server copy and register it as verified content
created: 2026-09-29
depends_on: [008-p2p-schema-and-config-127966]
est_files: 7
---
# 009 — Server content hashes → verified P2P content
## Objective
Implements flow 1 of `docs/p2p-architecture.md`. After this plan:
- Every copy the media cache promotes (fetch lane AND compression lane) gets a
server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are
hashed by a background backfill 30 s after boot.
- Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only
when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted.
- `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: <cid>`.
- `/api/streams`' cached payload gets an additive `data.cid` (web-only, like
`data.serverCached`).
The media-cache edits were written and tested ahead of time (25/25 media-cache
tests pass, incl. 2 new ones); they ship as patch files.
## Context the executor must NOT rediscover
- Patches (made against the current tree; `media-cache.js` and its test are untouched by
plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`.
They add to `createMediaCache()` the options `hashFile` (default `sha256File` from
`./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the
file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call
`onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export
`backfillHashes()`.
- `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last
property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`.
- `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers:
```js
headers: {
'Content-Type': 'video/mp4',
'Content-Length': String(file.size),
'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
'Cache-Control': 'no-store',
'Access-Control-Allow-Origin': '*',
},
```
- `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns
`{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`.
- Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`).
## Steps
1. Create `server/hash.js` with exactly:
```js
/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
* The hex digest of a validated file is its P2P content id (cid). */
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
export function sha256File(path) {
return new Promise((resolve, reject) => {
const h = createHash('sha256');
createReadStream(path, { highWaterMark: 1024 * 1024 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
// Hash of bytes [offset, offset+length) — used for holder range challenges.
export function sha256Range(path, offset, length) {
return new Promise((resolve, reject) => {
if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
const h = createHash('sha256');
createReadStream(path, { start: offset, end: offset + length - 1 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
```
2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B.
4. Apply the patches from the repo root:
```bash
git apply plans/patches/009-media-cache.diff
git apply plans/patches/009-media-cache-test.diff
```
If either fails, STOP and report the error (do not hand-edit).
5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`.
6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from
plan 008 may already exist; merge into it):
```js
import { admitFile } from './p2p-admit.js';
import { P2P } from './p2p-config.js';
import * as p2pDb from './p2p-db.js';
```
If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in
`main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import.
7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add:
```js
// P2P (docs/p2p-architecture.md): every validated copy's server-computed
// hash becomes verified content, after the optional malware scan.
onReady: (info) => admitFile(
{ ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
{ cfg: P2P, upsertContent: p2pDb.upsertContent },
),
```
8. `server/server.js` `cachedDownloadResponse` — add to the headers object:
```js
...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),
```
9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add
`cid: row.sha256 || null,` and add a comment above the return:
`// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.`
10. `Dockerfile` — optional ClamAV, off by default. After the existing
`RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add:
```dockerfile
# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
ARG INSTALL_CLAMAV=0
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
fi
```
## Out of scope / do NOT touch
- `validateMedia()` itself, the eviction logic, any frontend file.
- Never serve anything from the intake dir; no new routes in this plan.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
bun test ./p2p-admit.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js
```
Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail`
(media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-admit.js
```js
/* ============================================================================
* p2p-admit.js — the ONLY way a content id enters p2p_content
* (docs/p2p-architecture.md, "Security rules").
*
* Callers must already have: the complete file on the server's own disk, its
* SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
* optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
* row. The scan command gets the path as its last argument: exit 0 = clean,
* 1 = infected (rejected), anything else = scanner error (not admitted now).
* ========================================================================== */
import { spawn } from 'node:child_process';
export function scanFile(path, cmd) {
const parts = String(cmd).split(/\s+/).filter(Boolean);
return new Promise((resolve) => {
let child;
try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
catch (e) { resolve({ result: 'error', detail: e.message }); return; }
let out = '';
child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
child.on('close', (code) => resolve(
code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
));
});
}
const CID_RE = /^[0-9a-f]{64}$/;
// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
// → { ok: true, scan } | { ok: false, reason }
export async function admitFile(info, deps) {
const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
let scanResult = 'skipped';
if (cfg.malwareScan) {
const r = await scan(info.path, cfg.scanCmd);
if (r.result !== 'clean') {
log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
return { ok: false, reason: 'scan ' + r.result };
}
scanResult = 'clean';
}
await upsertContent({
cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
scan: scanResult, now: now(),
});
return { ok: true, scan: scanResult };
}
```
## Appendix B — server/p2p-admit.test.js
```js
import { test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { admitFile, scanFile } from './p2p-admit.js';
import { sha256File, sha256Range } from './hash.js';
const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
const file = join(dir, 'f.bin');
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const quiet = { warn() {}, info() {} };
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };
test('sha256File / sha256Range match node:crypto', async () => {
expect(await sha256File(file)).toBe(CID);
const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
expect(await sha256Range(file, 1000, 65536)).toBe(want);
});
test('scan off by default: admitted with scan=skipped', async () => {
const rows = [];
const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
expect(r).toEqual({ ok: true, scan: 'skipped' });
expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
});
test('scan on: clean admits, infected and scanner errors do not', async () => {
for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
const rows = [];
const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
expect(r.ok).toBe(ok);
expect(rows.length).toBe(ok ? 1 : 0);
}
});
test('disabled P2P or a malformed cid never admits', async () => {
const rows = [];
expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect(rows.length).toBe(0);
});
test('scanFile maps exit codes', async () => {
expect((await scanFile(file, 'true')).result).toBe('clean');
expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
});
```

168
plans/queue/010-views-and-retention-d0c6ca.md Normal file
View File

@@ -0,0 +1,168 @@
---
id: 010-views-and-retention-d0c6ca
title: Count views and evict server copies by retention criteria before LRU
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f]
est_files: 5
---
# 010 — Views + retention-ordered eviction
## Objective
Implements flow 9 of `docs/p2p-architecture.md`. After this plan:
- Every play (`GET /api/streams`) and save (`GET /api/download/:id`) counts one view in
`video_views` (deduped per client + video for 30 min; warm-ups don't count).
- When the media cache needs room it evicts copies that are neither **top**
(≥ `P2P_KEEP_MIN_VIEWS` views in `P2P_KEEP_DAYS`) nor **recent** (played within
`P2P_KEEP_RECENT_DAYS`) first — fewest views, then oldest — and only then qualifying
ones by LRU. Budget, disk guard and 10-min play protection are unchanged.
- Eviction never touches `p2p_content` / holders (the catalog keeps growing).
The query + media-cache change were tested ahead of time (retention test 1/1,
media-cache 26/26).
## Context the executor must NOT rediscover
- `plans/patches/010-retention-eviction.diff` patches `server/media-cache.js` (`makeRoom` uses
`db.listMediaEvictionOrder()` when present, else `db.listMediaLru()`) and adds one test to
`server/media-cache.test.js`. It applies on top of plan 009's patches.
- `server/p2p-db.js` (plan 008) has `dayKey`, `rowsOf`, `addView(videoId, now)`.
- `server/server.js` `createMediaCache({ dir: MEDIA_DIR, db: { getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats }, …` (~line 983).
- `app.get('/api/streams', async (c) => {` (~line 532): first lines validate `videoId` and
return 400 when empty. `app.get('/api/download/:videoId', async (c) => {` (~line 1311) does the same.
- Plan 009 imported `P2P` and `* as p2pDb` in server.js.
## Steps
1. `server/p2p-db.js` — append at the end of the file:
```js
// ---- retention (plan 010) -------------------------------------------------------
// Server copies in the order they should be evicted when the cache needs room:
// first the ones that are neither "top" (≥ keepMinViews views in keepDays) nor
// "recent" (played within keepRecentDays) — fewest views, then oldest — and
// only then the qualifying ones, least recently played first.
export async function listMediaEvictionOrder({ now, keepMinViews, keepDays, keepRecentDays }) {
const r = await db.execute({
sql: `SELECT m.video_id, m.size, m.last_access,
COALESCE((SELECT SUM(v.n) FROM video_views v
WHERE v.video_id = m.video_id AND v.day >= ?), 0) AS views
FROM media_cache m WHERE m.status = 'ready'`,
args: [dayKey(now - keepDays * 86400_000)],
});
const recentCut = now - keepRecentDays * 86400_000;
const rows = rowsOf(r).map((x) => ({
...x, views: Number(x.views) || 0,
qualifies: (Number(x.views) || 0) >= keepMinViews || Number(x.last_access) >= recentCut,
}));
rows.sort((a, b) => (a.qualifies - b.qualifies)
|| (a.qualifies ? 0 : a.views - b.views)
|| (a.last_access - b.last_access));
return rows;
}
```
2. Create `server/p2p-retention.test.js` — copy VERBATIM from the Appendix.
3. `server/package.json` "test" script — append ` && bun test ./p2p-retention.test.js`.
4. From the repo root: `git apply plans/patches/010-retention-eviction.diff` (STOP and report on failure).
5. `server/server.js` `createMediaCache({ … db: { … } …` — replace the `db:` line with:
```js
db: {
getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats,
// Retention (docs/p2p-architecture.md flow 9): cold copies go before popular ones.
listMediaEvictionOrder: () => p2pDb.listMediaEvictionOrder({
now: Date.now(), keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
}),
},
```
6. `server/server.js` — directly above `app.get('/api/streams', …)` (and above the plan-005
warm route if present) add:
```js
// One view per client per video per 30 min (a play and its save count once).
const viewSeen = new Map(); // `${who}|${id}` -> ms
function countView(c, videoId) {
const who = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || c.req.query('fp') || 'local';
const key = who + '|' + videoId;
const now = Date.now();
if (now - (viewSeen.get(key) || 0) < 30 * 60_000) return;
viewSeen.set(key, now);
if (viewSeen.size > 20000) viewSeen.clear();
p2pDb.addView(videoId, now).catch(() => {});
}
```
7. In `/api/streams`, directly after `if (!videoId) return c.json({ ok: false, error: 'missing videoId' }, 400);`
add `countView(c, videoId);`. Do the same in `/api/download/:videoId` after its
`missing videoId` guard.
## Out of scope / do NOT touch
- `MEDIA_CACHE_MAX_BYTES`, `EVICT_PROTECT_MS`, the disk guard, `listMediaLru` itself.
- Do not delete server files on a timer — eviction stays budget-driven.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-retention.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "countView(c, videoId)" server.js
```
Expected: `1 pass`; media-cache `26 pass 0 fail`; every file `0 fail`; `SERVER_OK`; grep `2`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — server/p2p-retention.test.js
```js
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-retention-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const DAY = 86400_000;
const NOW = Date.UTC(2026, 8, 29, 12);
const opts = { now: NOW, keepMinViews: 3, keepDays: 30, keepRecentDays: 14 };
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
const add = (id, lastAccessDaysAgo) => dbmod.upsertMedia(id, { status: 'ready', size: 100, last_access: NOW - lastAccessDaysAgo * DAY });
await add('topOld00001', 60); // 5 views in window → top, but old
await add('recent00001', 2); // 0 views, played 2 days ago → recent
await add('cold0000001', 40); // 1 view in window, old → evict first-ish
await add('cold0000002', 20); // 0 views, 20 days → evict first
await add('cold0000003', 50); // 0 views, 50 days → evict first (older than cold2)
for (let i = 0; i < 5; i++) await P.addView('topOld00001', NOW - 5 * DAY);
await P.addView('cold0000001', NOW - 3 * DAY);
await P.addView('cold0000002', NOW - 45 * DAY); // outside the 30-day window
});
test('non-qualifying copies go first (fewest views, then oldest); qualifying ones by LRU', async () => {
const order = (await P.listMediaEvictionOrder(opts)).map((r) => [r.video_id, r.qualifies]);
expect(order).toEqual([
['cold0000003', false],
['cold0000002', false],
['cold0000001', false],
['topOld00001', true],
['recent00001', true],
]);
});
```

242
plans/queue/011-browser-sha256-e1793d.md Normal file
View File

@@ -0,0 +1,242 @@
---
id: 011-browser-sha256-e1793d
title: Add an incremental SHA-256 library for the browser and node tests
created: 2026-09-29
depends_on: []
est_files: 5
---
# 011 — Incremental SHA-256 for the browser
## Objective
Devices must compute a file's content id (SHA-256) while bytes stream past —
during a download, a peer transfer, or a chunked OPFS read — without holding the
file in memory. WebCrypto's `digest()` needs the whole input at once, so add a
small pure-JS incremental hasher, usable from the page, from Web Workers
(`importScripts('/sha256.js')`) and from node tests. Pre-tested: all vectors and
every length 0–200 match `node:crypto`; random chunking of a 3 MiB buffer matches;
~116 MB/s in Node.
## Context the executor must NOT rediscover
- Module style to copy: `frontend/stats-core.js` — an IIFE `(function (root) { … })(typeof globalThis !== 'undefined' ? globalThis : this);`
that sets `module.exports` under node and `root.StatsCore` in the browser. Tests are
CommonJS `node:test` (`frontend/stats-core.test.js`).
- `frontend/index.html:562-569` loads scripts in this order:
```html
<script src="fingerprint.js"></script>
<script src="opfs.js"></script>
<script src="video-edit.js"></script>
<script src="lyrics-core.js"></script>
<script src="stats-core.js"></script>
<script src="async-guard.js"></script>
<script src="sw-update.js"></script>
<script src="app.js"></script>
```
- `frontend/sw.js` `SHELL` array (~line 55) must list every shell file.
## Steps
1. Create `frontend/sha256.js` — copy VERBATIM from Appendix A.
2. Create `frontend/sha256.test.js` — copy VERBATIM from Appendix B.
3. `frontend/index.html` — add ` <script src="sha256.js"></script>` directly after the
`stats-core.js` script line.
4. `frontend/sw.js` `SHELL` — add `'/sha256.js',` directly after `'/stats-core.js',`.
5. `CLAUDE.md` "Testing" section — in the list `(sw, sw-update, async-guard, video-edit, lyrics-core, stats-core)`
add `, sha256`.
## Out of scope / do NOT touch
- No callers yet (plans 012/013/017 use it). Do not touch `app.js`.
## Verification
```bash
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
node -e "const S=require('./frontend/sha256');console.log(S.hex(new TextEncoder().encode('abc')))"
grep -c "sha256.js" frontend/index.html frontend/sw.js
```
Expected: `# fail 0`; `ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad`; each grep `1`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — frontend/sha256.js
```js
/* ============================================================================
* sha256.js — incremental SHA-256 (pure JS; window.Sha256 / worker / node)
*
* WebCrypto's digest() needs the whole input at once, which would pull a
* multi-hundred-MB video into memory. This hasher takes chunks as they stream
* past (downloads, peer transfers, OPFS reads) and keeps ~100 bytes of state.
*
* const h = Sha256.create(); h.update(u8); …; const hex = h.hex();
* Sha256.hex(u8) // one-shot convenience
* ========================================================================== */
(function (root) {
'use strict';
const K = new Uint32Array([
0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2,
]);
function create() {
const H = new Uint32Array([
0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19,
]);
const W = new Uint32Array(64);
const block = new Uint8Array(64);
let blockLen = 0;
let total = 0; // bytes hashed so far (safe to 2^53)
let done = false;
function compress(buf, off) {
for (let i = 0; i < 16; i++) {
const j = off + i * 4;
W[i] = (buf[j] << 24) | (buf[j + 1] << 16) | (buf[j + 2] << 8) | buf[j + 3];
}
for (let i = 16; i < 64; i++) {
const w15 = W[i - 15], w2 = W[i - 2];
const s0 = ((w15 >>> 7) | (w15 << 25)) ^ ((w15 >>> 18) | (w15 << 14)) ^ (w15 >>> 3);
const s1 = ((w2 >>> 17) | (w2 << 15)) ^ ((w2 >>> 19) | (w2 << 13)) ^ (w2 >>> 10);
W[i] = (W[i - 16] + s0 + W[i - 7] + s1) | 0;
}
let a = H[0], b = H[1], c = H[2], d = H[3], e = H[4], f = H[5], g = H[6], h = H[7];
for (let i = 0; i < 64; i++) {
const S1 = ((e >>> 6) | (e << 26)) ^ ((e >>> 11) | (e << 21)) ^ ((e >>> 25) | (e << 7));
const ch = (e & f) ^ (~e & g);
const t1 = (h + S1 + ch + K[i] + W[i]) | 0;
const S0 = ((a >>> 2) | (a << 30)) ^ ((a >>> 13) | (a << 19)) ^ ((a >>> 22) | (a << 10));
const maj = (a & b) ^ (a & c) ^ (b & c);
const t2 = (S0 + maj) | 0;
h = g; g = f; f = e; e = (d + t1) | 0;
d = c; c = b; b = a; a = (t1 + t2) | 0;
}
H[0] = (H[0] + a) | 0; H[1] = (H[1] + b) | 0; H[2] = (H[2] + c) | 0; H[3] = (H[3] + d) | 0;
H[4] = (H[4] + e) | 0; H[5] = (H[5] + f) | 0; H[6] = (H[6] + g) | 0; H[7] = (H[7] + h) | 0;
}
function update(data) {
if (done) throw new Error('sha256: update() after digest');
const u8 = data instanceof Uint8Array ? data : new Uint8Array(data);
let i = 0;
total += u8.length;
if (blockLen) {
const take = Math.min(64 - blockLen, u8.length);
block.set(u8.subarray(0, take), blockLen);
blockLen += take;
i = take;
if (blockLen === 64) { compress(block, 0); blockLen = 0; }
}
for (; i + 64 <= u8.length; i += 64) compress(u8, i);
if (i < u8.length) { block.set(u8.subarray(i), 0); blockLen = u8.length - i; }
return api;
}
function digest() {
if (!done) {
done = true;
const bits = total * 8;
block[blockLen++] = 0x80;
if (blockLen > 56) { block.fill(0, blockLen); compress(block, 0); blockLen = 0; }
block.fill(0, blockLen, 56);
const hi = Math.floor(bits / 0x100000000), lo = bits >>> 0;
block[56] = hi >>> 24; block[57] = hi >>> 16; block[58] = hi >>> 8; block[59] = hi;
block[60] = lo >>> 24; block[61] = lo >>> 16; block[62] = lo >>> 8; block[63] = lo;
compress(block, 0);
}
const out = new Uint8Array(32);
for (let i = 0; i < 8; i++) {
out[i * 4] = H[i] >>> 24; out[i * 4 + 1] = H[i] >>> 16; out[i * 4 + 2] = H[i] >>> 8; out[i * 4 + 3] = H[i];
}
return out;
}
function hex() {
let s = '';
for (const b of digest()) s += (b < 16 ? '0' : '') + b.toString(16);
return s;
}
const api = { update, digest, hex, get bytes() { return total; } };
return api;
}
const Sha256 = {
create,
hex: (data) => create().update(data).hex(),
isHex: (s) => typeof s === 'string' && /^[0-9a-f]{64}$/.test(s),
};
if (typeof module !== 'undefined' && module.exports) module.exports = Sha256;
else root.Sha256 = Sha256;
})(typeof globalThis !== 'undefined' ? globalThis : this);
```
## Appendix B — frontend/sha256.test.js
```js
'use strict';
const { test } = require('node:test');
const assert = require('node:assert');
const crypto = require('node:crypto');
const Sha256 = require('./sha256');
const ref = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
test('known vectors', () => {
assert.strictEqual(Sha256.hex(new Uint8Array(0)), 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855');
assert.strictEqual(Sha256.hex(new TextEncoder().encode('abc')), 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad');
assert.strictEqual(
Sha256.hex(new TextEncoder().encode('abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq')),
'248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1');
});
test('every length around block boundaries matches node:crypto', () => {
for (let n = 0; n <= 200; n++) {
const buf = crypto.randomBytes(n);
assert.strictEqual(Sha256.hex(new Uint8Array(buf)), ref(buf), 'length ' + n);
}
});
test('random chunk boundaries give the same digest as one shot', () => {
const buf = crypto.randomBytes(3 * 1024 * 1024 + 17);
for (let round = 0; round < 5; round++) {
const h = Sha256.create();
let pos = 0;
while (pos < buf.length) {
const n = Math.min(buf.length - pos, 1 + Math.floor(Math.random() * 200000));
h.update(new Uint8Array(buf.buffer, buf.byteOffset + pos, n));
pos += n;
}
assert.strictEqual(h.hex(), ref(buf));
assert.strictEqual(h.bytes, buf.length);
}
});
test('update after digest throws; isHex', () => {
const h = Sha256.create();
h.hex();
assert.throws(() => h.update(new Uint8Array(1)));
assert.ok(Sha256.isHex(ref(Buffer.from('x'))));
assert.ok(!Sha256.isHex('ABC'));
});
```

215
plans/queue/012-device-file-registry-288d55.md Normal file
View File

@@ -0,0 +1,215 @@
---
id: 012-device-file-registry-288d55
title: Add the on-device IndexedDB file registry and hash saves while downloading
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f, 011-browser-sha256-e1793d]
est_files: 6
---
# 012 — On-device file registry + hash while saving
## Objective
Implements flow 2 of `docs/p2p-architecture.md`. After this plan:
- `frontend/device-db.js` (`window.DeviceDB`) keeps one IndexedDB record per saved
video: `{ videoId, cid, size, savedAt, lastCheckedAt, state }`.
- The OPFS download worker hashes bytes as it writes them. If the server sent
`X-Content-SHA256` (plan 009) and the hash differs, the save FAILS and the partial
file is deleted ("integrity check failed").
- `preload()` records the file (`state: 'verified'` when the hashes matched,
`'unverified'` when the server sent none, `'unhashed'` for the main-thread fallback);
deleting / clearing saved videos removes the records.
Pre-tested in Chromium with the harness in `plans/harness/` (good hash → saved and
`verified`; wrong hash → rejected, no file left; no header → saved with a local hash).
## Context the executor must NOT rediscover
- `plans/patches/012-opfs-hash.diff` patches `frontend/opfs-worker.js` (importScripts
`/sha256.js`, hash in the write loop, compare with `x-content-sha256`, post
`{ type:'done', ext, sha256, expectedSha, size }`) and `frontend/opfs.js`
(`downloadVideo` resolves `{ ok:true, sha256, expectedSha, size }`).
- `frontend/app.js:157-198` `async function opfsDownload(videoId, { mux = false } = {})` — worker path:
```js
if (typeof window.OPFS.downloadVideo === 'function' && typeof Worker !== 'undefined') {
const w = await window.OPFS.downloadVideo(videoId, url);
if (w.ok) return { ok: true, cached: true };
workerError = w.error || null;
}
```
- `frontend/app.js:258-268`:
```js
async function opfsDelete(videoId) {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.deleteVideo(videoId); } catch { /* ignore */ }
return { ok: true };
}
async function opfsClear() {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.clearAll(); } catch { /* ignore */ }
return { ok: true };
}
```
- `frontend/app.js` `async function preload(video, …)` (~line 1237), success branch:
```js
const res = await API.cacheDownload(id, { mux });
if (res && res.ok && res.cached) {
cachedIds.add(id);
cacheMutations++;
warmThumb(thumbUrlFor(id, video));
```
- Script order in `frontend/index.html` after plan 011: `… stats-core.js, sha256.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` after plan 011 contains `'/sha256.js',`.
## Steps
1. Create `frontend/device-db.js` — copy VERBATIM from the Appendix.
2. From the repo root: `git apply plans/patches/012-opfs-hash.diff` (STOP and report on failure).
3. `frontend/index.html` — add ` <script src="device-db.js"></script>` directly after the `sha256.js` line.
4. `frontend/sw.js` `SHELL` — add `'/device-db.js',` directly after `'/sha256.js',`.
5. `frontend/app.js` `opfsDownload` — the line `if (w.ok) return { ok: true, cached: true };` appears
TWICE in app.js; change ONLY the first one (inside `async function opfsDownload`, ~line 180),
NOT the one inside `opfsDownloadEdited` (edited cuts are never shared). Change it to:
```js
if (w.ok) return { ok: true, cached: true, sha256: w.sha256 || null, expectedSha: w.expectedSha || null, size: w.size || 0 };
```
(The main-thread fallback below keeps returning `{ ok: true, cached: true }` — no hash.)
6. `frontend/app.js` — directly ABOVE `async function preload(video, …)` add:
```js
// This device's record of what it holds and each file's content id
// (docs/p2p-architecture.md flow 2). The P2P client reports these.
function recordDeviceFile(id, res) {
if (!WEB || !window.DeviceDB) return;
const cid = res && window.Sha256 && window.Sha256.isHex(res.sha256) ? res.sha256 : null;
const state = cid ? (res.expectedSha === cid ? 'verified' : 'unverified') : 'unhashed';
const now = Date.now();
window.DeviceDB.putFile({ videoId: id, cid, size: (res && res.size) || 0, savedAt: now, lastCheckedAt: now, state })
.then(() => { if (window.P2PClient) window.P2PClient.changed(); })
.catch(() => {});
}
```
7. `frontend/app.js` `preload` success branch — after `cacheMutations++;` add
`recordDeviceFile(id, res);`
8. `frontend/app.js` `opfsDelete` — after the `try { await window.OPFS.deleteVideo(videoId); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.deleteFile(videoId); if (window.P2PClient) window.P2PClient.changed(); }
```
`opfsClear` — after its `try { await window.OPFS.clearAll(); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.clear(); if (window.P2PClient) window.P2PClient.changed(); }
```
## Out of scope / do NOT touch
- `opfsDownloadEdited` (edited cuts are device-only, never shared).
- No server calls here (plan 013 reports holdings). Do not hash old files here (plan 013).
- Do not migrate `_ytpdata` / playlists to IndexedDB.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/opfs.js frontend/opfs-worker.js frontend/device-db.js && echo SYNTAX_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "device-db.js" frontend/index.html frontend/sw.js
# Browser check (Chromium). Skip with a note in Findings if no Chromium/playwright is available.
cd plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1)
bun device-db-server.js >/tmp/ytp012.log 2>&1 & SRV=$!; sleep 2; timeout 60 node device-db-check.mjs; kill $SRV; true
```
Expected: `SYNTAX_OK`; `# fail 0`; grep `1` each; the browser check prints JSON containing
`"shaOk":true,"goodExpected":true`, `"bad":{"ok":false,"error":"integrity check failed (content hash mismatch)"}`,
`"list":["good","nohash"]`, `"rec":"verified"`, `"after":0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — frontend/device-db.js
```js
/* ============================================================================
* device-db.js — this device's own file registry (IndexedDB "ytp-device")
*
* One record per saved video, next to the bytes in OPFS:
* { videoId, cid, size, savedAt, lastCheckedAt, state }
* cid SHA-256 of the stored file (P2P content id) or null
* state 'verified' hash computed on save and equal to the server's
* 'unverified' hash computed, but the server sent none to compare
* 'unhashed' saved before hashing existed / main-thread fallback
* The P2P client (p2p-client.js) reads this to report holdings; OPFS stays the
* source of truth for what is playable (a record without a file is ignored).
* Every call resolves (null / [] on failure) — private windows can block IDB.
* See docs/p2p-architecture.md.
* ========================================================================== */
(function () {
'use strict';
const DB_NAME = 'ytp-device';
const VERSION = 1;
let _open = null;
function open() {
if (!_open) {
_open = new Promise((resolve, reject) => {
const r = indexedDB.open(DB_NAME, VERSION);
r.onupgradeneeded = () => {
const d = r.result;
if (!d.objectStoreNames.contains('files')) {
const s = d.createObjectStore('files', { keyPath: 'videoId' });
s.createIndex('cid', 'cid', { unique: false });
}
};
r.onsuccess = () => resolve(r.result);
r.onerror = () => reject(r.error);
r.onblocked = () => reject(new Error('device-db blocked'));
});
_open.catch(() => { _open = null; });
}
return _open;
}
const done = (req) => new Promise((resolve, reject) => {
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
async function store(mode) {
const d = await open();
return d.transaction('files', mode).objectStore('files');
}
async function safe(fn, fallback) {
try { return await fn(); } catch { return fallback; }
}
window.DeviceDB = {
isSupported: () => typeof indexedDB !== 'undefined',
putFile: (rec) => safe(async () => {
if (!rec || !rec.videoId) return null;
await done((await store('readwrite')).put({
videoId: String(rec.videoId),
cid: rec.cid || null,
size: Number(rec.size) || 0,
savedAt: Number(rec.savedAt) || Date.now(),
lastCheckedAt: Number(rec.lastCheckedAt) || Date.now(),
state: rec.state || 'unhashed',
}));
return true;
}, null),
getFile: (videoId) => safe(async () => (await done((await store('readonly')).get(String(videoId)))) || null, null),
getByCid: (cid) => safe(async () => (await done((await store('readonly')).index('cid').get(String(cid)))) || null, null),
listFiles: () => safe(async () => (await done((await store('readonly')).getAll())) || [], []),
deleteFile: (videoId) => safe(async () => { await done((await store('readwrite')).delete(String(videoId))); return true; }, null),
clear: () => safe(async () => { await done((await store('readwrite')).clear()); return true; }, null),
};
}());
```

622
plans/queue/013-device-identity-and-holdings-3ba493.md Normal file
View File

@@ -0,0 +1,622 @@
---
id: 013-device-identity-and-holdings-3ba493
title: Register devices and report verified holdings to the server
created: 2026-09-29
depends_on: [010-views-and-retention-d0c6ca, 012-device-file-registry-288d55]
est_files: 11
---
# 013 — Device identity + holdings sync
## Objective
Implements flow 3 of `docs/p2p-architecture.md`. After this plan, with default
settings (P2P ON):
- A device registers once (`POST /api/p2p/device` → `deviceId` + `secret`, kept in
`localStorage.ytpDevice`; only `sha256(secret)` is stored server-side).
- `P2PClient` reconciles `DeviceDB` with the files really in OPFS, hashes old/unhashed
saves in a worker, reports the FULL holdings list, answers the server's range
challenges, and marks accepted files `verified`.
- The server keeps holder rows forever (no TTL): refreshed `last_verified_at` on every
report, `removed` when a full report no longer lists them or a challenge fails.
- Turning sharing off (`data.settings.p2pShare === false`) withdraws every holding.
Pre-tested: server routes 6/6 (`bun:test`), and the whole client flow in Chromium via
`plans/harness/p2p-client-*.js` (ghost record dropped, legacy file hashed, verified
file accepted + challenge passed → `trust: challenged`, share off → 0 holders).
## Context the executor must NOT rediscover
- Plans 008–012 are applied: `server/p2p-db.js`, `p2p-config.js` (`P2P`), `hash.js`
(`sha256Range`), `p2p-admit.js`; `frontend/sha256.js`, `frontend/device-db.js`, and
`opfs.js`/`opfs-worker.js` hash on save. `server/server.js` imports `P2P` and `* as p2pDb`.
- `plans/patches/013-opfs-readrange.diff` adds `OPFS.readRange(videoId, offset, length)` to
`frontend/opfs.js` (applies on top of plan 012's patch).
- `server/server.js`: `const MEDIA_DIR = process.env.MEDIA_DIR || './data/media';` (~line 980).
Plan 009's `const notes = registerNoteRoutes(app, {…});` (~line 1823) is a good neighbourhood
for another `register…Routes(app, …)` call — add ours right after the uploads block
(`const isUpload = (id) => uploads.isUploadId(id);`).
- `frontend/app.js`:
- `const DEFAULT_SETTINGS = {` (~line 326), first line
`quality: 'auto', volume: 1, audioOnly: false, autoPreload: true,`.
- end of `async function boot()` (~line 9822):
```js
// Learn what's already cached, then top up any playlist videos that aren't.
await refreshCachedIds();
startCacheResyncWatch();
```
- `data.profile` is `{ name, syncedAt }` or null.
- Script order in `frontend/index.html` after plan 012:
`… stats-core.js, sha256.js, device-db.js, async-guard.js, sw-update.js, app.js`.
## Steps
1. `server/p2p-db.js`:
a. In `initP2pSchema()`, directly after the `ALTER TABLE media_cache ADD COLUMN sha256` try/catch add:
` await db.execute('CREATE INDEX IF NOT EXISTS idx_media_sha256 ON media_cache (sha256)');`
b. Append at the end of the file:
```js
// The server's own ready copy whose mp4 has this content id (or null).
export async function findMediaByCid(cid) {
const r = await db.execute({
sql: "SELECT video_id, gen FROM media_cache WHERE sha256 = ? AND status = 'ready' LIMIT 1",
args: [cid],
});
return rowsOf(r)[0] || null;
}
```
2. Create `server/p2p-routes.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-routes.test.js` — copy VERBATIM from Appendix B.
4. `server/package.json` "test" script — append ` && bun test ./p2p-routes.test.js`.
5. `server/server.js`:
a. imports: `import { registerP2pRoutes } from './p2p-routes.js';` and add `sha256Range` to an
import from `./hash.js`: `import { sha256Range } from './hash.js';`
b. after `const isUpload = (id) => uploads.isUploadId(id);` add:
```js
// ============================================================================
// Peer-to-peer sharing — devices + holdings (docs/p2p-architecture.md)
// ============================================================================
async function fileForCid(cid) {
const row = await p2pDb.findMediaByCid(cid);
if (!row) return null;
const path = `${MEDIA_DIR}/${row.video_id}.${row.gen}.mp4`;
const f = Bun.file(path);
return (await f.exists()) ? { path, size: f.size } : null;
}
const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });
```
(`p2p` is used by plan 014.)
6. Create `frontend/hash-worker.js` — copy VERBATIM from Appendix C.
7. Create `frontend/p2p-client.js` — copy VERBATIM from Appendix D.
8. From the repo root: `git apply plans/patches/013-opfs-readrange.diff` (STOP on failure).
9. `frontend/index.html` — add ` <script src="p2p-client.js"></script>` directly after the `device-db.js` line.
10. `frontend/sw.js` `SHELL` — add `'/hash-worker.js',` and `'/p2p-client.js',` directly after `'/device-db.js',`.
11. `frontend/app.js` `DEFAULT_SETTINGS` — add a new line after the first line:
```js
p2pShare: true, // P2P (docs/p2p-architecture.md): share my saved videos with other devices — ON by default
p2pReceive: true, // fetch from other devices when YouTube and the server can't — ON by default
```
12. `frontend/app.js` `boot()` — directly after `startCacheResyncWatch();` add:
```js
// Peer-to-peer: report what this device holds (on by default; settings.p2pShare).
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
## Out of scope / do NOT touch
- Presence/websocket, holders endpoint, UI (plans 014/015). Intake of unknown files (016).
- Do not change how saves download or play.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-routes.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js frontend/hash-worker.js frontend/opfs.js frontend/app.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
# Browser check (Chromium + playwright; see plans/harness/device-db-check.mjs header).
cd server
bun ../plans/harness/p2p-client-server.js >/tmp/ytp013.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node p2p-client-check.mjs
kill $SRV; true
```
Expected: routes `6 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; `# fail 0`; browser JSON
`{"accepted":1,"unknown":1,"challenges":1,"recs":[["goodAAAAAAA","verified",true],["legacyAAAAA","unverified",true]],"trust":["challenged"],"afterShareOff":0,"device":true}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files may be summarised as "verbatim from appendix").
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-routes.js
```js
/* ============================================================================
* p2p-routes.js — device registration + holdings (docs/p2p-architecture.md
* flows 3 and 5). Mounted by server.js; every route answers 404 when
* P2P_ENABLED=0.
*
* GET /api/p2p/config { ok, enabled, staleDays }
* POST /api/p2p/device { fingerprint?, profile? } → { ok, deviceId, secret }
* POST /api/p2p/holdings (device) { items:[{cid,size,videoId}], share } → { ok, accepted, unknown, challenges }
* POST /api/p2p/challenge (device) { answers:[{cid,offset,length,sha256}] } → { ok, passed, failed }
* GET /api/p2p/holders?v=<id>|cid= availability — added by plan 014
*
* Device auth: header `X-Device: <deviceId>.<secret>`; only sha256(secret) is
* stored. A holdings report is always the device's FULL list: anything it held
* before and no longer lists is marked removed. Holder rows never expire by
* time — last_verified_at is refreshed by each report (see the architecture doc).
* ========================================================================== */
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto';
const CID_RE = /^[0-9a-f]{64}$/;
const DEV_RE = /^dev_[0-9a-f]{16}$/;
const MAX_ITEMS = 5000;
const MAX_CHALLENGES = 5;
const CHALLENGE_LEN = 64 * 1024;
const sha = (s) => createHash('sha256').update(String(s)).digest('hex');
const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); };
export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12);
export function registerP2pRoutes(app, deps) {
const { cfg, p2pDb, fileForCid, sha256Range, now = () => Date.now(), log = console } = deps;
const pending = new Map(); // `${deviceId}|${cid}` -> { offset, length, at }
const regLog = new Map(); // ip -> [ms]
const gate = async (c, next) => {
if (!cfg.enabled) return c.json({ ok: false, error: 'p2p disabled' }, 404);
await next();
};
async function deviceOf(c) {
const m = String(c.req.header('x-device') || '').match(/^(dev_[0-9a-f]{16})\.([0-9a-f]{64})$/);
if (!m) return null;
const d = await p2pDb.getDevice(m[1]);
if (!d || !same(d.secret_hash, sha(m[2]))) return null;
return d;
}
const requireDevice = async (c, next) => {
const d = await deviceOf(c);
if (!d) return c.json({ ok: false, error: 'unknown device' }, 401);
c.set('device', d);
await next();
};
app.get('/api/p2p/config', (c) => c.json({ ok: true, enabled: cfg.enabled, staleDays: cfg.staleDays }));
app.post('/api/p2p/device', gate, async (c) => {
const ip = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || 'local';
const t = now();
const recent = (regLog.get(ip) || []).filter((x) => t - x < 3600_000);
if (recent.length >= 20) return c.json({ ok: false, error: 'too many registrations' }, 429);
recent.push(t);
regLog.set(ip, recent);
if (regLog.size > 10000) regLog.clear();
const body = await c.req.json().catch(() => ({}));
const deviceId = 'dev_' + randomBytes(8).toString('hex');
const secret = randomBytes(32).toString('hex');
await p2pDb.createDevice({
deviceId, secretHash: sha(secret),
fingerprint: String(body.fingerprint || '').slice(0, 128) || null,
profile: String(body.profile || '').slice(0, 64) || null,
now: t,
});
return c.json({ ok: true, deviceId, secret });
});
app.post('/api/p2p/holdings', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const share = body.share !== false;
await p2pDb.touchDevice(d.device_id, { now: t, share, profile: String(body.profile || '').slice(0, 64) || undefined });
const raw = Array.isArray(body.items) ? body.items.slice(0, MAX_ITEMS) : [];
const items = share ? raw.filter((x) => x && CID_RE.test(String(x.cid))) : [];
const known = await p2pDb.knownCids(items.map((x) => x.cid));
const accepted = [];
const unknown = [];
for (const it of items) {
if (!known.has(it.cid)) { unknown.push(it.cid); continue; }
await p2pDb.upsertHolder({ cid: it.cid, deviceId: d.device_id, now: t });
accepted.push(it.cid);
}
await p2pDb.removeHoldersExcept({ deviceId: d.device_id, keep: accepted, now: t });
// Spot-check a few holdings against the server's own copy when it has one.
const challenges = [];
for (const cid of accepted) {
if (challenges.length >= MAX_CHALLENGES) break;
const f = await fileForCid(cid);
if (!f || !(f.size > CHALLENGE_LEN)) continue;
const offset = Math.floor(Math.random() * (f.size - CHALLENGE_LEN));
pending.set(d.device_id + '|' + cid, { offset, length: CHALLENGE_LEN, at: t });
challenges.push({ cid, offset, length: CHALLENGE_LEN });
}
if (pending.size > 50000) pending.clear();
return c.json({ ok: true, accepted, unknown, challenges });
});
app.post('/api/p2p/challenge', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const passed = [];
const failed = [];
for (const a of (Array.isArray(body.answers) ? body.answers : []).slice(0, MAX_CHALLENGES)) {
const key = d.device_id + '|' + a.cid;
const p = pending.get(key);
if (!p || p.offset !== a.offset || p.length !== a.length || t - p.at > 10 * 60_000) continue;
pending.delete(key);
const f = await fileForCid(a.cid);
if (!f) continue;
const want = await sha256Range(f.path, p.offset, p.length);
if (String(a.sha256) === want) {
await p2pDb.setHolderTrust({ cid: a.cid, deviceId: d.device_id, trust: 'challenged', now: t });
passed.push(a.cid);
} else {
await p2pDb.removeHolder({ cid: a.cid, deviceId: d.device_id, now: t });
failed.push(a.cid);
log.warn?.(`[p2p] ${d.device_id} failed challenge for ${a.cid.slice(0, 12)}`);
}
}
return c.json({ ok: true, passed, failed });
});
return { deviceOf, peerIdOf, requireDevice, gate };
}
```
## Appendix B — server/p2p-routes.test.js
```js
// Device registration, holdings reports and range challenges (plan 013).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { Hono } from 'hono';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-routes-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const p2pDb = await import('./p2p-db.js');
const { registerP2pRoutes } = await import('./p2p-routes.js');
const { sha256Range } = await import('./hash.js');
const file = join(root, 'copy.mp4');
const bytes = Buffer.from(Array.from({ length: 400000 }, (_, i) => (i * 13) % 256));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const OTHER = 'c'.repeat(64); // verified but the server has no file
const UNKNOWN = 'd'.repeat(64); // not in p2p_content
const quiet = { warn() {}, info() {} };
let app;
let enabled = true;
const req = (path, { method = 'GET', body, dev } = {}) => app.request(path, {
method,
headers: { 'Content-Type': 'application/json', ...(dev ? { 'X-Device': dev.deviceId + '.' + dev.secret } : {}) },
body: body ? JSON.stringify(body) : undefined,
});
beforeAll(async () => {
await dbmod.initDb();
await p2pDb.initP2pSchema();
await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server', now: 1 });
await p2pDb.upsertContent({ cid: OTHER, videoId: 'dQw4w9WgXcQ', size: 10, origin: 'server', now: 1 });
app = new Hono();
registerP2pRoutes(app, {
cfg: { get enabled() { return enabled; }, staleDays: 7 },
p2pDb,
fileForCid: async (cid) => (cid === CID ? { path: file, size: bytes.length } : null),
sha256Range,
log: quiet,
});
});
async function newDevice() {
const r = await req('/api/p2p/device', { method: 'POST', body: { fingerprint: 'fp1' } });
expect(r.status).toBe(200);
const j = await r.json();
expect(j.deviceId).toMatch(/^dev_[0-9a-f]{16}$/);
expect(j.secret).toMatch(/^[0-9a-f]{64}$/);
return j;
}
test('config is public and says enabled + staleDays', async () => {
expect(await (await req('/api/p2p/config')).json()).toEqual({ ok: true, enabled: true, staleDays: 7 });
});
test('holdings need a valid device secret', async () => {
const dev = await newDevice();
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] } })).status).toBe(401);
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] }, dev: { ...dev, secret: 'e'.repeat(64) } })).status).toBe(401);
});
test('report accepts verified cids, returns unknown ones, and challenges the server-held file', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: true, items: [
{ cid: CID, size: bytes.length, videoId: 'dQw4w9WgXcQ' }, { cid: OTHER, size: 10 }, { cid: UNKNOWN, size: 5 }, { cid: 'nothex' },
] } })).json();
expect(r.accepted.sort()).toEqual([CID, OTHER].sort());
expect(r.unknown).toEqual([UNKNOWN]);
expect(r.challenges.length).toBe(1);
const ch = r.challenges[0];
expect(ch.cid).toBe(CID);
const good = createHash('sha256').update(bytes.subarray(ch.offset, ch.offset + ch.length)).digest('hex');
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: good }] } })).json();
expect(a).toEqual({ ok: true, passed: [CID], failed: [] });
const hs = await p2pDb.listHolders(CID);
expect(hs.find((h) => h.device_id === dev.deviceId).trust).toBe('challenged');
});
test('a wrong challenge answer removes the holder; a replayed answer is ignored', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID, size: bytes.length }] } })).json();
const ch = r.challenges[0];
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(a.failed).toEqual([CID]);
expect((await p2pDb.listHolders(CID)).some((h) => h.device_id === dev.deviceId)).toBe(false);
const again = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(again).toEqual({ ok: true, passed: [], failed: [] });
});
test('a full report without an item removes it; share=false withdraws everything', async () => {
const dev = await newDevice();
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID }, { cid: OTHER }] } });
expect((await p2pDb.activeHoldingsOf(dev.deviceId)).sort()).toEqual([CID, OTHER].sort());
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([OTHER]);
await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: false, items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([]);
expect((await p2pDb.getDevice(dev.deviceId)).share).toBe(0);
});
test('P2P_ENABLED=0 → routes answer 404 (config still says disabled)', async () => {
enabled = false;
try {
expect((await req('/api/p2p/device', { method: 'POST', body: {} })).status).toBe(404);
expect((await (await req('/api/p2p/config')).json()).enabled).toBe(false);
} finally { enabled = true; }
});
```
## Appendix C — frontend/hash-worker.js
```js
/* ============================================================================
* hash-worker.js — SHA-256 of a file already saved in OPFS, off the main
* thread, read in 4 MiB slices (never the whole video in memory).
*
* In: { name } file name under OPFS videos/ (e.g. "abc.mp4")
* Out: { ok: true, sha256, size } | { ok: false, error }
* ========================================================================== */
'use strict';
importScripts('/sha256.js');
self.onmessage = async (e) => {
const { name } = e.data || {};
try {
const root = await navigator.storage.getDirectory();
const dir = await root.getDirectoryHandle('videos');
const file = await (await dir.getFileHandle(String(name))).getFile();
const h = self.Sha256.create();
const STEP = 4 * 1024 * 1024;
for (let pos = 0; pos < file.size; pos += STEP) {
h.update(new Uint8Array(await file.slice(pos, pos + STEP).arrayBuffer()));
}
self.postMessage({ ok: true, sha256: h.hex(), size: file.size });
} catch (err) {
self.postMessage({ ok: false, error: err && err.message ? err.message : String(err) });
}
};
```
## Appendix D — frontend/p2p-client.js
```js
/* ============================================================================
* p2p-client.js — this device's side of peer-to-peer sharing
* (docs/p2p-architecture.md flows 2–3). window.P2PClient.
*
* start({ getSettings, getProfile }) called once from app.js boot()
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
*
* What it does, in order, each sync:
* 1. registers the device once (localStorage ytpDevice)
* 2. reconciles DeviceDB with the files really in OPFS (a record without a
* file is dropped; a file without a record is added as 'unhashed')
* 3. hashes 'unhashed' files and files not re-checked for 30 days, one at a
* time in hash-worker.js
* 4. reports the FULL holdings list (empty when sharing is off), answers the
* server's range challenges, and marks accepted files 'verified'
* P2P is ON by default: sharing runs unless settings.p2pShare === false or the
* server says P2P is disabled.
* ========================================================================== */
(function () {
'use strict';
const KEY = 'ytpDevice';
const REHASH_MS = 30 * 24 * 3600_000;
const RESYNC_MS = 6 * 3600_000;
let hooks = { getSettings: () => ({}), getProfile: () => '' };
let serverCfg = null;
let running = null;
let again = false;
let timer = null;
let lastSync = 0;
function device() {
try {
const d = JSON.parse(localStorage.getItem(KEY) || 'null');
return d && /^dev_[0-9a-f]{16}$/.test(d.deviceId) && /^[0-9a-f]{64}$/.test(d.secret) ? d : null;
} catch { return null; }
}
const authHeaders = () => { const d = device(); return d ? { 'X-Device': d.deviceId + '.' + d.secret } : {}; };
async function config() {
if (serverCfg) return serverCfg;
try {
const j = await (await fetch('/api/p2p/config')).json();
if (j && j.ok) serverCfg = j;
} catch { /* offline — try again next sync */ }
return serverCfg;
}
async function ensureDevice() {
const have = device();
if (have) return have;
const r = await fetch('/api/p2p/device', {
method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ fingerprint: window.getFingerprint ? window.getFingerprint() : '', profile: hooks.getProfile() || '' }),
});
const j = await r.json();
if (!j || !j.ok) throw new Error((j && j.error) || 'device registration failed');
const d = { deviceId: j.deviceId, secret: j.secret };
try { localStorage.setItem(KEY, JSON.stringify(d)); } catch { /* storage blocked */ }
return d;
}
function hashInWorker(name) {
return new Promise((resolve) => {
let w;
try { w = new Worker('/hash-worker.js'); } catch { resolve(null); return; }
w.onmessage = (e) => { w.terminate(); resolve(e.data && e.data.ok ? e.data : null); };
w.onerror = () => { w.terminate(); resolve(null); };
w.postMessage({ name });
});
}
async function sha256Range(videoId, offset, length) {
const bytes = window.OPFS && window.OPFS.readRange ? await window.OPFS.readRange(videoId, offset, length) : null;
if (!bytes) return null;
return window.Sha256.hex(bytes);
}
// DeviceDB ⇄ OPFS. Returns the records that describe a real file.
async function reconcile() {
const files = await window.OPFS.listVideos({ strict: true });
const byId = new Map(files.map((f) => [f.id, f]));
const recs = await window.DeviceDB.listFiles();
const out = [];
for (const r of recs) {
const f = byId.get(r.videoId);
if (!f) { await window.DeviceDB.deleteFile(r.videoId); continue; }
if (f.size !== r.size && r.size) { r.cid = null; r.state = 'unhashed'; r.size = f.size; await window.DeviceDB.putFile(r); }
r.name = f.name;
out.push(r);
byId.delete(r.videoId);
}
for (const f of byId.values()) {
if (String(f.id).startsWith('edit_')) continue; // edited cuts are never shared
const r = { videoId: f.id, cid: null, size: f.size, savedAt: Date.now(), lastCheckedAt: 0, state: 'unhashed' };
await window.DeviceDB.putFile(r);
out.push({ ...r, name: f.name });
}
return out;
}
async function hashPending(recs) {
const now = Date.now();
for (const r of recs) {
if (r.state !== 'unhashed' && now - (r.lastCheckedAt || 0) < REHASH_MS) continue;
const h = await hashInWorker(r.name);
if (!h) continue;
const changedCid = h.sha256 !== r.cid;
r.cid = h.sha256;
r.size = h.size;
r.lastCheckedAt = Date.now();
if (changedCid || r.state === 'unhashed') r.state = 'unverified';
await window.DeviceDB.putFile(r);
}
}
async function report(recs, dev) {
const share = hooks.getSettings().p2pShare !== false;
const items = recs.filter((r) => r.cid).map((r) => ({ cid: r.cid, size: r.size, videoId: r.videoId }));
const r = await fetch('/api/p2p/holdings', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ share, items: share ? items : [], profile: hooks.getProfile() || '' }),
});
if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; }
const j = await r.json();
if (!j || !j.ok) return null;
const accepted = new Set(j.accepted || []);
for (const rec of recs) {
if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); }
}
const answers = [];
for (const ch of j.challenges || []) {
const rec = recs.find((x) => x.cid === ch.cid);
if (!rec) continue;
const hex = await sha256Range(rec.videoId, ch.offset, ch.length);
if (hex) answers.push({ ...ch, sha256: hex });
}
if (answers.length) {
await fetch('/api/p2p/challenge', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ answers }),
}).catch(() => {});
}
return j;
}
async function syncOnce() {
if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null;
if (navigator.onLine === false) return null;
const cfg = await config();
if (!cfg || !cfg.enabled) return null;
const dev = await ensureDevice();
const recs = await reconcile();
await hashPending(recs);
const res = await report(recs, dev);
lastSync = Date.now();
return res;
}
// One sync at a time; a request during a sync schedules exactly one more.
function sync() {
if (running) { again = true; return running; }
running = syncOnce().catch(() => null).finally(() => {
running = null;
if (again) { again = false; sync(); }
});
return running;
}
function changed() {
clearTimeout(timer);
timer = setTimeout(sync, 5000);
}
function start(h) {
hooks = { ...hooks, ...(h || {}) };
const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1));
setTimeout(() => idle(() => sync()), 8000);
document.addEventListener('visibilitychange', () => {
if (document.visibilityState === 'visible' && Date.now() - lastSync > RESYNC_MS) sync();
});
}
window.P2PClient = { start, changed, sync, device, authHeaders, config };
}());
```

122
plans/queue/014-p2p-presence-hub-ceced8.md Normal file
View File

@@ -0,0 +1,122 @@
---
id: 014-p2p-presence-hub-ceced8
title: Add the /ws/p2p presence and signalling hub and the holders endpoint
created: 2026-09-29
depends_on: [013-device-identity-and-holdings-3ba493]
est_files: 6
---
# 014 — Presence + signalling hub, holders endpoint
## Objective
Implements flows 4–5 of `docs/p2p-architecture.md`. After this plan:
- `/ws/p2p` is a websocket hub (`server/p2p-hub.js`). A device authenticates with its
FIRST message `{type:'auth', device, secret}` (never in the URL — proxies log URLs),
gets `{type:'hello', peer}`, and is **online** while the socket is open (memory only).
It relays `{type:'signal', to, data}` between two authenticated online devices.
- `GET /api/p2p/holders?v=<videoId>` (or `?cid=`) lists each verified cid with its
holders: opaque `peer`, `online`, `lastVerifiedAt`, `stale`, `trust`, plus `serverHas`
and counts. Holder rows are NEVER hidden for age — `stale` only flags them.
Devices with sharing off are not listed. Device ids never appear in the payload.
- `P2PClient` keeps the socket open while sharing or receiving is on (both default ON),
with a 50 s keepalive (the server closes idle sockets after 120 s), and exposes
`onMessage(type, fn)`, `signal(to, data)`, `peer()`, `isConnected()`.
Pre-tested: hub/payload 3/3 (`bun:test`); two real browsers exchanged a signal through a
real Bun server (`plans/harness/presence-*.js`).
## Context the executor must NOT rediscover
- `plans/patches/014-p2p-hub-new.diff` creates `server/p2p-hub.js` (exports `createP2pHub`,
`holdersPayload`, `peerIdOf`) and `server/p2p-hub.test.js`.
- `plans/patches/014-p2p-client-presence.diff` patches `frontend/p2p-client.js` (from plan 013).
- `server/server.js` (~line 1778-1790):
```js
const remote = createRemoteHub({ requireSameNetwork: process.env.REMOTE_SAME_NETWORK === '1' });
const party = createPartyHub();
// Bun allows ONE websocket handler per server: party sockets are tagged
// (ws.data.hub === 'party'), everything else belongs to the remote relay.
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket : remote.websocket);
```
- `server/server.js` `Bun.serve({ … fetch(req, server) {` (~line 1926):
```js
const path = new URL(req.url).pathname;
if (path === '/ws/remote') return remote.upgrade(req, server, clientIpOf(req, server));
if (path === '/ws/party') return party.upgrade(req, server, clientIpOf(req, server));
return app.fetch(req, server);
```
- Plan 013 added in server.js: `async function fileForCid(cid)` and
`const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });`
(`p2p.gate` is a Hono middleware answering 404 when P2P is disabled).
## Steps
1. From the repo root:
```bash
git apply plans/patches/014-p2p-hub-new.diff
git apply plans/patches/014-p2p-client-presence.diff
```
STOP and report if either fails.
2. `server/package.json` "test" script — append ` && bun test ./p2p-hub.test.js`.
3. `server/server.js` — add import `import { createP2pHub, holdersPayload } from './p2p-hub.js';`
4. `server/server.js` — directly after `const p2p = registerP2pRoutes(app, { … });` add:
```js
const p2pHub = createP2pHub({ getDevice: p2pDb.getDevice, enabled: () => P2P.enabled });
// GET /api/p2p/holders?v=<videoId>|cid=<sha256> — who holds a copy. Rows are
// persistent; `stale` flags a holder not re-verified for P2P_STALE_DAYS.
app.get('/api/p2p/holders', p2p.gate, async (c) => {
const v = (c.req.query('v') || '').trim();
const cid = (c.req.query('cid') || '').trim().toLowerCase();
const hasCid = /^[0-9a-f]{64}$/.test(cid);
if (!hasCid && !/^[A-Za-z0-9_-]{6,64}$/.test(v)) return c.json({ ok: false, error: 'missing v or cid' }, 400);
const payload = await holdersPayload({
videoId: v, cid: hasCid ? cid : null, p2pDb, isOnline: p2pHub.isOnline,
staleDays: P2P.staleDays, serverHas: fileForCid,
});
return c.json(payload, 200, { 'Cache-Control': 'no-store' });
});
```
5. `server/server.js` — replace the `pickHub` line with:
```js
// …and P2P sockets are tagged ws.data.hub === 'p2p' (p2p-hub.js).
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket
: ws.data && ws.data.hub === 'p2p' ? p2pHub.websocket : remote.websocket);
```
(`p2pHub` is declared later in the file; that is fine — `pickHub` only runs once sockets exist.)
6. `server/server.js` `Bun.serve` fetch — after the `/ws/party` line add:
` if (path === '/ws/p2p') return p2pHub.upgrade(req, server);`
7. Copy nothing else; `frontend/index.html` / `sw.js` need no change (no new frontend files).
## Out of scope / do NOT touch
- `remote.js`, `party.js` and their behaviour. No UI (plan 015). No file transfer (plan 017).
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/presence-server.js >/tmp/ytp014.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node presence-check.mjs
kill $SRV; true
```
Expected: hub `3 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"peers":[true,true],"distinct":true,"sent":true,"got":[{"from":true,"data":{"hi":1}}]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

184
plans/queue/015-availability-ui-and-settings-3b9397.md Normal file
View File

@@ -0,0 +1,184 @@
---
id: 015-availability-ui-and-settings-3b9397
title: Show peer availability with stale markers and add Sharing settings
created: 2026-09-29
depends_on: [014-p2p-presence-hub-ceced8]
est_files: 6
---
# 015 — Availability line + Sharing settings
## Objective
Implements the UI half of flow 5 and the settings of `docs/p2p-architecture.md`.
After this plan:
- Under the now-playing title a small line shows peer availability, e.g.
`📡 On 3 devices · 1 online now · last checked 2d ago`. Hovering lists each holder
(`peer · online/offline · checked Xd ago (stale) · spot-checked`). When EVERY holder
is older than the server's `staleDays`, the line says `(not checked recently)` and is
styled dimmer — holders are **never hidden for age**. No holders → no line.
- Settings gets a "Sharing (peer-to-peer)" group: **Share my saved videos** and
**Get videos from other devices**, both ON by default, plus a status row
(registered / online peer id, verified/saved counts).
Pre-tested: `frontend/p2p-core.test.js` 5/5.
## Context the executor must NOT rediscover
- `plans/patches/015-p2p-core-new.diff` creates `frontend/p2p-core.js` (`window.P2PCore`:
`ago`, `formatAvailability(payload, now)`, `pickPeers(payload, cid)`) and its node test.
- `frontend/index.html:223-227`:
```html
<div id="nowPlayingMeta" class="now-meta hidden">
<div class="np-text">
<div class="np-title" id="npTitle"></div>
<div class="np-channel" id="npChannel"></div>
</div>
```
- `frontend/app.js` `function updateNowPlayingActions()` (~line 2389) begins
`if (!current || !current.meta) return; const id = current.meta.id;` and ends with the
save-button `if/else` chain followed by `}`.
- `frontend/app.js` `async function renderSettings()` (~line 7321). Its template contains:
```html
<div class="set-group">
<div class="set-group-title">Offline cache</div>
```
and further down the handlers:
```js
$('setAutoPreload').addEventListener('change', (e) => {
data.settings.autoPreload = e.target.checked;
persist();
if (e.target.checked) data.playlists.forEach(preloadPlaylist);
});
```
- `frontend/styles.css:737` — `.np-actions { display: flex; gap: 8px; flex-shrink: 0; }`;
the dim text colour token is `var(--text-dim)`.
- `window.P2PClient` (plans 013/014): `device()`, `isConnected()`, `peer()`, `sync()`.
`window.DeviceDB.listFiles()` (plan 012).
## Steps
1. From the repo root: `git apply plans/patches/015-p2p-core-new.diff` (STOP on failure).
2. `frontend/index.html` — after `<div class="np-channel" id="npChannel"></div>` add:
` <div class="np-avail hidden" id="npAvail" aria-live="polite"></div>`
and add ` <script src="p2p-core.js"></script>` directly after the `p2p-client.js` script line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-core.js',` directly after `'/p2p-client.js',`.
4. `frontend/styles.css` — directly after the `.np-actions { … }` line add:
```css
.np-avail { font-size: 12px; color: var(--text-dim); margin-top: 4px; cursor: default; }
.np-avail.stale { opacity: 0.6; font-style: italic; }
```
5. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` add:
```js
// P2P availability under the title (docs/p2p-architecture.md flow 5).
// Cached 60 s per video so the frequent updateNowPlayingActions() calls
// don't hammer the server.
const availCache = new Map(); // videoId -> { at, payload }
async function renderAvailability(id) {
const el = $('npAvail');
if (!el) return;
const off = data.settings.p2pShare === false && data.settings.p2pReceive === false;
if (!WEB || !window.P2PCore || off || navigator.onLine === false || String(id).startsWith('edit_')) {
el.classList.add('hidden');
return;
}
let hit = availCache.get(id);
if (!hit || Date.now() - hit.at > 60_000) {
hit = { at: Date.now(), payload: null };
availCache.set(id, hit);
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
hit.payload = r.ok ? await r.json() : null;
} catch { /* offline / disabled — no line */ }
if (availCache.size > 200) availCache.delete(availCache.keys().next().value);
}
if (!current || !current.meta || current.meta.id !== id) return; // moved on meanwhile
const f = window.P2PCore.formatAvailability(hit.payload, Date.now());
if (!f) { el.classList.add('hidden'); return; }
el.textContent = f.text;
el.title = f.title;
el.classList.toggle('stale', f.stale);
el.classList.remove('hidden');
}
```
6. `frontend/app.js` `updateNowPlayingActions` — directly after `const id = current.meta.id;` add
`renderAvailability(id);`
7. `frontend/app.js` `renderSettings` template — directly BEFORE the
`<div class="set-group">` whose title is `Offline cache`, insert:
```html
<div class="set-group">
<div class="set-group-title">Sharing (peer-to-peer)</div>
<label class="set-row">
<span>
Share my saved videos
<small>Other devices can download videos you saved when YouTube and the server can't provide them. Files are verified by their hash. On by default.</small>
</span>
<input id="setP2pShare" type="checkbox" ${data.settings.p2pShare !== false ? 'checked' : ''} />
</label>
<label class="set-row">
<span>
Get videos from other devices
<small>When a video is gone from YouTube and the server, download a verified copy from a device that has it. On by default.</small>
</span>
<input id="setP2pReceive" type="checkbox" ${data.settings.p2pReceive !== false ? 'checked' : ''} />
</label>
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
8. `frontend/app.js` `renderSettings` — directly after the `$('setAutoPreload').addEventListener(…);`
block add:
```js
$('setP2pShare').addEventListener('change', (e) => {
data.settings.p2pShare = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync(); // share off → withdraws every holding
});
$('setP2pReceive').addEventListener('change', (e) => {
data.settings.p2pReceive = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync();
});
(async () => {
const info = $('p2pDeviceInfo');
const cnt = $('p2pHoldingCount');
if (!info || !cnt) return;
const P = window.P2PClient;
const d = P && P.device();
info.textContent = !d ? 'Not registered yet'
: P.isConnected() ? `Online · peer ${P.peer()}` : 'Registered · not connected';
const files = window.DeviceDB ? await window.DeviceDB.listFiles() : [];
cnt.textContent = `${files.filter((f) => f.state === 'verified').length} verified / ${files.length} saved`;
})();
```
## Out of scope / do NOT touch
- No download button yet (plan 017 adds "Get from a device"). Do not change `p2p-client.js`.
- Do not show availability for edited cuts (`edit_…`).
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-core.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-core.js" frontend/index.html frontend/sw.js
grep -c "setP2pShare\|setP2pReceive\|renderAvailability(id)" frontend/app.js
```
Expected: `FRONT_OK`; `# fail 0`; `1` and `1`; app.js grep ≥ `5`.
Manual (optional, needs a running server with P2P data): open the app, play a video that has holders,
see the 📡 line; Settings shows the Sharing group with both boxes ticked.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

150
plans/queue/016-intake-and-server-verification-cfe031.md Normal file
View File

@@ -0,0 +1,150 @@
---
id: 016-intake-and-server-verification-cfe031
title: Let a device hand a file to the server for hashing and validation
created: 2026-09-29
depends_on: [015-availability-ui-and-settings-3b9397]
est_files: 9
---
# 016 — Intake: server-side verification of device files
## Objective
Implements flow 7 of `docs/p2p-architecture.md` — the owner's rule "a file must first
be downloaded by the server and checked before its hash is added to the server DB".
After this plan:
- `POST /api/p2p/intake` (device auth) opens a 30-min ticket; `PUT /api/p2p/intake/:ticket`
streams the bytes into `P2P_INTAKE_DIR` (never served). The SERVER hashes while writing,
a claimed cid must match, `validateMedia()` must pass, then `admitFile()` (malware scan
only when `P2P_MALWARE_SCAN=1`, off by default). Only then does the cid enter
`p2p_content` (`origin 'intake'`), and the uploader becomes a holder (`trust 'challenged'`).
- If the server has no copy of that video, the file is adopted into the media cache
byte-for-byte (`media.adoptFile`, new) so it can be streamed again. Otherwise deleted.
- Settings → Sharing shows "Verify & share N saved videos" for saves whose hash the server
doesn't know yet (old saves, fallback-path saves); one tap uploads them one by one.
Nothing uploads automatically in this plan.
Pre-tested: intake 4/4 (valid, hash-mismatch, truncated, non-media, size limits, ticket
reuse, scan-infected), media-cache 27/27 (incl. `adoptFile`), and the browser flow
(unknown → contribute → accepted) in Chromium.
## Context the executor must NOT rediscover
- Patches (apply in this order from the repo root):
- `plans/patches/016-p2p-intake-new.diff` → new `server/p2p-intake.js` (`registerIntakeRoutes`) + `server/p2p-intake.test.js`
- `plans/patches/016-media-cache-adopt.diff` → `adoptFile(id, src, { sha256, probe, meta })` in `server/media-cache.js` + a test
- `plans/patches/016-client-intake.diff` → `P2PClient.contribute(videoId, { cid, title, channel })`,
`P2PClient.unknownVideos()` in `frontend/p2p-client.js`; `OPFS.getFileObject(videoId)` in `frontend/opfs.js`
- `server/server.js` imports `{ createMediaCache, HIGH, LOW } from './media-cache.js'`; `FFMPEG`
const exists; ffprobe path is `process.env.FFPROBE_PATH || 'ffprobe'` (used in createMediaCache).
- Plan 013/014 added in server.js: `const p2p = registerP2pRoutes(app, …)` (exposes `gate`,
`requireDevice`) and `const p2pHub = createP2pHub(…)`. Plan 009 imported `admitFile`, `P2P`, `* as p2pDb`.
- Plan 015 added to `renderSettings()` the Sharing group ending with:
```html
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
and an `(async () => { const info = $('p2pDeviceInfo'); … cnt.textContent = …; })();` block.
- Settings buttons use `class="btn"`. `videoById(id)` (app.js ~7301) returns the known video object
(title/channel) or undefined. `toast(msg)` shows a toast.
## Steps
1. Apply the three patches (STOP and report on any failure):
```bash
git apply plans/patches/016-p2p-intake-new.diff
git apply plans/patches/016-media-cache-adopt.diff
git apply plans/patches/016-client-intake.diff
```
2. `server/package.json` "test" script — append ` && bun test --timeout 60000 ./p2p-intake.test.js`.
3. `server/server.js`:
a. change the media-cache import to `import { createMediaCache, HIGH, LOW, validateMedia } from './media-cache.js';`
b. add `import { registerIntakeRoutes } from './p2p-intake.js';`
c. directly after the `/api/p2p/holders` route (plan 014) add:
```js
// Device → server intake: the server hashes, validates (and scans when
// P2P_MALWARE_SCAN=1) before a cid is admitted. docs/p2p-architecture.md flow 7.
const intake = registerIntakeRoutes(app, {
cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile,
validateMedia: (path, expected, opts) => validateMedia(path, expected,
{ ...opts, ffmpeg: FFMPEG, ffprobe: process.env.FFPROBE_PATH || 'ffprobe' }),
adopt: (videoId, path, info) => media.adoptFile(videoId, path, info),
});
```
4. `docker-compose.yml` — under the P2P env lines from plan 008 add the commented lines:
```yaml
# P2P_INTAKE_DIR: "/app/data/p2p-intake" # quarantine for device uploads (never served)
# P2P_INTAKE_MAX_BYTES: "3221225472" # 3 GiB
```
5. `frontend/app.js` `renderSettings` template — directly BEFORE the closing `</div>` of the
Sharing group (i.e. after the `This device` row), insert:
```html
<div class="set-row hidden" id="p2pContributeRow">
<span>
Saved videos the server can't verify yet
<small>Uploading lets the server check them (hash + media validation) so other devices can get them. Uses your upload bandwidth.</small>
</span>
<button id="p2pContributeBtn" class="btn">Verify &amp; share</button>
</div>
```
6. `frontend/app.js` `renderSettings` — inside the plan-015 `(async () => { … })();` block, after the
`cnt.textContent = …;` line, add:
```js
const unknown = window.P2PClient ? await window.P2PClient.unknownVideos() : [];
const row = $('p2pContributeRow');
const btn = $('p2pContributeBtn');
if (row && btn && unknown.length) {
row.classList.remove('hidden');
btn.textContent = `Verify & share ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`;
btn.onclick = async () => {
btn.disabled = true;
let ok = 0;
for (let i = 0; i < unknown.length; i++) {
const v = videoById(unknown[i]) || {};
btn.textContent = `Uploading ${i + 1} of ${unknown.length}…`;
const r = await window.P2PClient.contribute(unknown[i], { title: v.title || '', channel: v.channel || '' });
if (r && r.ok) ok++;
}
toast(`Verified ${ok} of ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`);
btn.disabled = false;
row.classList.add('hidden');
};
}
```
## Out of scope / do NOT touch
- No automatic uploads (plan 018 adds server-requested ones). No UI for the admin (plan 019).
- Never serve files from `P2P_INTAKE_DIR`; never add a `/api/p2p/intake` GET.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — install it (apt-get install -y ffmpeg); intake/media tests need it"
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/app.js frontend/p2p-client.js frontend/opfs.js && echo FRONT_OK
cd server
bun ../plans/harness/intake-server.js >/tmp/ytp016.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node intake-check.mjs
kill $SRV; true
```
Expected: intake `4 pass`; media-cache `27 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`;
browser JSON `{"firstUnknown":1,"unknown":["upAAAAAAAA9"],"contribute":{"ok":true,"adopted":false,"cidOk":true},"secondAccepted":1,"secondUnknown":0}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

179
plans/queue/017-peer-transfer-1faaa7.md Normal file
View File

@@ -0,0 +1,179 @@
---
id: 017-peer-transfer-1faaa7
title: Download a verified file from another device over WebRTC
created: 2026-09-29
depends_on: [016-intake-and-server-verification-cfe031]
est_files: 6
---
# 017 — Peer download over WebRTC
## Objective
Implements flow 6 of `docs/p2p-architecture.md`. After this plan:
- `window.P2PTransfer` serves this device's saved files to other devices (only while
`settings.p2pShare !== false`, one upload at a time) and downloads from them over a
WebRTC data channel, signalled through `/ws/p2p` (plan 014). STUN only.
- The receiver writes through `p2p-recv-worker.js` into `videos/<id>.p2p.part` while
hashing, and renames to `<id>.mp4` ONLY when the SHA-256 equals the cid. A wrong,
corrupted or short file leaves nothing behind.
- When a video fails to load (YouTube and the server both failing), and
`settings.p2pReceive !== false`, the player shows **📡 Get it from a device (N online)**
next to **↻ Retry**. On success the video is saved offline, registered as `verified`
(so this device becomes a holder at its next report) and plays.
Pre-tested in Chromium with two browser contexts and a real hub
(`plans/harness/transfer-*.js`): 5 MiB file delivered and verified in ~2.5 s; unknown cid →
`peer declined: not here`; holder with a corrupted same-size file → `content hash mismatch`
and no file kept.
## Context the executor must NOT rediscover
- `plans/patches/017-p2p-transfer-new.diff` creates `frontend/p2p-transfer.js` and
`frontend/p2p-recv-worker.js`.
- Script order in `frontend/index.html` after plan 015:
`… sha256.js, device-db.js, p2p-client.js, p2p-core.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` has `'/p2p-core.js',` (plan 015).
- `frontend/app.js`:
- `Player.loadVideo` catch block (~line 1735):
```js
} catch (err) {
showSpinner(false);
toast('⚠ ' + err.message);
// Show retry button in the player pane
const retry = els.playerPane.querySelector('.retry-btn');
if (retry) retry.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn';
btn.textContent = '↻ Retry';
btn.addEventListener('click', () => {
btn.remove();
Player.loadVideo(videoObj, { preferStream, resume, reveal, nocache });
});
els.playerPane.appendChild(btn);
}
```
- globals/helpers: `downloading` (Set, ~line 372), `cachedIds`, `cacheMutations`,
`markCardCacheState(id, state)` (~1351), `updateDownloadBadge()` (~1313), `renderSidebar()`,
`recordDeviceFile(id, res)` (plan 012), `toast()`.
- `boot()` has (plan 013):
```js
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
- `window.P2PCore.pickPeers(payload, cid)` → `[{ peer, cid, size }]` online holders (plan 015).
## Steps
1. From the repo root: `git apply plans/patches/017-p2p-transfer-new.diff` (STOP on failure).
2. `frontend/index.html` — add ` <script src="p2p-transfer.js"></script>` directly after the `p2p-core.js` line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-transfer.js',` and `'/p2p-recv-worker.js',` directly after `'/p2p-core.js',`.
4. `frontend/styles.css` — append at the end of the file:
```css
/* P2P: "Get it from a device" next to Retry (plan 017) */
.peer-btn { margin-left: 8px; }
```
5. `frontend/app.js` `boot()` — inside the `if (WEB && window.P2PClient) { … }` block, after the
`start(...)` call add:
```js
if (window.P2PTransfer) window.P2PTransfer.start({ canShare: () => data.settings.p2pShare !== false });
```
6. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` (next to plan 015's
`renderAvailability`) add:
```js
// Fetch a verified copy from another device (docs/p2p-architecture.md flow 6).
// Download-then-play: on success the file is saved offline like any save.
async function getFromPeers(videoObj, c, peers, onProgress) {
const id = videoObj.id;
downloading.add(id);
markCardCacheState(id, 'downloading');
updateDownloadBadge();
try {
const res = await window.P2PTransfer.download({ videoId: id, cid: c.cid, size: c.size, peers, onProgress });
if (res.ok) {
cachedIds.add(id);
cacheMutations++;
recordDeviceFile(id, { sha256: res.sha256, expectedSha: c.cid, size: res.size });
toast(`Saved “${videoObj.title || id}” from another device ✓`);
}
return res;
} finally {
downloading.delete(id);
markCardCacheState(id, cachedIds.has(id) ? 'cached' : 'none');
updateDownloadBadge();
renderSidebar();
}
}
// Offered when a video won't load: only if some device holding it is online now.
async function offerPeerDownload(videoObj) {
if (!WEB || !window.P2PTransfer || !window.P2PCore || data.settings.p2pReceive === false) return;
const id = videoObj && videoObj.id;
if (!id || videoObj.custom || cachedIds.has(id)) return;
let payload = null;
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
payload = r.ok ? await r.json() : null;
} catch { return; }
const c = payload && Array.isArray(payload.cids) ? payload.cids[0] : null;
const peers = c ? window.P2PCore.pickPeers(payload, c.cid) : [];
if (!peers.length || !els.playerPane.querySelector('.retry-btn')) return; // nobody online / user moved on
const old = els.playerPane.querySelector('.peer-btn');
if (old) old.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn peer-btn';
btn.textContent = `📡 Get it from a device (${peers.length} online)`;
btn.addEventListener('click', async () => {
btn.disabled = true;
const res = await getFromPeers(videoObj, c, peers, (got, total) => {
btn.textContent = `📡 ${Math.min(99, Math.round((got / total) * 100))}%…`;
});
if (res.ok) {
els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());
Player.loadVideo(videoObj);
} else {
btn.disabled = false;
btn.textContent = '📡 Try again';
toast('⚠ ' + res.error);
}
});
els.playerPane.appendChild(btn);
}
```
7. `frontend/app.js` `Player.loadVideo` catch block — directly after `els.playerPane.appendChild(btn);`
add `offerPeerDownload(videoObj);`. Also, in the Retry button's click handler, change
`btn.remove();` to
`els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());` so the peer button
goes away with it.
## Out of scope / do NOT touch
- No TURN server, no progressive (streaming) playback from peers, no partial seeding.
- Don't change `p2p-client.js` or the server.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-transfer.js frontend/p2p-recv-worker.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-transfer.js\|p2p-recv-worker.js" frontend/index.html frontend/sw.js
cd server && bun install >/dev/null 2>&1
bun ../plans/harness/transfer-server.js >/tmp/ytp017.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node transfer-check.mjs 2>&1 | grep -v "status of 500"
kill $SRV; true
```
Expected: `FRONT_OK`; `# fail 0`; index.html `1`, sw.js `2`; browser JSON containing
`"good":{"ok":true`, `"list":[["gotAAAAAAAA",5243657]]`, `"bad":{"ok":false,"error":"peer declined: not here"}`,
`"corrupt":{"r":{"ok":false,"error":"content hash mismatch"},"names":["gotAAAAAAAA.mp4"]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

120
plans/queue/018-server-rehydrate-from-peer-4fb8bd.md Normal file
View File

@@ -0,0 +1,120 @@
---
id: 018-server-rehydrate-from-peer-4fb8bd
title: Restore an evicted server copy from an online holder
created: 2026-09-29
depends_on: [017-peer-transfer-1faaa7]
est_files: 6
---
# 018 — Rehydrate the server from a device
## Objective
Implements flow 8 of `docs/p2p-architecture.md`: the owner's "the original source can go
offline, the video stays reachable from devices". After this plan, when `/api/streams`
fails at the source (removed/private video, yt-dlp failing) AND the server has no ready
copy AND a device holding a verified cid of that video is online with sharing on:
- the server sends that ONE device `{type:'upload-request', videoId, cid}` over `/ws/p2p`
(at most once per video per 10 min) and answers
`503 { ok:false, restoring:true, error:'…try again in a minute.' }`;
- the device uploads the file through intake with `restore: true`; intake accepts a known
cid again ONLY when the server no longer holds those bytes, re-hashes + re-validates,
and adopts it into the media cache — the next play is served from the server again.
Pre-tested: rehydrator + restore intake (`bun:test` hub 4/4, intake 5/5) and the whole loop in
Chromium (`plans/harness/rehydrate-*.js`: first request → `restoring:true`, device uploads,
server adopts the exact cid).
## Context the executor must NOT rediscover
- Patches (apply in order):
- `plans/patches/018-server-rehydrate.diff` — `server/p2p-hub.js` gains
`createRehydrator({ p2pDb, hub, hasServerCopy, enabled, now })`; `server/p2p-intake.js` gains the
`restore` body flag and a `serverHasCid(cid)` dep; tests for both.
- `plans/patches/018-client-restore.diff` — `frontend/p2p-client.js` answers `upload-request`
(only when sharing is on, only for the exact cid it holds, one at a time) and `contribute()`
accepts `restore`.
- `server/server.js` `/api/streams` handler ends (~line 623):
```js
} catch (err) {
return c.json({ ok: false, error: err.message }, 500);
}
});
```
(this catch covers the yt-dlp resolve path; the cached-copy path returned earlier).
- Plans 013–016 created in server.js: `fileForCid(cid)`, `const p2p = registerP2pRoutes(…)`,
`const p2pHub = createP2pHub(…)`, the `/api/p2p/holders` route, and
`const intake = registerIntakeRoutes(app, { cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile, validateMedia: …, adopt: … });`
- `media.getReady(videoId)` resolves the ready row or null.
## Steps
1. From the repo root (STOP on failure):
```bash
git apply plans/patches/018-server-rehydrate.diff
git apply plans/patches/018-client-restore.diff
```
2. `server/server.js` — change the import from `./p2p-hub.js` to
`import { createP2pHub, holdersPayload, createRehydrator } from './p2p-hub.js';`
3. `server/server.js` — in the `registerIntakeRoutes(app, { … })` options add
`serverHasCid: async (cid) => !!(await fileForCid(cid)),`
4. `server/server.js` — directly after the `const intake = registerIntakeRoutes(…);` statement add:
```js
// Source gone + server copy evicted → ask one online holder to send it back
// through intake (docs/p2p-architecture.md flow 8).
const p2pRehydrate = createRehydrator({
p2pDb, hub: p2pHub, enabled: () => P2P.enabled,
hasServerCopy: async (id) => !!(await media.getReady(id).catch(() => null)),
});
```
5. `server/server.js` `/api/streams` — replace the final catch block shown in Context with:
```js
} catch (err) {
// The source failed. If a device holds a verified copy, ask it to send one
// to the server so the video comes back (P2P flow 8).
let restoring = false;
try { restoring = await p2pRehydrate(videoId); } catch { /* best effort */ }
if (restoring) {
return c.json({
ok: false, restoring: true,
error: 'This video is unavailable at the source — a device that has it is sending a copy to the server. Try again in a minute.',
}, 503);
}
return c.json({ ok: false, error: err.message }, 500);
}
```
(`p2pRehydrate` is declared further down the file; it is only called at request time, after startup.)
## Out of scope / do NOT touch
- The client error UI (the existing toast + Retry + plan-017 peer button already cover it).
- Do not ask more than one device per video per 10 minutes; do not upload without sharing on.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -3
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -3
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/rehydrate-server.js >/tmp/ytp018.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node rehydrate-check.mjs
kill $SRV; true
```
Expected: hub `4 pass`; intake `5 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"accepted":1,"first":{"ok":false,"restoring":true,"error":"restoring from a device"},"adopted":[["goneAAAAAAA",true]],"second":{"ok":false,"error":"source unavailable"}}`
(the harness's stand-in `/api/streams` has no media cache, so the second answer is expected).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

169
plans/queue/019-admin-p2p-panel-4dc623.md Normal file
View File

@@ -0,0 +1,169 @@
---
id: 019-admin-p2p-panel-4dc623
title: Add a P2P panel to the admin page
created: 2026-09-29
depends_on: [018-server-rehydrate-from-peer-4fb8bd]
est_files: 3
---
# 019 — Admin P2P panel
## Objective
Give the admin one place to see and steer peer-to-peer sharing
(`docs/p2p-architecture.md`). After this plan `/admin` has a **Peer-to-peer** section showing:
- config: P2P on/off, malware scan on/off (with a note that it is OFF by default and hashing +
validation always run), stale-after days, retention thresholds;
- counts: verified content, revoked, devices, active holder rows, cids with ≥1 holder,
devices online now, open/active intake uploads;
- the 30 most recently verified files (video, cid prefix, origin, scan, size, verified at,
holder count) with a **Revoke** button (revoked cids are never offered to or accepted
from devices again — plan 013 only accepts `verified` cids).
## Context the executor must NOT rediscover
- Admin auth middleware: `notes.requireAdminOrToken` (returned by `registerNoteRoutes`,
`server/server.js` ~line 1823). Used like `app.get('/api/admin/media', requireAdminOrToken, async (c) => …)`.
- In server.js (plans 008–018): `P2P` (config), `p2pDb` (module), `p2pHub.onlineCount()`,
`intake.openTickets()`, `intake.active()`.
- `server/p2p-db.js` has `rowsOf`, `p2pStats()`, `revokeContent(cid)`.
- `frontend/admin.html`:
- sections are `<section>…</section>` inside `<div id="app">`; the "Recent edits" section
(~line 190) starts with
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Recent edits</h2>
```
- helpers inside the script IIFE: `$(id)`, `esc(s)`, `api(path, opts)` (JSON in/out,
`opts.body` is JSON-encoded), classes `tbl`, `scroll`, `muted`, `row`, `spacer`.
- `async function boot()` calls `loadTokens(); loadRevs(); loadUploads();`.
## Steps
1. `server/p2p-db.js` — append:
```js
// Admin panel (plan 019): newest verified/revoked files with their holder counts.
export async function recentContent(limit = 30) {
const r = await db.execute({
sql: `SELECT c.cid, c.video_id, c.size, c.height, c.vcodec, c.origin, c.status, c.scan, c.verified_at, c.meta,
(SELECT COUNT(*) FROM p2p_holders h WHERE h.cid = c.cid AND h.status = 'active') AS holders
FROM p2p_content c ORDER BY c.verified_at DESC LIMIT ?`,
args: [limit],
});
return rowsOf(r).map((x) => ({ ...x, holders: Number(x.holders) || 0 }));
}
```
2. `server/server.js` — directly after the `const p2pRehydrate = createRehydrator({…});` statement add:
```js
// Admin: P2P overview + revoke (frontend/admin.html → Peer-to-peer).
app.get('/api/admin/p2p', notes.requireAdminOrToken, async (c) => c.json({
ok: true,
config: {
enabled: P2P.enabled, malwareScan: P2P.malwareScan, staleDays: P2P.staleDays,
keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
},
stats: { ...(await p2pDb.p2pStats()), online: p2pHub.onlineCount(), intakeOpen: intake.openTickets(), intakeActive: intake.active() },
recent: await p2pDb.recentContent(30),
}, 200, { 'Cache-Control': 'no-store' }));
app.post('/api/admin/p2p/revoke', notes.requireAdminOrToken, async (c) => {
const body = await c.req.json().catch(() => ({}));
const cid = String(body.cid || '').toLowerCase();
if (!/^[0-9a-f]{64}$/.test(cid)) return c.json({ ok: false, error: 'bad cid' }, 400);
await p2pDb.revokeContent(cid);
console.warn(`[p2p] admin revoked ${cid.slice(0, 12)}`);
return c.json({ ok: true });
});
```
3. `frontend/admin.html` — directly BEFORE the "Recent edits" `<section>` insert:
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Peer-to-peer</h2>
<span class="spacer"></span>
<button id="p2pRefreshBtn">↻ Refresh</button>
</div>
<p class="muted">Devices share verified copies with each other (docs/p2p-architecture.md). A file's hash is only
added after the server itself hashed and validated it. The malware scan is off by default
(<code>P2P_MALWARE_SCAN=1</code> to enable). Holders are never expired — ones not re-checked for the
stale period are only marked stale.</p>
<div id="p2pSummary" class="msg"></div>
<div class="scroll"><table class="tbl" id="p2pTable"></table></div>
</section>
```
4. `frontend/admin.html` script — directly ABOVE `async function boot() {` add:
```js
async function loadP2p() {
const j = await api('/api/admin/p2p');
if (!j.ok) { $('p2pSummary').textContent = j.error || 'P2P unavailable'; return; }
const c = j.config, s = j.stats;
$('p2pSummary').textContent =
`P2P ${c.enabled ? 'ON' : 'OFF'} · malware scan ${c.malwareScan ? 'ON' : 'off'} · stale after ${c.staleDays} d · ` +
`keep ≥${c.keepMinViews} views/${c.keepDays} d or played in ${c.keepRecentDays} d — ` +
`${s.content} verified (${s.revoked} revoked) · ${s.devices} devices, ${s.online} online · ` +
`${s.holders} holdings over ${s.heldCids} files · intake ${s.intakeActive} running / ${s.intakeOpen} open`;
const mb = (b) => (Number(b) / 1048576).toFixed(1) + ' MB';
$('p2pTable').innerHTML = '<tr><th>Video</th><th>cid</th><th>Origin</th><th>Scan</th><th>Size</th><th>Verified</th><th>Holders</th><th></th></tr>' +
j.recent.map((r) => {
let title = '';
try { title = JSON.parse(r.meta || '{}').title || ''; } catch { /* no meta */ }
return `<tr${r.status === 'revoked' ? ' class="muted"' : ''}><td>${esc(title || r.video_id)}<br><small>${esc(r.video_id)}</small></td>` +
`<td><code>${esc(r.cid.slice(0, 12))}</code></td><td>${esc(r.origin)}</td><td>${esc(r.scan)}</td><td>${mb(r.size)}</td>` +
`<td>${esc(new Date(Number(r.verified_at)).toLocaleString())}</td><td>${r.holders}</td>` +
`<td>${r.status === 'verified' ? `<button data-revoke="${esc(r.cid)}">Revoke</button>` : 'revoked'}</td></tr>`;
}).join('');
}
$('p2pRefreshBtn').addEventListener('click', loadP2p);
$('p2pTable').addEventListener('click', async (e) => {
const b = e.target.closest('[data-revoke]');
if (!b || !confirm('Revoke this file? Devices will stop sharing it and the server will never accept it again.')) return;
const j = await api('/api/admin/p2p/revoke', { method: 'POST', body: { cid: b.dataset.revoke } });
if (!j.ok) alert(j.error || 'failed');
loadP2p();
});
```
5. `frontend/admin.html` `boot()` — `loadUploads();` appears 3 times in the file; use ONLY the one
inside `async function boot()`, i.e. this exact pair of lines:
```js
loadUploads();
const want = videoIdFrom(new URLSearchParams(location.search).get('v') || '');
```
and insert ` loadP2p();` between them.
## Out of scope / do NOT touch
- No editing of config from the page (env vars stay the source of truth).
- `admin.html` is never cached by the SW — no `sw.js` change.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1 && [ -e public ] || ln -s ../frontend public
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
DBDIR=$(mktemp -d); DB_PATH=$DBDIR/t.db MEDIA_DIR=$DBDIR/media ADMIN_PASSWORD=test-pass PORT=3995 bun server.js >/tmp/ytp019.log 2>&1 & SRV=$!; sleep 4
curl -s -c /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"password":"test-pass"}' http://localhost:3995/api/admin/login
echo
curl -s -b /tmp/ytp019.jar http://localhost:3995/api/admin/p2p | head -c 400; echo
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3995/api/admin/p2p
curl -s -b /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"cid":"nope"}' http://localhost:3995/api/admin/p2p/revoke
echo
kill $SRV; true
grep -c "loadP2p" ../frontend/admin.html
```
Expected: `SERVER_OK`; every test file `0 fail`; login `{"ok":true…}`; the p2p call returns
`{"ok":true,"config":{"enabled":true,"malwareScan":false,"staleDays":7,…},"stats":{"content":0,…},"recent":[]}`;
without the cookie `401`; bad cid `{"ok":false,"error":"bad cid"}`; grep ≥ `3`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.