josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Queue executable speed and peer-to-peer sharing plans with tested patches and harnesses

Browse Source
This commit is contained in:
Claude
2026-09-29 19:03:12 +00:00
parent a0ffc1b493
commit c17bd4c9ac
54 changed files with 6369 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
docs/improvement-plan.md
View File

@@ -1,7 +1,25 @@
# ytplayer — Speed & Features Plan (2026-09-29) # ytplayer — Speed & Features Plan (2026-09-29)
Scope: **make the app faster** (startup, search, pressing play, moving around) Scope: **make the app faster** (startup, search, pressing play, moving around)
and **add new features**. Hardening/refactor work is intentionally out of scope. and **add new features**, led by **peer-to-peer video sharing**. Hardening/refactor
work is intentionally out of scope.
## Integrated roadmap (what is queued and executable today)
Everything below is broken into step-by-step plans a smaller model can execute
without exploring the code — see `plans/INDEX.md`. All 19 were dry-run end to end.
| Order | Plans | What it delivers | Source |
|-------|-------|------------------|--------|
| 1 | `001` | Timing marks (`ytp:boot`, `ytp:search`, `ytp:tap-to-play`) + yt-dlp duration logs | A "Measuring" |
| 2 | `002`–`003` | Compressed + ETagged shell (app.js 426 → 94 KB), self-hosted fonts | A1.1–A1.3 |
| 3 | `004`–`005` | One yt-dlp resolve per video at a time; resolve likely next plays before the tap | A3.1–A3.2 |
| 4 | `006`–`007` | Search via InnerTube (~0.8 s, yt-dlp fallback); long-lived yt-dlp workers | A2.1, A3.3 |
| 5 | `008`–`019` | **Peer-to-peer sharing** — design and rules in `docs/p2p-architecture.md` | Part B0 |
Not queued yet (write plans for them with `/plan-queue` when reached): A1.4 minify,
A1.5 deferred boot work, A1.6 lazy modules, A2.2 suggestions, A2.3/A2.4, A3.4/A3.5,
A4 (VPS media cache), A5 polish, and the Part B feature list.
## Where the time goes today (measured against prod) ## Where the time goes today (measured against prod)
@@ -99,6 +117,21 @@ before the user taps.**
## Part B — New features ## Part B — New features
### B0. Peer-to-peer video sharing (queued: plans 008–019)
The server keeps the user list, the video list and metadata; every validated copy
gets a content id = SHA-256 of its bytes; devices that save a video become persistent
**holders**; when YouTube and the server copy are gone, devices serve each other over
WebRTC and can restore the server's copy. Rules the owner set (2026-09-29):
**P2P is on by default**, the **server malware scan is off by default** (hashing +
media validation always run), and holder records are **persistent** — the UI shows
when each was last verified and marks old ones **stale** instead of dropping them.
Full design, data model, flows and security rules: `docs/p2p-architecture.md`.
The earlier phase 02–06 draft is superseded; its differences are listed at the end of
that document.
### Other features (not queued yet)
Ranked by fit with how the app is actually used (worship sets, sing-alongs, Ranked by fit with how the app is actually used (worship sets, sing-alongs,
offline playback). Each is sized; most reuse machinery that already exists. offline playback). Each is sized; most reuse machinery that already exists.
@@ -144,16 +177,15 @@ offline playback). Each is sized; most reuse machinery that already exists.
## Suggested order ## Suggested order
1. **Week 1 — quick speed wins:** add the timing marks (Measuring), A1.1 1. **Run the queue** (`/run-queue`): plans `001`–`007` (speed), then `008`–`019`
compression, A1.2 ETags, A1.3 fonts, A3.1 coalescing, A3.2 warm-on-intent, (peer-to-peer). Deploy after `007` and measure with the `ytp:*` marks and the
A3.4 instant UI. These are all small, and together they fix most of what `[ytdlp]` log lines before starting P2P.
feels slow today. 2. **Next speed plans to write:** A3.4 instant UI on tap, A2.2 suggestions,
2. **Week 2:** A2.1 InnerTube search + A2.2 suggestions, A1.4 defer/minify. A1.5 deferred boot work, then measure the WireGuard link (A4) and decide on the
3. **Week 3:** A3.3 persistent yt-dlp worker, A1.5 deferred boot work, then VPS media cache.
measure the WireGuard link (A4) and decide on the VPS media cache. 3. **Then features**, starting with the small high-fit ones: section loops,
4. **Then features**, starting with the small high-fit ones: section loops,
confidence monitor, count-in, lyrics search, stats wrap-up — then confidence monitor, count-in, lyrics search, stats wrap-up — then
service plans and chord charts. service plans and chord charts.
Each row is sized to be one commit (CLAUDE.md commit rules) and can be queued Each item is sized to be one commit (CLAUDE.md commit rules); queue new ones with
with the `plan-queue` skill. the `plan-queue` skill.

155
docs/p2p-architecture.md Normal file
View File

@@ -0,0 +1,155 @@
# Peer-to-peer video sharing — architecture (2026-09-29)
Source of truth for plans `008`–`019` in `plans/queue/`. Every P2P plan links here
instead of repeating the rules. It replaces the earlier "phase 02–06" draft; the
differences are listed at the end.
## What the owner asked for
> Peer-to-peer saving of videos. The server stores the user list, the video list and
> metadata. While the original source is online, a video is available for streaming
> and download. When it is downloaded to a device, the server records that device in
> the list of holders, so the video stays reachable from devices after the source is
> gone. Top or recent files stay on the server under a total space limit; files that
> don't meet the criteria (e.g. number of views, also stored on the server) are
> deleted first. The video id is the file hash of the highest-quality copy. The
> database grows over time. Each device has its own database that can be synced or
> added to the server's, with verification that the file exists. A file must first be
> downloaded by the server and checked before its hash is added to the server DB.
Corrections to the earlier draft (owner, 2026-09-29):
1. **P2P is ON by default** (server and every client).
2. **The server's malware scan is OFF by default** (admin opt-in). Hashing and the
media validation gate are ALWAYS on and cannot be turned off.
3. **Holder records are persistent, not short-lived leases.** A device stays listed
as a holder until it says the file is gone, fails a check, or its reported list no
longer contains it. The UI shows when each holder was last verified and marks it
**stale** when that is older than `P2P_STALE_DAYS` (default 7). Online-right-now is
a separate, live signal.
## Vocabulary
| Term | Meaning |
|------|---------|
| **source** | Where bytes originally come from: YouTube (via yt-dlp) or a server upload (`upl_…`). |
| **video id** | Existing ids (`dQw4w9WgXcQ`, `upl_…`). Still used everywhere in the app and API. |
| **content id (`cid`)** | Lowercase hex SHA-256 of the exact file bytes. The P2P identity of a file. One video id can have several cids over time (a better master, the HEVC copy). |
| **master** | The best copy the server keeps for a video: today the validated ≤720p H.264+AAC faststart MP4 in `MEDIA_DIR` (`<id>.<gen>.mp4`). The compression lane's HEVC copy is a second, separately hashed file. Raising the master quality later just creates new cids linked by `video_id`. |
| **verified content** | A `p2p_content` row. Exists only after the server itself held the complete bytes, computed the SHA-256 itself and `validateMedia()` passed (plus the malware scan if enabled). |
| **holder** | A device that reported holding a cid. Row in `p2p_holders`, never deleted by time. |
| **online** | The device has an open `/ws/p2p` socket right now (in memory only). |
| **stale** | `now - last_verified_at > P2P_STALE_DAYS`. Shown in the UI, still listed. |
## Data model (server, libsql — grows forever)
Added by plan 008 in `server/p2p-db.js` (`initP2pSchema()` runs after `initDb()`):
```sql
p2p_content (cid PK, video_id, size, height, vcodec, acodec, duration, meta JSON,
origin 'server'|'intake', status 'verified'|'revoked',
scan 'skipped'|'clean', created_at ms, verified_at ms)
p2p_devices (device_id PK 'dev_<16hex>', secret_hash, fingerprint, profile,
share 0|1, created_at ms, last_seen_at ms)
p2p_holders (cid, device_id, status 'active'|'removed', trust 'reported'|'challenged',
first_reported_at ms, last_verified_at ms, removed_at ms NULL,
PRIMARY KEY (cid, device_id))
video_views (video_id, day 'YYYY-MM-DD', n, PRIMARY KEY (video_id, day))
media_cache.sha256 -- new column: cid of the current <id>.<gen>.mp4
```
"User list" = the existing `users` (fingerprints) and `profiles` tables plus
`p2p_devices`. Nothing is ever deleted from `p2p_content`; a bad file is `revoked`.
## Configuration (`server/p2p-config.js`, plan 008)
| Env | Default | Meaning |
|-----|---------|---------|
| `P2P_ENABLED` | `1` (on) | `0` turns off every P2P route, the hub and client features. |
| `P2P_MALWARE_SCAN` | `0` (off) | `1` runs `P2P_SCAN_CMD <file>` before admission; exit 0 = clean, 1 = infected (rejected), other = error (not admitted, retried later). |
| `P2P_SCAN_CMD` | `clamscan --no-summary --infected` | Needs an image built with `--build-arg INSTALL_CLAMAV=1`. |
| `P2P_STALE_DAYS` | `7` | Holder older than this is shown as stale. |
| `P2P_KEEP_MIN_VIEWS` | `3` | Retention: views in the last `P2P_KEEP_DAYS` that make a server copy "top". |
| `P2P_KEEP_DAYS` | `30` | Window for counting views. |
| `P2P_KEEP_RECENT_DAYS` | `14` | Retention: played this recently = "recent". |
| `P2P_INTAKE_DIR` | `<DB dir>/p2p-intake` | Quarantine for device uploads. Never served. |
| `P2P_INTAKE_MAX_BYTES` | `3 GiB` | Largest accepted intake upload. |
Client settings (`data.settings`, per profile): `p2pShare: true` (let other devices
download my saved videos, and report holdings), `p2pReceive: true` (fetch from other
devices when YouTube and the server can't serve).
## Flows
1. **Server fetch (existing media cache) → verified content** (plan 009).
`runFetch` / `runOptimize` hash the promoted file, store `media_cache.sha256`, run the
scan if enabled, then upsert `p2p_content` (`origin 'server'`). `/api/download` sends
`X-Content-SHA256`. A backfill hashes already-cached files at boot, one at a time.
2. **Device save** (plan 012). The OPFS worker hashes while it writes. If the server
sent `X-Content-SHA256` and the hash differs, the save fails (bonus integrity check).
The device DB (`IndexedDB ytp-device`, store `files`) records `{videoId, cid, size,
savedAt, lastCheckedAt, state}`; `state` is `verified` when the hashes matched,
`unverified` when the server sent no hash, `unhashed` for old saves and the
main-thread fallback path.
3. **Holdings sync** (plan 013). Device registers once (`POST /api/p2p/device` →
`deviceId` + `secret`, kept in `localStorage.ytpDevice`). It reports its holdings
(`POST /api/p2p/holdings`, full list at launch, deltas after save/delete). The server
accepts only cids in `p2p_content` with `status='verified'`; unknown cids come back
in `unknown` (candidates for intake). When the server still has the file it returns
up to 5 **range challenges**; a correct answer sets `trust='challenged'`, a wrong one
removes the holder. `last_verified_at` = time of the last report where the device
re-checked the file (exists, same size; full re-hash every 30 days). A full report
marks every active holder row of that device that is missing from the list as
`removed`. There is **no TTL**.
4. **Presence** (plan 014). `/ws/p2p` socket per device, authenticated with the device
secret. Online status lives only in memory. The hub also relays WebRTC signalling
between two online devices and carries server → device requests (plan 018).
5. **Availability** (plans 014/015). `GET /api/p2p/holders?v=<videoId>` lists cids and
their holders: opaque peer id (never the fingerprint/profile), `online`,
`lastVerifiedAt`, `stale`, `trust`, plus `serverHas`. The UI shows e.g.
"📡 On 3 devices · 1 online now · last checked 2 d ago", with stale holders greyed.
6. **Peer download** (plan 017). WebRTC data channel (STUN only, same ICE list as watch
party), 64 KiB frames with `bufferedAmount` back-pressure, receiver writes through a
worker into OPFS while hashing; only a matching SHA-256 is committed. The new copy
is a holder at the next report. Download-then-play; no progressive peer streaming.
Used when `/api/streams` fails and the device has no copy, and from a
"Get from a device" button.
7. **Intake** (plan 016). A device can hand a file to the server
(`POST /api/p2p/intake` → ticket, `PUT` the bytes). The server writes it to the
quarantine dir, hashes it, runs `validateMedia()`, runs the scan if enabled, and only
then inserts `p2p_content` (`origin 'intake'`). If the server has no copy of that
video it adopts the file into the media cache (budget permitting).
8. **Rehydrate** (plan 018). When a video is requested, its source fails, the server
evicted its copy, and a verified holder is online with `p2pShare` on, the hub asks
that device to upload it through intake (known cid → quick accept).
9. **Retention** (plan 010). Views are counted per video per day. When the media cache
needs room it evicts in this order: copies that are neither "top"
(`views in P2P_KEEP_DAYS ≥ P2P_KEEP_MIN_VIEWS`) nor "recent" (played within
`P2P_KEEP_RECENT_DAYS`), fewest views first, then oldest; only then the qualifying
ones by LRU. The 10-minute play protection and `MEDIA_CACHE_MAX_BYTES` stay.
Evicting a server copy never deletes `p2p_content` or holder rows.
## Security rules every plan must keep
- No cid enters `p2p_content` unless the SERVER computed it over bytes it holds and
`validateMedia` passed. Clients can never insert or edit content rows, views or trust.
- Intake files live in `P2P_INTAKE_DIR`, never under `./public` or `MEDIA_DIR`, and
are deleted on failure.
- Device secrets: 32 random bytes, only `sha256(secret)` stored, compared with
`timingSafeEqual`.
- Holder lists never expose fingerprints, profile names or IPs; a peer id is
`sha256('peer:' + device_id).slice(0, 12)`.
- The hub relays signalling only between two authenticated, online devices, with a
per-socket message budget.
- `P2P_ENABLED=0` must leave the rest of the app working exactly as before.
- Jobs stay server-owned; never pass a request `AbortSignal` into them (CLAUDE.md).
## Where this differs from the earlier phase 02–06 draft
| Earlier draft | Now |
|---------------|-----|
| "Default-off" P2P subsystem; "no inventory/upload from default settings" | P2P on by default; devices report holdings and seed by default (can be turned off). |
| Mandatory scanner, "scan skip is failure" | Scanner off by default (`P2P_MALWARE_SCAN=0`); hash + `validateMedia` mandatory. |
| Short-lived online leases; "cache availability only as an expiring hint" | Persistent holder rows with `last_verified_at` and a stale marker; online status is separate. |
| Migrate all localStorage (`_ytpdata`) to IndexedDB first | Not now: the device DB holds files + cids only. `_ytpdata` stays in localStorage (lower risk). |
| Collections, invitations, scoped principals, signed manifests, TURN, renditions lineage | Deferred. Scope is one shared catalog + device secrets; add later if needed. |
| Separate `server/p2p/*` directory with migrations ledger | Flat files `server/p2p-*.js` matching the repo's style (`party.js`, `remote.js`). |

29
plans/INDEX.md Normal file
View File

@@ -0,0 +1,29 @@
# Plan Queue Index
Speed first (001–007, independent of P2P), then peer-to-peer sharing (008–019,
see `docs/p2p-architecture.md`). Run in order; each plan's `depends_on` lists what
must be in `done/` first. Every plan was dry-run end to end on 2026-09-29 (all 19
applied in order to a clean checkout: 13 server test files and 61 frontend tests
green, app boots with no JS errors, P2P on by default, offline boot works).
| Seq | ID | Title | Status | Commit | Notes |
|-----|----|-------|--------|--------|-------|
| 001 | 001-perf-timing-marks-105acc | Add startup, search and play timing marks plus yt-dlp duration logs | queued | | baseline numbers for the rest |
| 002 | 002-compress-and-etag-shell-bd459c | Serve the app shell gzip/brotli-compressed with ETags | queued | | app.js 426 KB → 94 KB (br) |
| 003 | 003-self-host-fonts-89466b | Self-host the three web fonts and drop the render-blocking Google Fonts CSS | queued | | needs network once; CSP font-src fix |
| 004 | 004-coalesce-stream-resolves-a92d40 | Coalesce concurrent resolveStreams calls for the same video | queued | | |
| 005 | 005-warm-streams-on-intent-47b3d3 | Warm the stream cache for likely next plays | queued | | cold play 7.5 s → cached 1.2 s |
| 006 | 006-innertube-search-48066b | Answer searches from YouTube InnerTube directly with yt-dlp fallback | queued | | search 4–5 s → ~0.8 s |
| 007 | 007-ytdlp-worker-045800 | Keep one long-lived yt-dlp worker process instead of spawning per call | queued | | ~1 s per yt-dlp call |
| 008 | 008-p2p-schema-and-config-127966 | Add P2P tables, config flags and db helpers | queued | | P2P ON, malware scan OFF by default |
| 009 | 009-server-content-hash-186e7f | Hash every validated server copy and register it as verified content | queued | | uses plans/patches/009-* |
| 010 | 010-views-and-retention-d0c6ca | Count views and evict server copies by retention criteria before LRU | queued | | |
| 011 | 011-browser-sha256-e1793d | Add an incremental SHA-256 library for the browser and node tests | queued | | |
| 012 | 012-device-file-registry-288d55 | Add the on-device IndexedDB file registry and hash saves while downloading | queued | | browser harness |
| 013 | 013-device-identity-and-holdings-3ba493 | Register devices and report verified holdings to the server | queued | | persistent holders, no TTL |
| 014 | 014-p2p-presence-hub-ceced8 | Add the /ws/p2p presence and signalling hub and the holders endpoint | queued | | stale flag, never hidden |
| 015 | 015-availability-ui-and-settings-3b9397 | Show peer availability with stale markers and add Sharing settings | queued | | |
| 016 | 016-intake-and-server-verification-cfe031 | Let a device hand a file to the server for hashing and validation | queued | | needs ffmpeg for tests |
| 017 | 017-peer-transfer-1faaa7 | Download a verified file from another device over WebRTC | queued | | STUN only |
| 018 | 018-server-rehydrate-from-peer-4fb8bd | Restore an evicted server copy from an online holder | queued | | |
| 019 | 019-admin-p2p-panel-4dc623 | Add a P2P panel to the admin page | queued | | |

22
plans/README.md Normal file
View File

@@ -0,0 +1,22 @@
# plans/
Executable plan queue (skills `plan-queue` / `run-queue`). Each file in `queue/` is
self-contained: exact files, anchors, code and copy-paste verification.
- `INDEX.md` — the order and status of every plan.
- `queue/ active/ done/ failed/` — plan lifecycle.
- `patches/` — unified diffs referenced by plans (`git apply plans/patches/<name>.diff`).
They were produced from code that passed its tests, and are meant to be applied in plan
order (e.g. `016-media-cache-adopt.diff` expects `009` and `010` applied). If a patch
does not apply, stop and report — never hand-edit around it.
- `harness/` — small throwaway servers + Playwright checks that exercise the browser
side of the P2P plans for real (OPFS, IndexedDB, WebSocket, WebRTC). Run the server
from `server/` (`bun ../plans/harness/<x>-server.js`), then `node <x>-check.mjs` from
`plans/harness/` (needs `npm i --no-save playwright` and a Chromium; the cloud image has
one at `/opt/pw-browsers/chromium-1194/chrome-linux/chrome`, override with `CHROMIUM=`).
Environment notes: server tests need `cd server && bun install`; media / intake tests need
`ffmpeg` + `ffprobe`; plan 003 needs network access to Google Fonts once; plan 007's
pool test needs `bin/yt-dlp` (`npm run setup`) or skips.
Stop a background server by PID (`… & SRV=$!` … `kill $SRV`) — `pkill -f <name>` also
matches the shell running the command and kills it.

0
plans/active/.gitkeep Normal file
View File

0
plans/done/.gitkeep Normal file
View File

0
plans/failed/.gitkeep Normal file
View File

22
plans/harness/device-db-check.mjs Normal file
View File

@@ -0,0 +1,22 @@
// Plan 012 harness check. Needs playwright (npm i --no-save playwright) and a
// Chromium: CHROMIUM=/path/to/chrome (default: the cloud image's /opt/pw-browsers one).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
p.on('console', (m) => console.log('console:', m.text()));
await p.goto('http://localhost:8766/');
const out = await p.evaluate(async () => {
const sha = await (await fetch('/sha')).text();
const good = await OPFS.downloadVideo('good', '/api/download/good');
const bad = await OPFS.downloadVideo('bad', '/api/download/bad');
const nohash = await OPFS.downloadVideo('nohash', '/api/download/nohash');
const list = (await OPFS.listVideos()).map((x) => x.id).sort();
await DeviceDB.putFile({ videoId: 'good', cid: good.sha256, size: good.size, state: 'verified' });
const rec = await DeviceDB.getFile('good');
const byCid = await DeviceDB.getByCid(good.sha256);
await DeviceDB.deleteFile('good');
const after = await DeviceDB.listFiles();
return { shaOk: good.sha256 === sha, goodExpected: good.expectedSha === sha, bad, nohashSha: nohash.sha256 === sha, nohashExpected: nohash.expectedSha, list, rec: rec && rec.state, byCid: byCid && byCid.videoId, after: after.length };
});
console.log(JSON.stringify(out));
await b.close();

17
plans/harness/device-db-server.js Normal file
View File

@@ -0,0 +1,17 @@
// Plan 012 harness: serves ../../frontend plus fake /api/download/<id> responses
// (good hash / wrong hash / no hash). Run: bun device-db-server.js
import { createHash } from 'node:crypto';
const FE = new URL('../../frontend/', import.meta.url).pathname;
const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256);
const sha = createHash('sha256').update(body).digest('hex');
Bun.serve({ port: 8766, fetch(req) {
const u = new URL(req.url);
if (u.pathname === '/api/download/good') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } });
if (u.pathname === '/api/download/bad') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': 'f'.repeat(64) } });
if (u.pathname === '/api/download/nohash') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length) } });
if (u.pathname === '/sha') return new Response(sha);
if (u.pathname === '/') return new Response('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script>', { headers: { 'Content-Type': 'text/html' } });
const f = Bun.file(FE + u.pathname.slice(1));
return new Response(f, { headers: { 'Content-Type': 'text/javascript' } });
} });
console.log('ready');

19
plans/harness/intake-check.mjs Normal file
View File

@@ -0,0 +1,19 @@
// Plan 016 harness check: an unknown saved file → contribute → accepted.
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
await p.goto('http://localhost:8769/');
const out = await p.evaluate(async () => {
const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer());
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('upAAAAAAAA9.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
P2PClient.start({ getSettings: () => ({}) });
const first = await P2PClient.sync();
const unknown = await P2PClient.unknownVideos();
const c = await P2PClient.contribute(unknown[0]);
const second = await P2PClient.sync();
return { firstUnknown: first.unknown.length, unknown, contribute: { ok: c.ok, adopted: c.adopted, cidOk: /^[0-9a-f]{64}$/.test(c.cid) }, secondAccepted: second.accepted.length, secondUnknown: second.unknown.length };
});
console.log(JSON.stringify(out));
await b.close();

29
plans/harness/intake-server.js Normal file
View File

@@ -0,0 +1,29 @@
// Plan 016 harness: routes + intake with real validateMedia (needs ffmpeg).
// Run from server/: bun ../plans/harness/intake-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, readFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h16-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js');
const { admitFile } = await import(process.cwd() + '/p2p-admit.js');
const { validateMedia } = await import(process.cwd() + '/media-cache.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const fx = join(root, 'fx.mp4');
spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]);
const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 };
const app = new Hono();
const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile });
app.get('/fixture.mp4', () => new Response(readFileSync(fx)));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8769, fetch: app.fetch });
console.log('ready');

26
plans/harness/p2p-client-check.mjs Normal file
View File

@@ -0,0 +1,26 @@
// Plan 013 harness check (see device-db-check.mjs for requirements).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
p.on('console', (m) => { if (m.type() === 'error') console.log('console:', m.text()); });
await p.goto('http://localhost:8767/');
const out = await p.evaluate(async () => {
const good = await OPFS.downloadVideo('goodAAAAAAA', '/api/download/goodAAAAAAA');
// A legacy save: in OPFS but no DeviceDB record, unknown to the server.
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('legacyAAAAA.mp4', { create: true })).createWritable();
await w.write(new Uint8Array(1000).fill(9)); await w.close();
await DeviceDB.putFile({ videoId: 'goodAAAAAAA', cid: good.sha256, size: good.size, state: 'unverified' });
await DeviceDB.putFile({ videoId: 'ghostAAAAAA', cid: 'e'.repeat(64), size: 5, state: 'verified' }); // no file
P2PClient.start({ getSettings: () => ({}) });
const res = await P2PClient.sync();
const recs = (await DeviceDB.listFiles()).map((r) => [r.videoId, r.state, !!r.cid]).sort();
const holders = await (await fetch('/debug/holders')).json();
// Turn sharing off → full withdrawal.
P2PClient.start({ getSettings: () => ({ p2pShare: false }) });
await P2PClient.sync();
const after = await (await fetch('/debug/holders')).json();
return { accepted: res && res.accepted.length, unknown: res && res.unknown.length, challenges: res && res.challenges.length, recs, trust: holders.map((h) => h.trust), afterShareOff: after.length, device: !!P2PClient.device() };
});
console.log(JSON.stringify(out));
await b.close();

27
plans/harness/p2p-client-server.js Normal file
View File

@@ -0,0 +1,27 @@
// Plan 013 harness: real p2p-routes + temp DB + the frontend files.
// Run from server/: bun ../plans/harness/p2p-client-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
const root = mkdtempSync(join(tmpdir(), 'ytp-h13-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { sha256Range } = await import(process.cwd() + '/hash.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256);
const sha = createHash('sha256').update(body).digest('hex');
const file = join(root, 'good.mp4'); writeFileSync(file, body);
await p2pDb.upsertContent({ cid: sha, videoId: 'goodAAAAAAA', size: body.length, origin: 'server', now: Date.now() });
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async (c) => (c === sha ? { path: file, size: body.length } : null), sha256Range });
app.get('/api/download/:id', () => new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } }));
app.get('/debug/holders', async (c) => c.json(await p2pDb.listHolders(sha)));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8767, fetch: app.fetch });
console.log('ready');

21
plans/harness/presence-check.mjs Normal file
View File

@@ -0,0 +1,21 @@
// Plan 014 harness check: two browsers, one signal (see device-db-check.mjs for requirements).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const pages = [];
for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); await p.goto('http://localhost:8768/'); pages.push(p); }
const peers = [];
for (const p of pages) {
peers.push(await p.evaluate(async () => {
P2PClient.start({ getSettings: () => ({}) });
await P2PClient.sync();
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
window.__got = [];
P2PClient.onMessage('signal', (m) => window.__got.push(m));
return P2PClient.peer();
}));
}
const sent = await pages[0].evaluate((to) => P2PClient.signal(to, { hi: 1 }), peers[1]);
await new Promise((r) => setTimeout(r, 500));
const got = await pages[1].evaluate(() => window.__got);
console.log(JSON.stringify({ peers: peers.map((x) => typeof x === 'string' && x.length === 12), distinct: peers[0] !== peers[1], sent, got: got.map((m) => ({ from: m.from === peers[0], data: m.data })) }));
await b.close();

24
plans/harness/presence-server.js Normal file
View File

@@ -0,0 +1,24 @@
// Plan 014 harness: p2p-routes + p2p-hub on a real Bun server (websocket).
// Run from server/: bun ../plans/harness/presence-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h14-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8768, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

21
plans/harness/rehydrate-check.mjs Normal file
View File

@@ -0,0 +1,21 @@
// Plan 018 harness check: a holder is asked to restore an evicted copy.
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
await p.goto('http://localhost:8771/');
const out = await p.evaluate(async () => {
const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer());
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('goneAAAAAAA.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
P2PClient.start({ getSettings: () => ({}) });
const rep = await P2PClient.sync(); // hashes the file, reports it → accepted (cid is verified content)
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
const s1 = await (await fetch('/api/streams?v=goneAAAAAAA')).json();
let dbg;
for (let i = 0; i < 50; i++) { dbg = await (await fetch('/debug')).json(); if (dbg.adopted.length) break; await new Promise((r) => setTimeout(r, 200)); }
const s2 = await (await fetch('/api/streams?v=goneAAAAAAA')).json();
return { accepted: rep.accepted.length, first: s1, adopted: dbg.adopted.map((a) => [a.videoId, a.sha256 === dbg.cid]), second: s2 };
});
console.log(JSON.stringify(out));
await b.close();

49
plans/harness/rehydrate-server.js Normal file
View File

@@ -0,0 +1,49 @@
// Plan 018 harness: hub + intake + rehydrator + a stand-in /api/streams whose
// source always fails and whose server copy is gone. Port 8771.
// Run from server/: bun ../plans/harness/rehydrate-server.js (needs ffmpeg)
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, readFileSync, unlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
const root = mkdtempSync(join(tmpdir(), 'ytp-h18-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js');
const { createP2pHub, createRehydrator } = await import(process.cwd() + '/p2p-hub.js');
const { admitFile } = await import(process.cwd() + '/p2p-admit.js');
const { validateMedia } = await import(process.cwd() + '/media-cache.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const fx = join(root, 'fx.mp4');
spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]);
const bytes = readFileSync(fx);
const cid = createHash('sha256').update(bytes).digest('hex');
await p2pDb.upsertContent({ cid, videoId: 'goneAAAAAAA', size: bytes.length, origin: 'server', now: Date.now() }); // verified earlier, file since evicted
const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 };
const adopted = [];
const app = new Hono();
const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile,
serverHasCid: async () => adopted.length > 0,
adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256 }); unlinkSync(path); return { adopted: true }; } });
const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => adopted.length > 0 });
app.get('/api/streams', async (c) => {
const v = c.req.query('v');
if (await rehydrate(v)) return c.json({ ok: false, restoring: true, error: 'restoring from a device' }, 503);
return c.json({ ok: false, error: 'source unavailable' }, 500);
});
app.get('/debug', (c) => c.json({ adopted, cid }));
app.get('/fixture.mp4', () => new Response(bytes));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8771, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

52
plans/harness/transfer-check.mjs Normal file
View File

@@ -0,0 +1,52 @@
// Plan 017 harness check: device A holds a file; device B downloads it over WebRTC.
import { chromium } from 'playwright';
const b = await chromium.launch({
executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome',
args: ['--disable-features=WebRtcHideLocalIpsWithMdns'],
});
const pages = [];
for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); p.on('console', (m) => { if (m.type() === 'error') console.log('console', i, m.text()); }); await p.goto('http://localhost:8770/'); pages.push(p); }
const setup = (p) => p.evaluate(async () => {
P2PClient.start({ getSettings: () => ({}) });
await P2PClient.sync();
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
P2PTransfer.start({ iceServers: [] });
return P2PClient.peer();
});
// Holder: write a 5 MB file + registry record.
const held = await pages[0].evaluate(async () => {
const bytes = new Uint8Array(5 * 1024 * 1024 + 777).map((_, i) => (i * 31 + 7) % 256);
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('heldAAAAAAA.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
const cid = Sha256.hex(bytes);
await DeviceDB.putFile({ videoId: 'heldAAAAAAA', cid, size: bytes.length, state: 'verified' });
return { cid, size: bytes.length };
});
const peerA = await setup(pages[0]);
await setup(pages[1]);
const t0 = Date.now();
const res = await pages[1].evaluate(async ({ peerA, held }) => {
const good = await P2PTransfer.download({ videoId: 'gotAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] });
const list = (await OPFS.listVideos()).map((x) => [x.id, x.size]);
const bad = await P2PTransfer.download({ videoId: 'badAAAAAAAA', cid: 'f'.repeat(64), size: held.size, peers: [{ peer: peerA }] });
const list2 = (await OPFS.listVideos()).map((x) => x.id);
return { good, list, bad, list2 };
}, { peerA, held });
// Holder's file gets corrupted (same size, one byte flipped) but its record still claims the cid.
await pages[0].evaluate(async () => {
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos');
const fh = await dir.getFileHandle('heldAAAAAAA.mp4');
const bytes = new Uint8Array(await (await fh.getFile()).arrayBuffer());
bytes[12345] ^= 0xff;
const w = await fh.createWritable(); await w.write(bytes); await w.close();
});
res.corrupt = await pages[1].evaluate(async ({ peerA, held }) => {
const r = await P2PTransfer.download({ videoId: 'corAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] });
await new Promise((x) => setTimeout(x, 300));
const names = [];
for await (const [n] of (await (await navigator.storage.getDirectory()).getDirectoryHandle('videos')).entries()) names.push(n);
return { r, names: names.sort() };
}, { peerA, held });
console.log(JSON.stringify({ ...res, ms: Date.now() - t0 }));
await b.close();

24
plans/harness/transfer-server.js Normal file
View File

@@ -0,0 +1,24 @@
// Plan 017 harness: p2p-routes + p2p-hub + the transfer scripts (port 8770).
// Run from server/: bun ../plans/harness/transfer-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h14-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script><script src="/p2p-transfer.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8770, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

63
plans/patches/009-media-cache-test.diff Normal file
View File

@@ -0,0 +1,63 @@
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -10,6 +10,8 @@
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const { createMediaCache, validateMedia, MediaSkip, HIGH, LOW, OPT_REV } = await import('./media-cache.js');
+const { initP2pSchema } = await import('./p2p-db.js');
+const { sha256File } = await import('./hash.js');
const fx = (name) => join(root, name);
const ff = (...args) => {
@@ -26,6 +28,7 @@
beforeAll(async () => {
await dbmod.initDb();
+ await initP2pSchema(); // adds media_cache.sha256
// 10 s 640x360 H.264 + AAC — the shape of a good cached copy.
ff('-f', 'lavfi', '-i', 'testsrc=size=640x360:rate=30:duration=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=10',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', fx('good.mp4'));
@@ -102,6 +105,7 @@
download: async (id, out) => { calls.push(id); await sleep(30); copyFileSync(fx(state.fixture), out); return out; },
transcode: { enabled: false },
freeBytes: () => 100 * 1024 ** 3,
+ backfillDelayMs: -1,
...opts,
});
return { cache, dir, calls, state };
@@ -110,6 +114,35 @@
for (const r of await dbmod.listMedia()) await dbmod.deleteMedia(r.video_id);
}
+describe('content hashes (P2P cids)', () => {
+ test('a promoted copy stores the sha256 of its mp4 and reports it once', async () => {
+ await clearDb();
+ const seen = [];
+ const { cache, dir } = makeCache({ onReady: (i) => { seen.push(i); } });
+ await cache.init();
+ const row = await cache.ensureCached('hashAAAAAA1', { priority: HIGH });
+ const want = await sha256File(join(dir, `hashAAAAAA1.${row.gen}.mp4`));
+ expect((await dbmod.getMedia('hashAAAAAA1')).sha256).toBe(want);
+ expect(await until(() => seen.length === 1)).toBe(true);
+ expect(seen[0]).toMatchObject({ id: 'hashAAAAAA1', gen: row.gen, sha256: want, vcodec: 'h264' });
+ expect(seen[0].meta.title).toBe('T hashAAAAAA1');
+ });
+
+ test('backfill hashes ready copies that have no sha256 yet', async () => {
+ await clearDb();
+ const seen = [];
+ const { cache } = makeCache({ onReady: (i) => { seen.push(i); } });
+ await cache.init();
+ await cache.ensureCached('hashAAAAAA2', { priority: HIGH });
+ await dbmod.upsertMedia('hashAAAAAA2', { sha256: null });
+ seen.length = 0;
+ expect(await cache.backfillHashes()).toBe(1);
+ expect((await dbmod.getMedia('hashAAAAAA2')).sha256).toMatch(/^[0-9a-f]{64}$/);
+ expect(await until(() => seen.length === 1)).toBe(true);
+ expect(await cache.backfillHashes()).toBe(0); // nothing left
+ });
+});
+
describe('media cache jobs', () => {
test('fetches once, dedupes concurrent requests, then serves from disk', async () => {
await clearDb();

120
plans/patches/009-media-cache.diff Normal file
View File

@@ -0,0 +1,120 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -24,6 +24,7 @@
statSync, statfsSync, unlinkSync,
} from 'node:fs';
import { join } from 'node:path';
+import { sha256File } from './hash.js';
export const HIGH = 0; // explicit save / Broken re-download
export const LOW = 1; // auto-cache after a play
@@ -167,6 +168,9 @@
freeBytes = () => { const s = statfsSync(dir); return s.bavail * s.bsize; },
now = () => Date.now(),
log = console,
+ hashFile = sha256File, // server-computed SHA-256 = the file's P2P content id
+ onReady = null, // ({ id, gen, path, sha256, size, … }) after a validated copy lands
+ backfillDelayMs = 30_000, // hash pre-existing copies this long after init()
} = {}) {
const tmpDir = join(dir, '.tmp');
const fileFor = (id, gen, kind = 'mp4') => join(dir, `${id}.${gen}.${kind}`);
@@ -236,6 +240,20 @@
}
}
+ // Tell the P2P layer a validated file with a server-computed hash exists.
+ // Never throws and never delays the caller.
+ function notifyReady(id, gen, sha256, probe, metaJson) {
+ if (!onReady || !sha256) return;
+ let meta = {};
+ try { meta = JSON.parse(metaJson || '{}'); } catch { /* keep {} */ }
+ Promise.resolve()
+ .then(() => onReady({
+ id, gen, path: fileFor(id, gen), sha256, size: probe.size, height: probe.height,
+ vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta,
+ }))
+ .catch((e) => log.warn?.(`[media] onReady ${id} failed: ${e.message}`));
+ }
+
async function setStatus(job, status) {
job.status = status;
await db.upsertMedia(job.id, { status, updated_at: now() });
@@ -308,6 +326,7 @@
'-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]);
if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr));
+ const sha256 = await hashFile(norm);
const prev = await db.getMedia(id);
const gen = ((prev && prev.gen) || 0) + 1;
renameSync(norm, fileFor(id, gen));
@@ -318,13 +337,14 @@
size: probe.size + sizeOf(fileFor(id, gen, 'm4a')),
height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration,
optimized: 0, meta: JSON.stringify(metaFromInfo(info, probe.duration)),
- attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t,
+ attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t, sha256,
};
await db.upsertMedia(id, fields);
removeFiles(id, gen);
job.status = 'ready';
log.info?.(`[media] cached ${id} ${probe.height}p ${fmtMB(fields.size)}`);
enqueueOptimize(id);
+ notifyReady(id, gen, sha256, probe, fields.meta);
return { ...(prev || {}), video_id: id, ...fields };
} catch (err) {
job.status = 'failed';
@@ -427,6 +447,7 @@
log.info?.(`[media] kept original ${id}: ${probe.vcodec} ${fmtMB(probe.size)} vs ${fmtMB(oldSize)}`);
return;
}
+ const sha256 = await hashFile(out);
const gen = row.gen + 1;
renameSync(out, fileFor(id, gen));
const oldAudio = fileFor(id, row.gen, 'm4a');
@@ -434,8 +455,9 @@
try { linkSync(oldAudio, newAudio); } catch { copyFileSync(oldAudio, newAudio); }
await db.upsertMedia(id, {
gen, size: probe.size + sizeOf(newAudio), height: probe.height, vcodec: probe.vcodec,
- duration: probe.duration, optimized: OPT_REV, updated_at: now(),
+ duration: probe.duration, optimized: OPT_REV, updated_at: now(), sha256,
});
+ notifyReady(id, gen, sha256, probe, cur.meta);
removeFiles(id, gen, OLD_GEN_GRACE_MS);
log.info?.(`[media] optimized ${id}: ${fmtMB(oldSize)} → ${fmtMB(probe.size)} (${probe.vcodec})`);
} catch (err) {
@@ -622,7 +644,34 @@
}
const s = await db.mediaStats();
log.info?.(`[media] ${s.count} cached (${fmtMB(s.bytes)}), ${resumed} job(s) resumed, dir ${dir}`);
+ if (backfillDelayMs >= 0) {
+ const t = setTimeout(() => backfillHashes().catch(() => {}), backfillDelayMs);
+ t.unref?.();
+ }
+ }
+
+ // Copies cached before hashing existed get their sha256 in the background,
+ // one at a time, so their content ids can be admitted too.
+ async function backfillHashes() {
+ let n = 0;
+ for (const r of await db.listMedia()) {
+ if (r.status !== 'ready' || r.sha256) continue;
+ const p = fileFor(r.video_id, r.gen);
+ if (!existsSync(p)) continue;
+ try {
+ const sha256 = await hashFile(p);
+ const cur = await db.getMedia(r.video_id);
+ if (!cur || cur.status !== 'ready' || cur.gen !== r.gen) continue;
+ await db.upsertMedia(r.video_id, { sha256 });
+ notifyReady(r.video_id, r.gen, sha256,
+ { size: sizeOf(p), height: r.height, vcodec: r.vcodec, acodec: r.acodec, duration: r.duration }, r.meta);
+ n++;
+ } catch (e) {
+ log.warn?.(`[media] hash backfill ${r.video_id}: ${e.message}`);
+ }
+ }
+ return n;
}
- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId };
+ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes };
}

44
plans/patches/010-retention-eviction.diff Normal file
View File

@@ -0,0 +1,44 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -226,7 +226,10 @@
let { bytes } = await db.mediaStats();
if (bytes + est > maxBytes) {
const cutoff = now() - EVICT_PROTECT_MS;
- for (const r of await db.listMediaLru()) {
+ // Retention order (plan 010) when the server provides it: copies that
+ // are neither top nor recent go first. Otherwise plain LRU.
+ const order = db.listMediaEvictionOrder ? await db.listMediaEvictionOrder() : await db.listMediaLru();
+ for (const r of order) {
if (bytes + est <= maxBytes) break;
if (r.video_id === id || r.last_access > cutoff || jobs.has(r.video_id) || optActive === r.video_id) continue;
await evict(r.video_id, 'budget');
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -240,6 +240,27 @@
expect(await cache.getReady('lruAAAAAAA3')).not.toBeNull();
});
+ test('eviction follows listMediaEvictionOrder when the db provides it', async () => {
+ await clearDb();
+ const size = readFileSync(fx('good.mp4')).length;
+ const db = {
+ getMedia: dbmod.getMedia, upsertMedia: dbmod.upsertMedia, deleteMedia: dbmod.deleteMedia,
+ listMedia: dbmod.listMedia, listMediaLru: dbmod.listMediaLru, touchMedia: dbmod.touchMedia,
+ mediaStats: dbmod.mediaStats,
+ // Retention says the MORE recently played #2 is the one to drop (e.g. no views).
+ listMediaEvictionOrder: async () => (await dbmod.listMediaLru()).reverse(),
+ };
+ const { cache } = makeCache({ db, maxBytes: size * 2 + 60 * 250 * 1024 + 50_000, duration: 10 });
+ await cache.init();
+ await cache.ensureCached('retAAAAAAA1', { priority: HIGH });
+ await cache.ensureCached('retAAAAAAA2', { priority: HIGH });
+ await dbmod.upsertMedia('retAAAAAAA1', { last_access: 1000 });
+ await dbmod.upsertMedia('retAAAAAAA2', { last_access: 2000 });
+ await cache.ensureCached('retAAAAAAA3', { priority: HIGH });
+ expect(await dbmod.getMedia('retAAAAAAA2')).toBeNull();
+ expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull();
+ });
+
test('free-disk guard skips caching', async () => {
await clearDb();
const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 });

75
plans/patches/012-opfs-hash.diff Normal file
View File

@@ -0,0 +1,75 @@
--- a/frontend/opfs-worker.js
+++ b/frontend/opfs-worker.js
@@ -13,12 +13,20 @@
* Out messages:
* { type: 'unsupported' } → caller falls back to main thread
* { type: 'progress', received } → bytes written so far
- * { type: 'done', ext } → file stored as <videoId>.<ext>
+ * { type: 'done', ext, sha256, expectedSha, size }
+ * → file stored as <videoId>.<ext>;
+ * sha256 = hash of the stored bytes
+ * (P2P content id), expectedSha = the
+ * server's X-Content-SHA256 or null
* { type: 'error', error } → failed; .part cleaned up
* ========================================================================== */
'use strict';
+// Incremental SHA-256 (frontend/sha256.js): the file is hashed while it is
+// written, so the device knows its content id without re-reading the file.
+try { importScripts('/sha256.js'); } catch { /* hashing unavailable — save still works */ }
+
async function getVideosDir() {
const root = await navigator.storage.getDirectory();
return root.getDirectoryHandle('videos', { create: true });
@@ -60,6 +68,7 @@
dir = await getVideosDir();
const partHandle = await dir.getFileHandle(partName, { create: true });
const access = await partHandle.createSyncAccessHandle();
+ const hasher = self.Sha256 ? self.Sha256.create() : null;
let offset = 0;
try {
const reader = res.body.getReader();
@@ -67,6 +76,7 @@
const { done, value } = await reader.read();
if (done) break;
access.write(value, { at: offset });
+ if (hasher) hasher.update(value);
offset += value.byteLength;
self.postMessage({ type: 'progress', received: offset });
}
@@ -81,6 +91,14 @@
if (expected > 0 && offset !== expected) {
throw new Error(`download cut short (${offset} of ${expected} bytes)`);
}
+ // The server names the hash of what it sent (media cache copies). A
+ // mismatch means the bytes were damaged on the way — never keep them.
+ const sha256 = hasher ? hasher.hex() : null;
+ const sent = (res.headers.get('x-content-sha256') || '').trim().toLowerCase();
+ const expectedSha = /^[0-9a-f]{64}$/.test(sent) ? sent : null;
+ if (sha256 && expectedSha && sha256 !== expectedSha) {
+ throw new Error('integrity check failed (content hash mismatch)');
+ }
// Finalize: .part → permanent name. Prefer the native rename, but treat
// ANY move() failure as "unavailable" and fall back to a chunked copy —
@@ -111,7 +129,7 @@
await dir.removeEntry(partName);
}
- self.postMessage({ type: 'done', ext });
+ self.postMessage({ type: 'done', ext, sha256, expectedSha, size: offset });
} catch (err) {
// Never leave a corrupt partial behind
try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ }
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -136,7 +136,7 @@
};
worker.onmessage = (e) => {
const m = e.data || {};
- if (m.type === 'done') finish({ ok: true });
+ if (m.type === 'done') finish({ ok: true, sha256: m.sha256 || null, expectedSha: m.expectedSha || null, size: m.size || 0 });
else if (m.type === 'unsupported') finish({ ok: false, fallback: true });
else if (m.type === 'error') finish({ ok: false, error: m.error });
// 'progress' messages are informational; ignored here

22
plans/patches/013-opfs-readrange.diff Normal file
View File

@@ -0,0 +1,22 @@
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -108,6 +108,19 @@
}
},
+ // Bytes [offset, offset+length) of a saved video — answers the server's
+ // P2P range challenges (p2p-client.js). null when the file is missing.
+ async readRange(videoId, offset, length) {
+ try {
+ const found = await findHandle(videoId);
+ if (!found) return null;
+ const file = await found[0].getFile();
+ return new Uint8Array(await file.slice(offset, offset + length).arrayBuffer());
+ } catch {
+ return null;
+ }
+ },
+
revokeUrl(url) {
if (url && _blobUrls.has(url)) {
URL.revokeObjectURL(url);

105
plans/patches/014-p2p-client-presence.diff Normal file
View File

@@ -0,0 +1,105 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -6,6 +6,9 @@
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
+ * onMessage(type, fn) / signal(to, data) / peer()
+ * live /ws/p2p socket (plan 014): the
+ * device is "online" while it is open
*
* What it does, in order, each sync:
* 1. registers the device once (localStorage ytpDevice)
@@ -148,6 +151,74 @@
return j;
}
+ // ---- presence socket (/ws/p2p) — plan 014 ------------------------------------
+ // Open while sharing OR receiving is on. Credentials go in the first message,
+ // never the URL. Reconnects with backoff (5 s … 5 min).
+ let ws = null;
+ let myPeer = null;
+ let wsRetry = 0;
+ let wsTimer = null;
+ const listeners = new Map(); // type -> Set<fn>
+
+ const wantSocket = () => {
+ const s = hooks.getSettings();
+ return s.p2pShare !== false || s.p2pReceive !== false;
+ };
+
+ function onMessage(type, fn) {
+ if (!listeners.has(type)) listeners.set(type, new Set());
+ listeners.get(type).add(fn);
+ return () => listeners.get(type).delete(fn);
+ }
+ function emit(m) {
+ for (const fn of listeners.get(m.type) || []) { try { fn(m); } catch { /* a listener's bug is not ours */ } }
+ }
+
+ function connect() {
+ if (ws || !wantSocket()) return;
+ const d = device();
+ if (!d) return;
+ const proto = location.protocol === 'https:' ? 'wss:' : 'ws:';
+ let sock;
+ try { sock = new WebSocket(`${proto}//${location.host}/ws/p2p`); } catch { return; }
+ ws = sock;
+ let ping = null;
+ sock.onopen = () => {
+ sock.send(JSON.stringify({ type: 'auth', device: d.deviceId, secret: d.secret }));
+ // The server drops sockets idle for 120 s (server.js websocketHandler).
+ ping = setInterval(() => { try { sock.send('{"type":"ping"}'); } catch { /* closing */ } }, 50_000);
+ };
+ sock.onmessage = (e) => {
+ let m;
+ try { m = JSON.parse(e.data); } catch { return; }
+ if (m.type === 'hello') { myPeer = m.peer; wsRetry = 0; }
+ emit(m);
+ };
+ sock.onclose = () => {
+ clearInterval(ping);
+ if (ws !== sock) return;
+ ws = null;
+ myPeer = null;
+ if (!wantSocket()) return;
+ clearTimeout(wsTimer);
+ wsTimer = setTimeout(connect, Math.min(300_000, 5000 * 2 ** wsRetry++));
+ };
+ }
+
+ function disconnect() {
+ clearTimeout(wsTimer);
+ const sock = ws;
+ ws = null;
+ myPeer = null;
+ if (sock) { try { sock.close(); } catch { /* gone */ } }
+ }
+
+ function signal(to, data) {
+ if (!ws || ws.readyState !== 1) return false;
+ ws.send(JSON.stringify({ type: 'signal', to, data }));
+ return true;
+ }
+
async function syncOnce() {
if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null;
if (navigator.onLine === false) return null;
@@ -158,6 +229,7 @@
await hashPending(recs);
const res = await report(recs, dev);
lastSync = Date.now();
+ if (wantSocket()) connect(); else disconnect();
return res;
}
@@ -185,5 +257,8 @@
});
}
- window.P2PClient = { start, changed, sync, device, authHeaders, config };
+ window.P2PClient = {
+ start, changed, sync, device, authHeaders, config,
+ onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer),
+ };
}());

213
plans/patches/014-p2p-hub-new.diff Normal file
View File

@@ -0,0 +1,213 @@
--- /dev/null
+++ b/server/p2p-hub.js
@@ -0,0 +1,120 @@
+/* ============================================================================
+ * p2p-hub.js — live presence + WebRTC signalling for P2P devices, and the
+ * availability payload (docs/p2p-architecture.md flows 4–5).
+ *
+ * /ws/p2p (credentials never go in the URL — proxies log URLs)
+ * you → server {type:'auth', device, secret} first message, within 10 s
+ * server → you {type:'hello', peer} after auth
+ * you → peer {type:'signal', to:<peer>, data} relayed as
+ * server → peer {type:'signal', from:<your peer>, data}
+ * server → you {type:'error', error}
+ * Server-initiated messages (plan 018) go through hub.send(deviceId, msg).
+ *
+ * "Online" lives only in memory: a server restart forgets it, a closed
+ * socket drops it. Holder ROWS are persistent (p2p_holders) — the payload
+ * reports both: `online` (now) and `lastVerifiedAt` / `stale` (history).
+ * Peers are addressed by an opaque id (peerIdOf), never by device id.
+ * ========================================================================== */
+import { createHash, timingSafeEqual } from 'node:crypto';
+
+const sha = (s) => createHash('sha256').update(String(s)).digest('hex');
+const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); };
+export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12);
+
+const MAX_MSG = 64 * 1024;
+const BUDGET_PER_MIN = 300;
+
+export function createP2pHub({ getDevice, enabled = () => true, now = () => Date.now(), log = console } = {}) {
+ const online = new Map(); // deviceId -> ws
+ const byPeer = new Map(); // peer -> deviceId (online only)
+
+ const send = (ws, m) => { try { ws.send(JSON.stringify(m)); } catch { /* gone */ } };
+
+ function upgrade(req, server) {
+ if (!enabled()) return new Response('p2p disabled', { status: 404 });
+ const ok = server.upgrade(req, { data: { hub: 'p2p', authed: false, budget: [] } });
+ return ok ? undefined : new Response('upgrade failed', { status: 400 });
+ }
+
+ function open(ws) {
+ const t = setTimeout(() => { if (!ws.data.authed) { try { ws.close(4401, 'auth timeout'); } catch { /* gone */ } } }, 10_000);
+ t.unref?.();
+ }
+
+ async function auth(ws, m) {
+ const deviceId = String(m.device || '');
+ const secret = String(m.secret || '');
+ const d = /^dev_[0-9a-f]{16}$/.test(deviceId) ? await getDevice(deviceId).catch(() => null) : null;
+ if (!d || !same(d.secret_hash, sha(secret))) { try { ws.close(4401, 'unknown device'); } catch { /* gone */ } return; }
+ ws.data.authed = true;
+ ws.data.deviceId = deviceId;
+ ws.data.peer = peerIdOf(deviceId);
+ const prev = online.get(deviceId);
+ if (prev && prev !== ws) { prev.data.deviceId = null; try { prev.close(4409, 'replaced'); } catch { /* gone */ } }
+ online.set(deviceId, ws);
+ byPeer.set(ws.data.peer, deviceId);
+ send(ws, { type: 'hello', peer: ws.data.peer });
+ }
+
+ async function message(ws, raw) {
+ const t = now();
+ ws.data.budget = ws.data.budget.filter((x) => t - x < 60_000);
+ if (ws.data.budget.length >= BUDGET_PER_MIN) { send(ws, { type: 'error', error: 'slow down' }); return; }
+ ws.data.budget.push(t);
+ const s = typeof raw === 'string' ? raw : Buffer.from(raw).toString('utf8');
+ if (s.length > MAX_MSG) return;
+ let m;
+ try { m = JSON.parse(s); } catch { return; }
+ if (!ws.data.authed) { if (m && m.type === 'auth' && !ws.data.authing) { ws.data.authing = true; await auth(ws, m); } return; }
+ if (m && m.type === 'signal' && typeof m.to === 'string') {
+ const target = online.get(byPeer.get(m.to));
+ if (!target || target === ws) { send(ws, { type: 'error', error: 'peer offline', to: m.to }); return; }
+ send(target, { type: 'signal', from: ws.data.peer, data: m.data });
+ }
+ }
+
+ function close(ws) {
+ const { deviceId, peer } = ws.data || {};
+ if (deviceId && online.get(deviceId) === ws) {
+ online.delete(deviceId);
+ byPeer.delete(peer);
+ }
+ }
+
+ return {
+ upgrade,
+ websocket: { open, message, close },
+ isOnline: (deviceId) => online.has(deviceId),
+ send: (deviceId, msg) => { const ws = online.get(deviceId); if (!ws) return false; send(ws, msg); return true; },
+ onlineCount: () => online.size,
+ };
+}
+
+// GET /api/p2p/holders payload. Only devices that share are listed; holders
+// are never dropped for age — `stale` just flags an old last check.
+export async function holdersPayload({ videoId, cid, p2pDb, isOnline, staleDays, serverHas, now = Date.now() }) {
+ const contents = cid
+ ? [await p2pDb.getContent(cid)].filter((c) => c && c.status === 'verified')
+ : await p2pDb.listContentForVideo(videoId);
+ const staleMs = staleDays * 86400_000;
+ const out = [];
+ for (const c of contents) {
+ const holders = (await p2pDb.listHolders(c.cid, 50))
+ .filter((h) => Number(h.share) === 1)
+ .map((h) => ({
+ peer: peerIdOf(h.device_id),
+ online: isOnline(h.device_id),
+ lastVerifiedAt: Number(h.last_verified_at),
+ stale: now - Number(h.last_verified_at) > staleMs,
+ trust: h.trust,
+ }))
+ .sort((a, b) => (b.online - a.online) || (b.lastVerifiedAt - a.lastVerifiedAt));
+ out.push({
+ cid: c.cid, videoId: c.video_id, size: Number(c.size), height: Number(c.height), vcodec: c.vcodec,
+ serverHas: !!(await serverHas(c.cid)),
+ counts: { holders: holders.length, online: holders.filter((h) => h.online).length, fresh: holders.filter((h) => !h.stale).length },
+ holders,
+ });
+ }
+ return { ok: true, staleDays, cids: out };
+}
--- /dev/null
+++ b/server/p2p-hub.test.js
@@ -0,0 +1,87 @@
+// Presence/signalling hub and the holders payload (plan 014).
+import { test, expect, beforeAll } from 'bun:test';
+import { mkdtempSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createHash } from 'node:crypto';
+
+const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-hub-'));
+process.env.DB_PATH = join(root, 'test.db');
+const dbmod = await import('./db.js');
+const p2pDb = await import('./p2p-db.js');
+const { createP2pHub, holdersPayload, peerIdOf } = await import('./p2p-hub.js');
+
+const sha = (s) => createHash('sha256').update(s).digest('hex');
+const SECRET_A = '1'.repeat(64);
+const SECRET_B = '2'.repeat(64);
+const CID = 'a'.repeat(64);
+const DAY = 86400_000;
+const NOW = Date.UTC(2026, 8, 29, 12);
+
+function fakeWs(data) {
+ return { data, sent: [], closed: null, send(s) { this.sent.push(JSON.parse(s)); }, close(code) { this.closed = code; } };
+}
+
+beforeAll(async () => {
+ await dbmod.initDb();
+ await p2pDb.initP2pSchema();
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000a', secretHash: sha(SECRET_A), now: NOW });
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000b', secretHash: sha(SECRET_B), now: NOW });
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000c', secretHash: sha('x'), now: NOW });
+ await p2pDb.touchDevice('dev_000000000000000c', { now: NOW, share: false });
+ await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1234, height: 720, vcodec: 'h264', origin: 'server', now: NOW });
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000a', now: NOW - 1 * DAY });
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000b', now: NOW - 30 * DAY }); // stale, still listed
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000c', now: NOW }); // share off → hidden
+});
+
+test('auth on open, hello with opaque peer id, bad secret is closed', async () => {
+ const hub = createP2pHub({ getDevice: p2pDb.getDevice });
+ const a = fakeWs({ budget: [] });
+ hub.websocket.open(a);
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: 'x', data: {} })); // ignored before auth
+ expect(a.sent.length).toBe(0);
+ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A }));
+ expect(a.sent[0]).toEqual({ type: 'hello', peer: peerIdOf('dev_000000000000000a') });
+ expect(hub.isOnline('dev_000000000000000a')).toBe(true);
+ const bad = fakeWs({ budget: [] });
+ hub.websocket.open(bad);
+ await hub.websocket.message(bad, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_A }));
+ expect(bad.closed).toBe(4401);
+ expect(hub.isOnline('dev_000000000000000b')).toBe(false);
+});
+
+test('signals are relayed between online peers only; close drops presence', async () => {
+ const hub = createP2pHub({ getDevice: p2pDb.getDevice });
+ const a = fakeWs({ budget: [] });
+ const b = fakeWs({ budget: [] });
+ hub.websocket.open(a);
+ hub.websocket.open(b);
+ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A }));
+ await hub.websocket.message(b, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_B }));
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: { sdp: 'x' } }));
+ expect(b.sent.at(-1)).toEqual({ type: 'signal', from: a.data.peer, data: { sdp: 'x' } });
+ hub.websocket.close(b);
+ expect(hub.isOnline('dev_000000000000000b')).toBe(false);
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: {} }));
+ expect(a.sent.at(-1)).toMatchObject({ type: 'error', error: 'peer offline' });
+ expect(hub.send('dev_000000000000000a', { type: 'ping' })).toBe(true);
+ expect(hub.send('dev_000000000000000b', { type: 'ping' })).toBe(false);
+});
+
+test('holders payload: persistent rows, online flag, stale marker, share-off hidden', async () => {
+ const online = new Set(['dev_000000000000000b']);
+ const p = await holdersPayload({
+ videoId: 'dQw4w9WgXcQ', p2pDb, isOnline: (d) => online.has(d), staleDays: 7,
+ serverHas: async () => false, now: NOW,
+ });
+ expect(p.staleDays).toBe(7);
+ expect(p.cids.length).toBe(1);
+ const c = p.cids[0];
+ expect(c).toMatchObject({ cid: CID, size: 1234, height: 720, serverHas: false, counts: { holders: 2, online: 1, fresh: 1 } });
+ expect(c.holders.map((h) => [h.peer, h.online, h.stale])).toEqual([
+ [peerIdOf('dev_000000000000000b'), true, true], // online first even though stale
+ [peerIdOf('dev_000000000000000a'), false, false],
+ ]);
+ expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids
+});

107
plans/patches/015-p2p-core-new.diff Normal file
View File

@@ -0,0 +1,107 @@
--- /dev/null
+++ b/frontend/p2p-core.js
@@ -0,0 +1,55 @@
+/* ============================================================================
+ * p2p-core.js — pure helpers for peer-to-peer UI (window.P2PCore / node)
+ *
+ * formatAvailability(payload, now) turns GET /api/p2p/holders into the line
+ * under the now-playing title, e.g.
+ * "📡 On 3 devices · 1 online now · last checked 2d ago"
+ * Holders are persistent (docs/p2p-architecture.md): they are never hidden
+ * for age; when every holder is older than the server's staleDays the line
+ * says so and is marked stale.
+ * ========================================================================== */
+(function (root) {
+ 'use strict';
+
+ function ago(ms, now) {
+ const s = Math.max(0, Math.round((now - ms) / 1000));
+ if (s < 90) return 'just now';
+ const m = Math.round(s / 60);
+ if (m < 90) return m + 'm ago';
+ const h = Math.round(m / 60);
+ if (h < 48) return h + 'h ago';
+ return Math.round(h / 24) + 'd ago';
+ }
+
+ function formatAvailability(payload, now) {
+ if (!payload || !payload.ok || !Array.isArray(payload.cids)) return null;
+ const holders = [];
+ for (const c of payload.cids) for (const h of c.holders || []) holders.push(h);
+ if (!holders.length) return null;
+ const online = holders.filter((h) => h.online).length;
+ const fresh = holders.filter((h) => !h.stale).length;
+ const newest = Math.max(...holders.map((h) => Number(h.lastVerifiedAt) || 0));
+ const n = holders.length;
+ const allStale = fresh === 0;
+ const text = `📡 On ${n} device${n === 1 ? '' : 's'} · ${online ? online + ' online now' : 'none online'}`
+ + ` · last checked ${ago(newest, now)}` + (allStale ? ' (not checked recently)' : '');
+ const title = holders.map((h) =>
+ `${h.peer} · ${h.online ? 'online' : 'offline'} · checked ${ago(Number(h.lastVerifiedAt) || 0, now)}`
+ + `${h.stale ? ' (stale)' : ''}${h.trust === 'challenged' ? ' · spot-checked' : ''}`).join('\n');
+ return { text, title, stale: allStale, online, holders: n };
+ }
+
+ // Holders worth trying for a download: online first, then freshest check.
+ function pickPeers(payload, cid) {
+ if (!payload || !Array.isArray(payload.cids)) return [];
+ const c = payload.cids.find((x) => x.cid === cid) || payload.cids[0];
+ if (!c) return [];
+ return (c.holders || []).filter((h) => h.online)
+ .sort((a, b) => (b.trust === 'challenged') - (a.trust === 'challenged') || b.lastVerifiedAt - a.lastVerifiedAt)
+ .map((h) => ({ peer: h.peer, cid: c.cid, size: c.size }));
+ }
+
+ const P2PCore = { ago, formatAvailability, pickPeers };
+ if (typeof module !== 'undefined' && module.exports) module.exports = P2PCore;
+ else root.P2PCore = P2PCore;
+})(typeof globalThis !== 'undefined' ? globalThis : this);
--- /dev/null
+++ b/frontend/p2p-core.test.js
@@ -0,0 +1,46 @@
+'use strict';
+const { test } = require('node:test');
+const assert = require('node:assert');
+const C = require('./p2p-core');
+
+const NOW = Date.UTC(2026, 8, 29, 12);
+const H = 3600_000;
+const payload = (holders, extra = {}) => ({ ok: true, staleDays: 7, cids: [{ cid: 'a'.repeat(64), size: 10, holders, ...extra }] });
+
+test('ago buckets', () => {
+ assert.strictEqual(C.ago(NOW - 30_000, NOW), 'just now');
+ assert.strictEqual(C.ago(NOW - 20 * 60_000, NOW), '20m ago');
+ assert.strictEqual(C.ago(NOW - 5 * H, NOW), '5h ago');
+ assert.strictEqual(C.ago(NOW - 72 * H, NOW), '3d ago');
+});
+
+test('no holders → nothing to show', () => {
+ assert.strictEqual(C.formatAvailability(payload([]), NOW), null);
+ assert.strictEqual(C.formatAvailability(null, NOW), null);
+});
+
+test('mixed holders: counts, newest check, not stale', () => {
+ const f = C.formatAvailability(payload([
+ { peer: 'p1', online: true, lastVerifiedAt: NOW - 30 * 24 * H, stale: true, trust: 'reported' },
+ { peer: 'p2', online: false, lastVerifiedAt: NOW - 48 * H, stale: false, trust: 'challenged' },
+ ]), NOW);
+ assert.strictEqual(f.text, '📡 On 2 devices · 1 online now · last checked 2d ago');
+ assert.strictEqual(f.stale, false);
+ assert.match(f.title, /p1 · online · checked 30d ago \(stale\)/);
+ assert.match(f.title, /p2 · offline · checked 2d ago · spot-checked/);
+});
+
+test('every holder stale → still listed, flagged', () => {
+ const f = C.formatAvailability(payload([{ peer: 'p1', online: false, lastVerifiedAt: NOW - 20 * 24 * H, stale: true }]), NOW);
+ assert.strictEqual(f.text, '📡 On 1 device · none online · last checked 20d ago (not checked recently)');
+ assert.strictEqual(f.stale, true);
+});
+
+test('pickPeers: online only, spot-checked first', () => {
+ const p = payload([
+ { peer: 'off', online: false, lastVerifiedAt: NOW, trust: 'challenged' },
+ { peer: 'on1', online: true, lastVerifiedAt: NOW, trust: 'reported' },
+ { peer: 'on2', online: true, lastVerifiedAt: NOW - H, trust: 'challenged' },
+ ]);
+ assert.deepStrictEqual(C.pickPeers(p).map((x) => x.peer), ['on2', 'on1']);
+});

91
plans/patches/016-client-intake.diff Normal file
View File

@@ -0,0 +1,91 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -6,6 +6,9 @@
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
+ * contribute(videoId) / unknownVideos()
+ * hand a saved file the server doesn't
+ * know yet to /api/p2p/intake (plan 016)
* onMessage(type, fn) / signal(to, data) / peer()
* live /ws/p2p socket (plan 014): the
* device is "online" while it is open
@@ -33,6 +36,7 @@
let again = false;
let timer = null;
let lastSync = 0;
+ let lastUnknown = []; // cids the server did not recognise at the last report
function device() {
try {
@@ -130,6 +134,7 @@
if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; }
const j = await r.json();
if (!j || !j.ok) return null;
+ lastUnknown = Array.isArray(j.unknown) ? j.unknown : [];
const accepted = new Set(j.accepted || []);
for (const rec of recs) {
if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); }
@@ -151,6 +156,33 @@
return j;
}
+ // ---- intake (plan 016) ---------------------------------------------------------
+ // Send one saved video to the server so it can hash + validate it and add its
+ // cid to the catalog. Uploads the whole file: only on the user's request
+ // ("Verify & share") or when the server asks for a copy (plan 018).
+ async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) {
+ const dev = await ensureDevice();
+ const rec = await window.DeviceDB.getFile(videoId);
+ const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null;
+ if (!file) return { ok: false, error: 'not saved on this device' };
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
+ const t = await (await fetch('/api/p2p/intake', {
+ method: 'POST', headers: { ...h, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }),
+ })).json().catch(() => ({ ok: false, error: 'server unreachable' }));
+ if (!t.ok || t.known) { if (t.known) changed(); return t; }
+ const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file }))
+ .json().catch(() => ({ ok: false, error: 'upload failed' }));
+ if (r.ok) changed();
+ return r;
+ }
+
+ async function unknownVideos() {
+ if (!lastUnknown.length || !window.DeviceDB) return [];
+ const recs = await window.DeviceDB.listFiles();
+ return recs.filter((r) => r.cid && lastUnknown.includes(r.cid)).map((r) => r.videoId);
+ }
+
// ---- presence socket (/ws/p2p) — plan 014 ------------------------------------
// Open while sharing OR receiving is on. Credentials go in the first message,
// never the URL. Reconnects with backoff (5 s … 5 min).
@@ -258,7 +290,7 @@
}
window.P2PClient = {
- start, changed, sync, device, authHeaders, config,
+ start, changed, sync, device, authHeaders, config, contribute, unknownVideos,
onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer),
};
}());
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -108,6 +108,17 @@
}
},
+ // The saved File itself (disk-backed, not read into memory) — used as an
+ // upload body by P2P intake. null when not saved.
+ async getFileObject(videoId) {
+ try {
+ const found = await findHandle(videoId);
+ return found ? await found[0].getFile() : null;
+ } catch {
+ return null;
+ }
+ },
+
// Bytes [offset, offset+length) of a saved video — answers the server's
// P2P range challenges (p2p-client.js). null when the file is missing.
async readRange(videoId, offset, length) {

81
plans/patches/016-media-cache-adopt.diff Normal file
View File

@@ -0,0 +1,81 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -522,6 +522,42 @@
}
// The "Broken" button: drop the copy and fetch it again at high priority.
+ // Promote a file that came from somewhere else (P2P intake / rehydrate from
+ // a device) as this video's server copy. The caller has already hashed it
+ // and run validateMedia(). The bytes are kept EXACTLY (no remux — the cid
+ // must stay true); only the audio sidecar is derived. Never replaces an
+ // existing ready copy or races a running fetch.
+ async function adoptFile(id, src, { sha256, probe, meta = {} }) {
+ if (!isMediaId(id)) throw new Error('adopt: bad video id');
+ const cur = await db.getMedia(id);
+ if (cur && cur.status === 'ready' && readyFilesExist(cur)) return { adopted: false, reason: 'already cached' };
+ if (jobs.has(id)) return { adopted: false, reason: 'fetch in progress' };
+ await makeRoom(id, probe.size);
+ const prefix = `${id}-${now()}-adopt`;
+ const m4a = join(tmpDir, `${prefix}.m4a`);
+ try {
+ const ex = await run(ffmpeg, ['-v', 'error', '-nostdin', '-y', '-i', src,
+ '-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]);
+ if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr));
+ const gen = ((cur && cur.gen) || 0) + 1;
+ try { renameSync(src, fileFor(id, gen)); } catch { copyFileSync(src, fileFor(id, gen)); unlinkSync(src); }
+ renameSync(m4a, fileFor(id, gen, 'm4a'));
+ const t = now();
+ await db.upsertMedia(id, {
+ status: 'ready', gen, size: probe.size + sizeOf(fileFor(id, gen, 'm4a')),
+ height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration,
+ optimized: OPT_REV, // re-encoding would change the bytes other devices hold
+ meta: JSON.stringify(meta), priority: HIGH, auto: 0, attempts: 0, error: null, retry_at: 0,
+ created_at: (cur && cur.created_at) || t, updated_at: t, last_access: t, sha256,
+ });
+ removeFiles(id, gen);
+ log.info?.(`[media] adopted ${id} ${probe.height}p ${fmtMB(probe.size)} from P2P`);
+ return { adopted: true, gen };
+ } finally {
+ sweepTmp(prefix);
+ }
+ }
+
async function redownload(id) {
if (!isMediaId(id)) throw new MediaSkip('invalid video id');
const existing = jobs.get(id);
@@ -676,5 +712,5 @@
return n;
}
- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes };
+ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes, adoptFile };
}
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -261,6 +261,26 @@
expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull();
});
+ test('adoptFile promotes a validated outside file byte-for-byte, once', async () => {
+ await clearDb();
+ const { cache, dir, calls } = makeCache();
+ await cache.init();
+ const src = join(root, 'adopt-src.mp4');
+ copyFileSync(fx('good-fs.mp4'), src);
+ const bytes = readFileSync(src);
+ const probe = await validateMedia(src, 0);
+ const r = await cache.adoptFile('adoptAAAAA1', src, { sha256: 'e'.repeat(64), probe, meta: { title: 'from a device' } });
+ expect(r.adopted).toBe(true);
+ const row = await cache.getReady('adoptAAAAA1');
+ expect(row.sha256).toBe('e'.repeat(64));
+ expect(row.optimized).toBe(OPT_REV);
+ expect(readFileSync(join(dir, `adoptAAAAA1.${row.gen}.mp4`)).equals(bytes)).toBe(true);
+ expect(existsSync(join(dir, `adoptAAAAA1.${row.gen}.m4a`))).toBe(true);
+ expect(calls.length).toBe(0); // nothing fetched from the source
+ copyFileSync(fx('good-fs.mp4'), src);
+ expect((await cache.adoptFile('adoptAAAAA1', src, { sha256: 'f'.repeat(64), probe })).adopted).toBe(false);
+ });
+
test('free-disk guard skips caching', async () => {
await clearDb();
const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 });

230
plans/patches/016-p2p-intake-new.diff Normal file
View File

@@ -0,0 +1,230 @@
--- /dev/null
+++ b/server/p2p-intake.js
@@ -0,0 +1,121 @@
+/* ============================================================================
+ * p2p-intake.js — a device hands a file to the server for verification
+ * (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did
+ * not come from the server's own fetch can become verified content.
+ *
+ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? }
+ * → { ok, known:true } cid already verified — nothing to send
+ * → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min
+ * PUT /api/p2p/intake/:ticket (device) raw file body
+ * → { ok, cid, adopted } admitted (+ adopted into the media cache)
+ * → 4xx { ok:false, error } rejected; nothing kept
+ *
+ * Order is fixed: bytes land in P2P_INTAKE_DIR (never served) → the SERVER
+ * hashes them while writing → a claimed cid must match → validateMedia() →
+ * admitFile() (malware scan only when P2P_MALWARE_SCAN=1) → holder row for
+ * the uploader → optional adoption into the media cache. Any failure deletes
+ * the file.
+ * ========================================================================== */
+import { createHash, randomBytes } from 'node:crypto';
+import { createWriteStream, mkdirSync, unlinkSync } from 'node:fs';
+import { join } from 'node:path';
+
+const CID_RE = /^[0-9a-f]{64}$/;
+const ID_RE = /^[A-Za-z0-9_-]{6,64}$/;
+const TICKET_TTL = 30 * 60_000;
+const MAX_ACTIVE = 2;
+// Display text from the device — untrusted: control chars/brackets stripped, capped.
+const clean = (v, max) => String(v || '').replace(/[\u0000-\u001f\u007f<>]+/g, ' ').replace(/\s+/g, ' ').trim().slice(0, max);
+
+export function registerIntakeRoutes(app, deps) {
+ const {
+ cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null,
+ now = () => Date.now(), log = console,
+ } = deps;
+ const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy }
+ let active = 0;
+ mkdirSync(cfg.intakeDir, { recursive: true });
+
+ const sweep = () => { const t = now(); for (const [k, v] of tickets) if (!v.busy && v.expiresAt < t) tickets.delete(k); };
+
+ app.post('/api/p2p/intake', gate, requireDevice, async (c) => {
+ const d = c.get('device');
+ const body = await c.req.json().catch(() => ({}));
+ const videoId = String(body.videoId || '');
+ const cid = body.cid ? String(body.cid).toLowerCase() : null;
+ const size = Number(body.size);
+ if (!ID_RE.test(videoId) || (cid && !CID_RE.test(cid))) return c.json({ ok: false, error: 'bad videoId or cid' }, 400);
+ if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413);
+ if (cid) {
+ const known = await p2pDb.getContent(cid);
+ if (known && known.status === 'verified') return c.json({ ok: true, known: true });
+ if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410);
+ }
+ sweep();
+ for (const v of tickets.values()) if (v.deviceId === d.device_id) return c.json({ ok: false, error: 'an upload from this device is already open' }, 429);
+ const ticket = randomBytes(16).toString('hex');
+ const expiresAt = now() + TICKET_TTL;
+ const meta = { title: clean(body.title, 300), channel: clean(body.channel, 200) };
+ tickets.set(ticket, { deviceId: d.device_id, videoId, cid, size, meta, expiresAt, busy: false });
+ return c.json({ ok: true, ticket, url: `/api/p2p/intake/${ticket}`, expiresAt });
+ });
+
+ app.put('/api/p2p/intake/:ticket', gate, requireDevice, async (c) => {
+ const d = c.get('device');
+ const ticket = c.req.param('ticket');
+ const tk = tickets.get(ticket);
+ if (!tk || tk.deviceId !== d.device_id || tk.expiresAt < now()) return c.json({ ok: false, error: 'unknown or expired ticket' }, 404);
+ if (tk.busy) return c.json({ ok: false, error: 'upload already running' }, 409);
+ if (active >= MAX_ACTIVE) return c.json({ ok: false, error: 'server busy, retry later' }, 503);
+ tk.busy = true;
+ active++;
+ const path = join(cfg.intakeDir, ticket + '.mp4');
+ let keep = false;
+ try {
+ const h = createHash('sha256');
+ let got = 0;
+ const out = createWriteStream(path);
+ const reader = c.req.raw.body ? c.req.raw.body.getReader() : null;
+ if (!reader) throw Object.assign(new Error('empty body'), { status: 400 });
+ try {
+ for (;;) {
+ const { done, value } = await reader.read();
+ if (done) break;
+ got += value.byteLength;
+ if (got > tk.size) throw Object.assign(new Error('more bytes than announced'), { status: 413 });
+ h.update(value);
+ if (!out.write(value)) await new Promise((r) => out.once('drain', r));
+ }
+ } finally {
+ await new Promise((r) => out.end(r));
+ }
+ if (got !== tk.size) throw Object.assign(new Error(`size mismatch (${got} of ${tk.size})`), { status: 400 });
+ const cid = h.digest('hex');
+ if (tk.cid && cid !== tk.cid) throw Object.assign(new Error('content hash mismatch'), { status: 400 });
+ let probe;
+ try { probe = await validateMedia(path, 0, { codecs: ['h264', 'hevc'] }); }
+ catch (e) { throw Object.assign(new Error(e.message), { status: 422 }); }
+ const adm = await admitFile(
+ { path, cid, videoId: tk.videoId, size: got, height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta: tk.meta, origin: 'intake' },
+ { cfg, upsertContent: p2pDb.upsertContent, log },
+ );
+ if (!adm.ok) throw Object.assign(new Error('not admitted: ' + adm.reason), { status: 422 });
+ await p2pDb.upsertHolder({ cid, deviceId: d.device_id, trust: 'challenged', now: now() });
+ let adopted = false;
+ if (adopt) {
+ try { adopted = !!(await adopt(tk.videoId, path, { sha256: cid, probe, meta: tk.meta })).adopted; keep = adopted; }
+ catch (e) { log.warn?.(`[p2p] adopt ${tk.videoId} failed: ${e.message}`); }
+ }
+ log.info?.(`[p2p] intake ${tk.videoId} ${cid.slice(0, 12)} admitted${adopted ? ' + adopted' : ''}`);
+ return c.json({ ok: true, cid, adopted });
+ } catch (e) {
+ return c.json({ ok: false, error: e.message }, e.status || 500);
+ } finally {
+ tickets.delete(ticket);
+ active--;
+ if (!keep) { try { unlinkSync(path); } catch { /* never written */ } }
+ }
+ });
+
+ return { openTickets: () => tickets.size, active: () => active };
+}
--- /dev/null
+++ b/server/p2p-intake.test.js
@@ -0,0 +1,103 @@
+// Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg.
+import { test, expect, beforeAll } from 'bun:test';
+import { mkdtempSync, readFileSync, readdirSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createHash } from 'node:crypto';
+import { Hono } from 'hono';
+
+const root = mkdtempSync(join(tmpdir(), 'ytp-intake-'));
+process.env.DB_PATH = join(root, 'test.db');
+const dbmod = await import('./db.js');
+const p2pDb = await import('./p2p-db.js');
+const { registerP2pRoutes } = await import('./p2p-routes.js');
+const { registerIntakeRoutes } = await import('./p2p-intake.js');
+const { admitFile } = await import('./p2p-admit.js');
+const { validateMedia } = await import('./media-cache.js');
+
+const quiet = { warn() {}, info() {} };
+const intakeDir = join(root, 'intake');
+const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir, intakeMaxBytes: 50 * 1024 * 1024 };
+let app;
+let dev;
+let good;
+let goodCid;
+const adopted = [];
+
+const req = (path, { method = 'GET', body, raw } = {}) => app.request(path, {
+ method,
+ headers: { ...(raw ? {} : { 'Content-Type': 'application/json' }), 'X-Device': dev.deviceId + '.' + dev.secret },
+ body: raw || (body ? JSON.stringify(body) : undefined),
+});
+
+beforeAll(async () => {
+ await dbmod.initDb();
+ await p2pDb.initP2pSchema();
+ const f = join(root, 'good.mp4');
+ const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
+ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', f]);
+ if (r.status !== 0) throw new Error('ffmpeg fixture failed: ' + r.stderr);
+ good = readFileSync(f);
+ goodCid = createHash('sha256').update(good).digest('hex');
+ app = new Hono();
+ const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ registerIntakeRoutes(app, {
+ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet,
+ adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256, meta: info.meta }); return { adopted: false }; },
+ });
+ dev = await (await app.request('/api/p2p/device', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' })).json();
+});
+
+test('a valid file is hashed by the server, validated, admitted and its uploader becomes a holder', async () => {
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, title: 'Song <b>x</b>', channel: 'Choir' } })).json();
+ expect(t.ok).toBe(true);
+ const r = await req(t.url, { method: 'PUT', raw: good });
+ expect(r.status).toBe(200);
+ expect(await r.json()).toEqual({ ok: true, cid: goodCid, adopted: false });
+ expect((await p2pDb.getContent(goodCid))).toMatchObject({ origin: 'intake', status: 'verified', scan: 'skipped', video_id: 'upAAAAAAAA1' });
+ expect((await p2pDb.listHolders(goodCid))[0]).toMatchObject({ device_id: dev.deviceId, trust: 'challenged' });
+ expect(adopted).toEqual([{ videoId: 'upAAAAAAAA1', sha256: goodCid, meta: { title: 'Song b x /b', channel: 'Choir' } }]);
+ expect(JSON.parse((await p2pDb.getContent(goodCid)).meta).title).toBe('Song b x /b');
+ expect(readdirSync(intakeDir)).toEqual([]); // not adopted → deleted
+ // Already verified → nothing to upload next time.
+ expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true });
+});
+
+test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => {
+ const cases = [
+ [{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/],
+ [{ bytes: good.subarray(0, Math.floor(good.length * 0.6)) }, 422, /validation/],
+ [{ bytes: Buffer.alloc(200 * 1024, 7) }, 422, /validation/],
+ ];
+ for (const [c, status, msg] of cases) {
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA2', cid: c.cid, size: c.bytes.length } })).json();
+ const r = await req(t.url, { method: 'PUT', raw: c.bytes });
+ expect(r.status).toBe(status);
+ expect((await r.json()).error).toMatch(msg);
+ }
+ expect(readdirSync(intakeDir)).toEqual([]);
+ expect((await p2pDb.listContentForVideo('upAAAAAAAA2')).length).toBe(0);
+});
+
+test('size limits and tickets are enforced', async () => {
+ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: cfg.intakeMaxBytes + 1 } })).status).toBe(413);
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: 10 } })).json();
+ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA4', size: 10 } })).status).toBe(429); // one open per device
+ const r = await req(t.url, { method: 'PUT', raw: Buffer.alloc(11) });
+ expect(r.status).toBe(413);
+ expect((await req(t.url, { method: 'PUT', raw: Buffer.alloc(10) })).status).toBe(404); // ticket consumed
+});
+
+test('malware scan on: an infected verdict is never admitted', async () => {
+ const cfg2 = { ...cfg, malwareScan: true, scanCmd: 'false' }; // exit 1 = infected
+ const app2 = new Hono();
+ const p2p = registerP2pRoutes(app2, { cfg: cfg2, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ registerIntakeRoutes(app2, { cfg: cfg2, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet });
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
+ const t = await (await app2.request('/api/p2p/intake', { method: 'POST', headers: { ...h, 'Content-Type': 'application/json' }, body: JSON.stringify({ videoId: 'upAAAAAAAA5', size: good.length }) })).json();
+ await p2pDb.revokeContent(goodCid); // make sure it is not simply "already known"
+ const r = await app2.request(t.url, { method: 'PUT', headers: h, body: good });
+ expect(r.status).toBe(422);
+ expect((await r.json()).error).toMatch(/scan infected/);
+});

280
plans/patches/017-p2p-transfer-new.diff Normal file
View File

@@ -0,0 +1,280 @@
--- /dev/null
+++ b/frontend/p2p-transfer.js
@@ -0,0 +1,184 @@
+/* ============================================================================
+ * p2p-transfer.js — download a verified file from another device over a
+ * WebRTC data channel (docs/p2p-architecture.md flow 6). window.P2PTransfer.
+ *
+ * start({ canShare }) serve requests from other devices (called once)
+ * download({ videoId, cid, size, peers, onProgress }) → { ok, error? }
+ *
+ * Signalling rides P2PClient.signal()/onMessage('signal') over /ws/p2p:
+ * {k:'offer', xid, cid, sdp} · {k:'answer', xid, sdp} · {k:'ice', xid, cand}
+ * {k:'deny', xid, reason}
+ * Data channel "file" (ordered): requester sends {t:'get'}; sender answers
+ * {t:'meta', size}, 64 KiB binary frames (back-pressured on bufferedAmount),
+ * then {t:'end'}. The receiver writes through p2p-recv-worker.js, which only
+ * commits the file when its SHA-256 equals the cid. STUN only — devices
+ * behind strict NATs won't connect (same as watch-party voice); the next
+ * holder is tried.
+ * ========================================================================== */
+(function () {
+ 'use strict';
+
+ const ICE = [{ urls: 'stun:stun.l.google.com:19302' }, { urls: 'stun:stun1.l.google.com:19302' }];
+ const FRAME = 64 * 1024;
+ const HIGH_WATER = 4 * 1024 * 1024;
+ const OPEN_TIMEOUT = 20_000;
+ const IDLE_TIMEOUT = 30_000;
+ const MAX_SERVING = 1;
+
+ let iceServers = ICE;
+ let canShare = () => true;
+ const sessions = new Map(); // xid -> { pc, onSignal }
+ let serving = 0;
+
+ const newXid = () => Math.random().toString(36).slice(2) + Date.now().toString(36);
+
+ function onSignal(m) {
+ const d = m && m.data;
+ if (!d || !d.xid) return;
+ const s = sessions.get(d.xid);
+ if (s) { s.onSignal(d, m.from); return; }
+ if (d.k === 'offer') serve(d, m.from).catch(() => {});
+ }
+
+ // ---- sender -------------------------------------------------------------------
+ async function serve(offer, from) {
+ const deny = (reason) => window.P2PClient.signal(from, { k: 'deny', xid: offer.xid, reason });
+ if (!canShare()) return deny('sharing off');
+ if (serving >= MAX_SERVING) return deny('busy');
+ const rec = await window.DeviceDB.getByCid(offer.cid);
+ const file = rec && window.OPFS.getFileObject ? await window.OPFS.getFileObject(rec.videoId) : null;
+ if (!rec || !file || file.size !== rec.size) return deny('not here');
+ serving++;
+ const pc = new RTCPeerConnection({ iceServers });
+ let done = false;
+ const finish = () => {
+ if (done) return;
+ done = true;
+ serving--;
+ sessions.delete(offer.xid);
+ try { pc.close(); } catch { /* closed */ }
+ };
+ sessions.set(offer.xid, {
+ pc,
+ onSignal: (d) => { if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {}); },
+ });
+ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(from, { k: 'ice', xid: offer.xid, cand: e.candidate.toJSON() }); };
+ pc.onconnectionstatechange = () => { if (['failed', 'closed', 'disconnected'].includes(pc.connectionState)) finish(); };
+ setTimeout(() => { if (pc.connectionState !== 'connected') finish(); }, OPEN_TIMEOUT);
+ pc.ondatachannel = (e) => {
+ const dc = e.channel;
+ dc.bufferedAmountLowThreshold = 1024 * 1024;
+ dc.onmessage = async (ev) => {
+ let msg;
+ try { msg = JSON.parse(ev.data); } catch { return; }
+ if (msg.t !== 'get') return;
+ try {
+ dc.send(JSON.stringify({ t: 'meta', size: file.size }));
+ for (let pos = 0; pos < file.size && !done; pos += 4 * FRAME) {
+ const buf = await file.slice(pos, pos + 4 * FRAME).arrayBuffer();
+ for (let i = 0; i < buf.byteLength; i += FRAME) {
+ if (dc.bufferedAmount > HIGH_WATER) {
+ await new Promise((r) => { dc.onbufferedamountlow = () => { dc.onbufferedamountlow = null; r(); }; });
+ }
+ dc.send(buf.slice(i, i + FRAME));
+ }
+ }
+ dc.send(JSON.stringify({ t: 'end' }));
+ } catch { finish(); }
+ };
+ dc.onclose = finish;
+ };
+ await pc.setRemoteDescription({ type: 'offer', sdp: offer.sdp });
+ const answer = await pc.createAnswer();
+ await pc.setLocalDescription(answer);
+ window.P2PClient.signal(from, { k: 'answer', xid: offer.xid, sdp: answer.sdp });
+ }
+
+ // ---- receiver -----------------------------------------------------------------
+ function tryPeer({ videoId, cid, size, peer, onProgress }) {
+ return new Promise((resolve) => {
+ const xid = newXid();
+ const pc = new RTCPeerConnection({ iceServers });
+ const worker = new Worker('/p2p-recv-worker.js');
+ let settled = false;
+ let idle = null;
+ let expected = size;
+ const end = (res) => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(idle);
+ clearTimeout(openTimer);
+ sessions.delete(xid);
+ try { pc.close(); } catch { /* closed */ }
+ if (!res.ok) worker.postMessage({ op: 'abort' });
+ setTimeout(() => worker.terminate(), res.ok ? 0 : 2000);
+ resolve(res);
+ };
+ const bump = () => { clearTimeout(idle); idle = setTimeout(() => end({ ok: false, error: 'peer went quiet' }), IDLE_TIMEOUT); };
+ const openTimer = setTimeout(() => end({ ok: false, error: 'could not connect' }), OPEN_TIMEOUT);
+
+ worker.onmessage = (e) => {
+ const m = e.data || {};
+ if (m.op === 'progress' && onProgress) onProgress(m.received, expected);
+ if (m.op === 'done') end(m.ok ? { ok: true, sha256: m.sha256, size: m.size } : { ok: false, error: m.error });
+ };
+ worker.postMessage({ op: 'open', videoId });
+
+ sessions.set(xid, {
+ pc,
+ onSignal: (d) => {
+ if (d.k === 'answer') pc.setRemoteDescription({ type: 'answer', sdp: d.sdp }).catch(() => end({ ok: false, error: 'bad answer' }));
+ else if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {});
+ else if (d.k === 'deny') end({ ok: false, error: 'peer declined: ' + d.reason });
+ },
+ });
+ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(peer, { k: 'ice', xid, cand: e.candidate.toJSON() }); };
+ const dc = pc.createDataChannel('file', { ordered: true });
+ dc.binaryType = 'arraybuffer';
+ dc.onopen = () => { clearTimeout(openTimer); bump(); dc.send(JSON.stringify({ t: 'get' })); };
+ dc.onmessage = (e) => {
+ bump();
+ if (typeof e.data === 'string') {
+ let m;
+ try { m = JSON.parse(e.data); } catch { return; }
+ if (m.t === 'meta') {
+ if (m.size !== size) end({ ok: false, error: 'peer has a different file size' });
+ expected = m.size;
+ } else if (m.t === 'end') {
+ clearTimeout(idle);
+ worker.postMessage({ op: 'finish', cid, size });
+ }
+ return;
+ }
+ worker.postMessage({ op: 'chunk', buf: e.data }, [e.data]);
+ };
+ dc.onclose = () => { if (!settled) setTimeout(() => end({ ok: false, error: 'channel closed' }), 5000); };
+ (async () => {
+ try {
+ const offer = await pc.createOffer();
+ await pc.setLocalDescription(offer);
+ if (!window.P2PClient.signal(peer, { k: 'offer', xid, cid, sdp: offer.sdp })) end({ ok: false, error: 'not connected to the P2P hub' });
+ } catch (err) { end({ ok: false, error: err.message }); }
+ })();
+ });
+ }
+
+ // Try each online holder in turn until one delivers a verified file.
+ async function download({ videoId, cid, size, peers, onProgress }) {
+ if (!window.P2PClient || !window.P2PClient.isConnected()) return { ok: false, error: 'not connected to the P2P hub' };
+ let last = { ok: false, error: 'no online device has this video' };
+ for (const p of peers || []) {
+ last = await tryPeer({ videoId, cid, size, peer: p.peer, onProgress });
+ if (last.ok) return last;
+ }
+ return last;
+ }
+
+ function start(opts = {}) {
+ if (opts.canShare) canShare = opts.canShare;
+ if (opts.iceServers) iceServers = opts.iceServers;
+ window.P2PClient.onMessage('signal', onSignal);
+ }
+
+ window.P2PTransfer = { start, download };
+}());
--- /dev/null
+++ b/frontend/p2p-recv-worker.js
@@ -0,0 +1,90 @@
+/* ============================================================================
+ * p2p-recv-worker.js — writes a file arriving from another device into OPFS
+ * while hashing it; commits it ONLY when the SHA-256 equals the expected
+ * content id (docs/p2p-architecture.md flow 6).
+ *
+ * In: { op:'open', videoId } start videos/<videoId>.p2p.part
+ * { op:'chunk', buf } ArrayBuffer (transferred)
+ * { op:'finish', cid, size } verify + rename to <videoId>.mp4
+ * { op:'abort' } drop the partial file
+ * Out: { op:'opened' } | { op:'progress', received } |
+ * { op:'done', ok:true, sha256, size } | { op:'done', ok:false, error }
+ * The ".part" suffix keeps OPFS.listVideos() from ever listing a partial file.
+ * ========================================================================== */
+'use strict';
+importScripts('/sha256.js');
+
+let dir = null;
+let handle = null;
+let access = null;
+let partName = '';
+let finalName = '';
+let hasher = null;
+let offset = 0;
+let lastProgress = 0;
+
+async function cleanup() {
+ try { if (access) access.close(); } catch { /* closed */ }
+ access = null;
+ try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ }
+}
+
+// Messages are handled strictly one after another: an async handler would
+// otherwise let 'chunk' or 'abort' run while 'open' is still awaiting OPFS.
+let chain = Promise.resolve();
+self.onmessage = (e) => { chain = chain.then(() => onOp(e.data || {})); };
+
+async function onOp(m) {
+ try {
+ if (m.op === 'open') {
+ const root = await navigator.storage.getDirectory();
+ dir = await root.getDirectoryHandle('videos', { create: true });
+ partName = `${m.videoId}.p2p.part`;
+ finalName = `${m.videoId}.mp4`;
+ handle = await dir.getFileHandle(partName, { create: true });
+ access = await handle.createSyncAccessHandle();
+ access.truncate(0);
+ hasher = self.Sha256.create();
+ offset = 0;
+ self.postMessage({ op: 'opened' });
+ } else if (m.op === 'chunk') {
+ const u8 = new Uint8Array(m.buf);
+ access.write(u8, { at: offset });
+ hasher.update(u8);
+ offset += u8.byteLength;
+ if (offset - lastProgress > 1024 * 1024) { lastProgress = offset; self.postMessage({ op: 'progress', received: offset }); }
+ } else if (m.op === 'finish') {
+ access.truncate(offset);
+ access.flush();
+ access.close();
+ access = null;
+ const sha256 = hasher.hex();
+ if (offset !== m.size) throw new Error(`size mismatch (${offset} of ${m.size})`);
+ if (sha256 !== m.cid) throw new Error('content hash mismatch');
+ try { await dir.removeEntry(finalName); } catch { /* no previous copy */ }
+ let renamed = false;
+ if (typeof handle.move === 'function') { try { await handle.move(finalName); renamed = true; } catch { /* copy below */ } }
+ if (!renamed) {
+ const out = await (await dir.getFileHandle(finalName, { create: true })).createSyncAccessHandle();
+ try {
+ const file = await handle.getFile();
+ for (let pos = 0; pos < file.size; pos += 8 * 1024 * 1024) {
+ const buf = new Uint8Array(await file.slice(pos, pos + 8 * 1024 * 1024).arrayBuffer());
+ out.write(buf, { at: pos });
+ }
+ out.truncate(file.size);
+ out.flush();
+ } finally { out.close(); }
+ await dir.removeEntry(partName);
+ }
+ partName = '';
+ self.postMessage({ op: 'done', ok: true, sha256, size: offset });
+ } else if (m.op === 'abort') {
+ await cleanup();
+ self.postMessage({ op: 'done', ok: false, error: 'aborted' });
+ }
+ } catch (err) {
+ await cleanup();
+ self.postMessage({ op: 'done', ok: false, error: err && err.message ? err.message : String(err) });
+ }
+}

40
plans/patches/018-client-restore.diff Normal file
View File

@@ -0,0 +1,40 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -160,7 +160,7 @@
// Send one saved video to the server so it can hash + validate it and add its
// cid to the catalog. Uploads the whole file: only on the user's request
// ("Verify & share") or when the server asks for a copy (plan 018).
- async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) {
+ async function contribute(videoId, { cid = null, title = '', channel = '', restore = false } = {}) {
const dev = await ensureDevice();
const rec = await window.DeviceDB.getFile(videoId);
const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null;
@@ -168,7 +168,7 @@
const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
const t = await (await fetch('/api/p2p/intake', {
method: 'POST', headers: { ...h, 'Content-Type': 'application/json' },
- body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }),
+ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel, restore }),
})).json().catch(() => ({ ok: false, error: 'server unreachable' }));
if (!t.ok || t.known) { if (t.known) changed(); return t; }
const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file }))
@@ -280,6 +280,19 @@
timer = setTimeout(sync, 5000);
}
+ // Plan 018: the server lost its copy and the source is gone — it asks one
+ // holder to send the file back. Only while sharing is on, one at a time,
+ // and only for the exact cid this device holds.
+ let restoring = false;
+ onMessage('upload-request', async (m) => {
+ if (restoring || hooks.getSettings().p2pShare === false) return;
+ const rec = window.DeviceDB ? await window.DeviceDB.getFile(m.videoId) : null;
+ if (!rec || rec.cid !== m.cid) return;
+ restoring = true;
+ try { await contribute(m.videoId, { cid: m.cid, restore: true }); } catch { /* next request retries */ }
+ finally { restoring = false; }
+ });
+
function start(h) {
hooks = { ...hooks, ...(h || {}) };
const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1));

125
plans/patches/018-server-rehydrate.diff Normal file
View File

@@ -0,0 +1,125 @@
--- a/server/p2p-hub.js
+++ b/server/p2p-hub.js
@@ -118,3 +118,28 @@
}
return { ok: true, staleDays, cids: out };
}
+
+// Flow 8 (plan 018): the source is gone and the server evicted its copy, so
+// ask ONE online holder that shares to upload it through intake
+// ({type:'upload-request', videoId, cid}). At most one request per video per
+// 10 minutes; returns true when a device was asked (or recently was).
+export function createRehydrator({ p2pDb, hub, hasServerCopy, enabled = () => true, now = () => Date.now() }) {
+ const asked = new Map(); // videoId -> ms
+ return async function rehydrate(videoId) {
+ if (!enabled()) return false;
+ if (await hasServerCopy(videoId)) return false;
+ const last = asked.get(videoId);
+ if (last && now() - last < 10 * 60_000) return true;
+ for (const c of await p2pDb.listContentForVideo(videoId)) {
+ for (const h of await p2pDb.listHolders(c.cid, 50)) {
+ if (Number(h.share) !== 1 || !hub.isOnline(h.device_id)) continue;
+ if (hub.send(h.device_id, { type: 'upload-request', videoId, cid: c.cid })) {
+ asked.set(videoId, now());
+ if (asked.size > 5000) asked.delete(asked.keys().next().value);
+ return true;
+ }
+ }
+ }
+ return false;
+ };
+}
--- a/server/p2p-hub.test.js
+++ b/server/p2p-hub.test.js
@@ -85,3 +85,21 @@
]);
expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids
});
+
+test('rehydrator asks one online sharing holder, once per 10 min, only when the server has no copy', async () => {
+ const { createRehydrator } = await import('./p2p-hub.js');
+ const sent = [];
+ let t = NOW;
+ let serverHas = false;
+ const hub = { isOnline: (d) => d === 'dev_000000000000000a' || d === 'dev_000000000000000c', send: (d, m) => { sent.push([d, m]); return true; } };
+ const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => serverHas, now: () => t });
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true);
+ // c is online but has sharing off; a is online and shares.
+ expect(sent).toEqual([['dev_000000000000000a', { type: 'upload-request', videoId: 'dQw4w9WgXcQ', cid: CID }]]);
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true); // recently asked — no second message
+ expect(sent.length).toBe(1);
+ t += 11 * 60_000;
+ serverHas = true;
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(false); // server copy is back
+ expect(await rehydrate('unknownVid1')).toBe(false); // nobody holds it
+});
--- a/server/p2p-intake.js
+++ b/server/p2p-intake.js
@@ -3,8 +3,10 @@
* (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did
* not come from the server's own fetch can become verified content.
*
- * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? }
+ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel?, restore? }
* → { ok, known:true } cid already verified — nothing to send
+ * (unless restore:true and the server
+ * lost its copy — plan 018)
* → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min
* PUT /api/p2p/intake/:ticket (device) raw file body
* → { ok, cid, adopted } admitted (+ adopted into the media cache)
@@ -30,6 +32,7 @@
export function registerIntakeRoutes(app, deps) {
const {
cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null,
+ serverHasCid = async () => true, // (cid) → does the server still hold these bytes?
now = () => Date.now(), log = console,
} = deps;
const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy }
@@ -48,7 +51,8 @@
if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413);
if (cid) {
const known = await p2pDb.getContent(cid);
- if (known && known.status === 'verified') return c.json({ ok: true, known: true });
+ const restore = body.restore === true && known && known.status === 'verified' && !(await serverHasCid(cid));
+ if (known && known.status === 'verified' && !restore) return c.json({ ok: true, known: true });
if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410);
}
sweep();
--- a/server/p2p-intake.test.js
+++ b/server/p2p-intake.test.js
@@ -1,6 +1,6 @@
// Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg.
import { test, expect, beforeAll } from 'bun:test';
-import { mkdtempSync, readFileSync, readdirSync } from 'node:fs';
+import { mkdtempSync, readFileSync, readdirSync, unlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
@@ -64,6 +64,27 @@
expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true });
});
+test('restore: a known cid is uploaded again only when the server lost its copy', async () => {
+ let serverHas = true;
+ const app3 = new Hono();
+ const p2p = registerP2pRoutes(app3, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ const got = [];
+ registerIntakeRoutes(app3, {
+ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet,
+ serverHasCid: async () => serverHas,
+ adopt: async (videoId, path, info) => { got.push(info.sha256); unlinkSync(path); return { adopted: true }; }, // a real adopt moves the file
+ });
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret, 'Content-Type': 'application/json' };
+ const open = async () => (await app3.request('/api/p2p/intake', { method: 'POST', headers: h, body: JSON.stringify({ videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, restore: true }) })).json();
+ expect(await open()).toEqual({ ok: true, known: true }); // server still has it
+ serverHas = false;
+ const t = await open();
+ expect(t.ticket).toMatch(/^[0-9a-f]{32}$/);
+ const r = await (await app3.request(t.url, { method: 'PUT', headers: { 'X-Device': h['X-Device'] }, body: good })).json();
+ expect(r).toEqual({ ok: true, cid: goodCid, adopted: true });
+ expect(got).toEqual([goodCid]);
+});
+
test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => {
const cases = [
[{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/],

0
plans/queue/.gitkeep Normal file
View File

123
plans/queue/001-perf-timing-marks-105acc.md Normal file
View File

@@ -0,0 +1,123 @@
---
id: 001-perf-timing-marks-105acc
title: Add startup, search and play timing marks plus yt-dlp duration logs
created: 2026-09-29
depends_on: []
est_files: 2
---
# 001 — Add startup, search and play timing marks plus yt-dlp duration logs
## Objective
Every later speed plan needs a before/after number. After this plan:
- The browser records `performance.measure` entries `ytp:boot`, `ytp:search`,
`ytp:tap-to-play`, and `window.__ytpPerf()` returns their latest values in ms.
- The server logs one line per yt-dlp call: `[ytdlp] <kind> <ms>ms ok|fail`.
Measured baseline (prod, 2026-09-29): search 4.1–5.0 s, first play `/api/streams` 7.5 s,
second play 1.2 s, `/api/version` 1.25 s.
## Context the executor must NOT rediscover
`server/server.js:143-165` — the only place yt-dlp is spawned:
```js
function runYtdlp(args, { signal } = {}) {
return new Promise((resolve, reject) => {
const child = spawn(YTDLP, args, { stdio: ['ignore', 'pipe', 'pipe'] });
...
child.on('error', (e) => reject(new Error('yt-dlp not found: ' + e.message)));
child.on('close', (code) => {
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
});
}
```
`frontend/app.js`:
- `async function boot()` starts at ~line 9749 (`wirePlayerEvents();` is its first line).
The first `render()` call inside boot happens after the `hasShareParam` block.
- `async function runSearchQuery(q, { instant = null } = {})` at ~line 8583; the
success path ends with `RecentSearches.cacheResults(q, results);`.
- `Player.loadVideo(videoObj, …)` at ~line 1650; first statement is
`if (!this._handoff) Transition.cancel();`.
- The master element `playing` listener at ~line 2287:
```js
el.addEventListener('playing', () => {
if (!masterIs(el)) return;
showSpinner(false);
```
## Steps
1. `server/server.js` — in `runYtdlp`, directly after the `const child = spawn(...)` line add:
```js
const t0 = Date.now();
const kind = String(args.find((a) => /^ytsearch|^https?:/.test(String(a))) || args[0] || '')
.replace(/^ytsearch\d*:.*/, 'search').replace(/^https?:\/\/[^/]+\/watch.*/, 'video').slice(0, 40);
```
and replace the `child.on('close', …)` handler body with:
```js
child.on('close', (code) => {
console.log(`[ytdlp] ${kind} ${Date.now() - t0}ms ${code === 0 ? 'ok' : 'fail'}`);
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
```
2. `frontend/app.js` — near the top of the file, directly after the line
`const APP_VERSION = '1.0.0';` (~line 24), add:
```js
// Timing marks for the speed work (plans/). performance.measure entries are
// visible in DevTools → Performance; __ytpPerf() prints the latest ones.
function perfMark(name) { try { performance.mark(name); } catch { /* old browser */ } }
function perfMeasure(name, start) {
try { performance.measure(name, start); } catch { /* start mark missing */ }
}
window.__ytpPerf = () => {
const out = {};
try { for (const m of performance.getEntriesByType('measure')) if (m.name.startsWith('ytp:')) out[m.name] = Math.round(m.duration); } catch { /* none */ }
return out;
};
```
3. `frontend/app.js` `boot()` — first line of the function body: `perfMark('ytp:boot-start');`.
Immediately after the FIRST `render();` call inside `boot()` add
`perfMeasure('ytp:boot', 'ytp:boot-start');`.
4. `frontend/app.js` `runSearchQuery` — after `const mySeq = ++searchSeq;` add
`perfMark('ytp:search-start');`. After `RecentSearches.cacheResults(q, results);` add
`perfMeasure('ytp:search', 'ytp:search-start');`.
5. `frontend/app.js` `Player.loadVideo` — first line of the body: `perfMark('ytp:tap');`.
6. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` add:
```js
if (performance.getEntriesByName('ytp:tap').length) {
perfMeasure('ytp:tap-to-play', 'ytp:tap');
try { performance.clearMarks('ytp:tap'); } catch { /* ignore */ }
}
```
## Out of scope / do NOT touch
- No reporting endpoint, no UI. Don't change any behaviour, only add marks/logs.
- Do not touch `runYtdlpResilient` or the fallback-client logic.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
grep -c "perfMark\|perfMeasure" frontend/app.js
```
Expected: no syntax errors, `SERVER_OK`, tests `fail 0`, grep count ≥ 8.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

156
plans/queue/002-compress-and-etag-shell-bd459c.md Normal file
View File

@@ -0,0 +1,156 @@
---
id: 002-compress-and-etag-shell-bd459c
title: Serve the app shell gzip/brotli-compressed with ETags
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 2
---
# 002 — Serve the app shell gzip/brotli-compressed with ETags
## Objective
Measured on prod: `app.js` is sent as 426 244 bytes with no `content-encoding`
and no `ETag` (2.8 s download at ~150 KB/s). gzip -9 makes it 117 527 bytes;
the whole shell drops from 618 KB to ~158 KB. After this plan:
- Every static text file under `./public` (`.js .css .html .json .webmanifest .svg`)
is served compressed (`br` preferred, else `gzip`) when the client accepts it,
with `Vary: Accept-Encoding`.
- Every static file carries a strong `ETag` and `If-None-Match` returns `304`.
- `index.html` (build-stamped at request time) and `/sw.js` (BUILD_TAG-injected) are
also compressed + ETagged using the text they actually send.
- `Cache-Control` values stay EXACTLY as they are today (`no-cache` for the shell,
`no-store` for sw.js) — CLAUDE.md "Update-flow architecture" depends on that.
## Context the executor must NOT rediscover
`server/server.js:1895-1910` today:
```js
function indexHtml(c) {
if (_indexSource === null) {
try { _indexSource = readFileSync('./public/index.html', 'utf8'); }
catch { return c.text('index.html not found', 404); }
}
return c.html(_indexSource.replace('__BUILD_TAG__', BUILD_TAG), 200, { 'Cache-Control': 'no-cache' });
}
app.get('/', indexHtml);
app.get('/index.html', indexHtml);
...
app.use('/*', serveStatic({ root: './public', onFound: (_path, c) => { c.header('Cache-Control', 'no-cache'); } }));
// SPA fallback — return index.html for any unmatched path
app.get('/*', indexHtml);
```
`/sw.js` handler at ~line 1860 ends with `return c.text(src, 200, { ...no-store headers })`.
Bun provides `Bun.gzipSync(buf, { level: 9 })`. Brotli: `import { brotliCompressSync, constants } from 'node:zlib'`
(works in Bun). `createHash` is already imported from `node:crypto` at the top.
**Never compress** `/api/play`, `/api/media/*`, `/api/download/*` (Range/binary) —
they are not served by `serveStatic`, so a static-only middleware cannot touch them.
## Steps
1. `server/server.js` — add near the top imports:
`import { brotliCompressSync, constants as zlibConstants } from 'node:zlib';`
2. `server/server.js` — directly ABOVE `function indexHtml(c) {` add this helper block:
```js
// Compressed + ETagged static text. The shell is ~620 KB raw / ~160 KB gzip
// and every byte crosses the slow VPS→homelab link, so compress once per
// file content and keep it in memory. ETag = sha256 of the RAW bytes, so a
// `no-cache` revalidation costs a 304 instead of the whole file.
const COMPRESSIBLE = /\.(js|css|html|json|webmanifest|svg|txt)$/i;
const compressedCache = new Map(); // key -> { etag, raw, gz, br, type }
function compressedEntry(key, raw, type) {
let e = compressedCache.get(key);
const etag = '"' + createHash('sha256').update(raw).digest('hex').slice(0, 32) + '"';
if (e && e.etag === etag) return e;
e = {
etag, raw, type,
gz: Bun.gzipSync(raw, { level: 9 }),
br: brotliCompressSync(raw, { params: { [zlibConstants.BROTLI_PARAM_QUALITY]: 11 } }),
};
compressedCache.set(key, e);
return e;
}
function sendCompressed(c, e, cacheControl) {
const headers = { 'Content-Type': e.type, 'Cache-Control': cacheControl, ETag: e.etag, Vary: 'Accept-Encoding' };
const inm = c.req.header('if-none-match') || '';
if (inm.split(',').map((s) => s.trim()).includes(e.etag)) return new Response(null, { status: 304, headers });
const ae = c.req.header('accept-encoding') || '';
if (/\bbr\b/.test(ae)) return new Response(e.br, { headers: { ...headers, 'Content-Encoding': 'br' } });
if (/\bgzip\b/.test(ae)) return new Response(e.gz, { headers: { ...headers, 'Content-Encoding': 'gzip' } });
return new Response(e.raw, { headers });
}
const MIME = { js: 'text/javascript; charset=utf-8', css: 'text/css; charset=utf-8', html: 'text/html; charset=utf-8',
json: 'application/json', webmanifest: 'application/manifest+json', svg: 'image/svg+xml', txt: 'text/plain; charset=utf-8' };
```
3. `server/server.js` — change `indexHtml` so its return line becomes:
```js
const html = _indexSource.replace('__BUILD_TAG__', BUILD_TAG);
return sendCompressed(c, compressedEntry('index.html', Buffer.from(html), MIME.html), 'no-cache');
```
4. `server/server.js` — in the `/sw.js` handler, replace its final return, which is exactly:
```js
return c.text(src, 200, {
'Content-Type': 'application/javascript; charset=utf-8',
'Cache-Control': 'no-store, no-cache, must-revalidate',
});
```
with:
```js
return sendCompressed(c, compressedEntry('sw.js', Buffer.from(src), MIME.js), 'no-store, no-cache, must-revalidate');
```
(`text/javascript` is a valid service-worker MIME type.) Dry-run result: app.js 426 244 → 93 717 bytes (br).
5. `server/server.js` — directly BEFORE the `app.use('/*', serveStatic(...))` line add:
```js
app.get('/*', async (c, next) => {
const p = decodeURIComponent(new URL(c.req.url).pathname);
if (!COMPRESSIBLE.test(p) || p.includes('..') || p.startsWith('/api/')) return next();
const file = Bun.file('./public' + p);
if (!(await file.exists())) return next();
const raw = Buffer.from(await file.arrayBuffer());
const ext = p.slice(p.lastIndexOf('.') + 1).toLowerCase();
return sendCompressed(c, compressedEntry(p, raw, MIME[ext] || 'application/octet-stream'), 'no-cache');
});
```
(`/sw.js`, `/` and `/index.html` are registered earlier and win; this only covers the rest.)
## Out of scope / do NOT touch
- `frontend/sw.js`, `frontend/sw-update.js`: no changes. The SW fetches shell files with
`cache: 'reload'` and `?__ytpfresh=` — query strings don't affect the pathname match, fine.
- Do not change any `Cache-Control` value. Do not add `hono/compress` globally (it would
hit Range media responses).
- Do not touch `computeBuildTag`.
## Verification
```bash
cd /home/user/ytplayer/server && [ -e public ] || ln -s ../frontend public
PORT=3999 bun server.js > /tmp/ytp002.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip, br' http://localhost:3999/app.js | grep -iE 'content-encoding|etag|cache-control|vary'
ET=$(curl -s -D - -o /dev/null http://localhost:3999/app.js | grep -i '^etag' | cut -d' ' -f2 | tr -d '\r')
curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" http://localhost:3999/app.js
curl -s -H 'Accept-Encoding: gzip' --compressed http://localhost:3999/ | grep -c 'ytp-build'
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip' http://localhost:3999/sw.js | grep -iE 'content-encoding|cache-control'
curl -s --compressed http://localhost:3999/sw.js | grep -c "__BUILD_TAG__ !== 'undefined'" ; true
kill $SRV; true
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `content-encoding: br`, an `etag`, `cache-control: no-cache`, `vary: Accept-Encoding`;
the If-None-Match request prints `304`; index grep prints `1`; sw.js shows `content-encoding: gzip`
and its original no-store cache-control; the last grep prints `0` (tag was injected); tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

131
plans/queue/003-self-host-fonts-89466b.md Normal file
View File

@@ -0,0 +1,131 @@
---
id: 003-self-host-fonts-89466b
title: Self-host the three web fonts and drop the render-blocking Google Fonts CSS
created: 2026-09-29
depends_on: [002-compress-and-etag-shell-bd459c]
est_files: 5
---
# 003 — Self-host the three web fonts
## Objective
`frontend/index.html` loads a render-blocking stylesheet from `fonts.googleapis.com`
(3 families, 10 weights), which costs two extra origins (DNS+TLS each) before first
paint and only works offline through the SW's `ytplayer-fonts` runtime cache. After
this plan the fonts are files under `frontend/fonts/`, declared in
`frontend/fonts/fonts.css`, precached with the shell, and the display face is
preloaded. Visual result must be identical.
## Context the executor must NOT rediscover
`frontend/index.html:20-26` today:
```html
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600;12..96,700;12..96,800&family=Hanken+Grotesk:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="styles.css" />
```
CSS tokens (`frontend/styles.css:27-29`) reference the family names
`"Bricolage Grotesque"`, `"Hanken Grotesk"`, `"JetBrains Mono"` — keep those names.
`frontend/sw.js:55-71` — the `SHELL` array (precache list). Every new shell file must be
listed there or it will be missing offline. `frontend/sw.js:165` keeps a runtime rule for
the Google hosts — leave it (harmless, and old clients may still request them).
Google serves one variable `woff2` per family per subset when asked with a modern UA.
`fitLyricLines` already re-fits on `document.fonts` load (CLAUDE.md), so swap is safe.
## Steps
1. Create `scripts/fetch-fonts.js` (Node ≥18, no deps):
```js
// Downloads the app's Google fonts once (latin + latin-ext subsets) into
// frontend/fonts/ and writes frontend/fonts/fonts.css pointing at them.
// Re-run only when the font list changes.
const fs = require('node:fs');
const path = require('node:path');
const CSS_URL = 'https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600..800&family=Hanken+Grotesk:wght@400..700&family=JetBrains+Mono:wght@400..700&display=swap';
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36';
const OUT = path.join(__dirname, '..', 'frontend', 'fonts');
(async () => {
fs.mkdirSync(OUT, { recursive: true });
const css = await (await fetch(CSS_URL, { headers: { 'User-Agent': UA } })).text();
// parts[k] ends with the "/* <subset> */" comment of the block in parts[k+1].
const parts = css.split('@font-face');
let out = '/* Generated by scripts/fetch-fonts.js — do not edit by hand. */\n';
let n = 0;
for (let k = 1; k < parts.length; k++) {
const subset = (parts[k - 1].match(/\/\*\s*([\w-]+)\s*\*\/\s*$/) || [])[1];
const b = parts[k].slice(0, parts[k].indexOf('}') + 1); // just this block
if (subset !== 'latin' && subset !== 'latin-ext') continue;
const url = (b.match(/url\((https:[^)]+\.woff2)\)/) || [])[1];
const fam = (b.match(/font-family:\s*'([^']+)'/) || [])[1];
if (!url || !fam) continue;
const file = `${fam.replace(/\s+/g, '')}-${subset}.woff2`;
const buf = Buffer.from(await (await fetch(url)).arrayBuffer());
fs.writeFileSync(path.join(OUT, file), buf);
out += '@font-face' + b.replace(url, file).trimEnd() + '\n';
n++;
}
fs.writeFileSync(path.join(OUT, 'fonts.css'), out);
console.log(`wrote ${n} font files + fonts.css`);
})().catch((e) => { console.error(e); process.exit(1); });
```
Run it: `node scripts/fetch-fonts.js`. Expect `wrote 6 font files + fonts.css`
(3 families × 2 subsets). If the network is blocked, STOP and report — do not hand-write fonts.
2. Open `frontend/fonts/fonts.css` and confirm every block has `font-display: swap;`; if a
block lacks it, add it.
3. `frontend/index.html` — replace the 5 lines from `<link rel="preconnect" href="https://fonts.googleapis.com" />`
through the closing `/>` of the Google stylesheet link with:
```html
<link rel="preload" href="fonts/BricolageGrotesque-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="preload" href="fonts/HankenGrotesk-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="stylesheet" href="fonts/fonts.css" />
```
(Use the exact filenames the script produced — check with `ls frontend/fonts`.)
4. `frontend/sw.js` `SHELL` array — after `'/styles.css',` add `'/fonts/fonts.css',` and one
`'/fonts/<file>.woff2',` line per generated woff2 file.
5. `frontend/index.html` line 16 — the Content-Security-Policy `<meta>` has
`font-src https://fonts.gstatic.com data:` which would BLOCK self-hosted fonts. Change that part to
`font-src 'self' https://fonts.gstatic.com data:` (leave the rest of the policy unchanged;
`style-src` already allows `'self'`). `frontend/admin.html` has no Google Fonts link — leave it alone.
6. `package.json` scripts — add `"fetch-fonts": "node scripts/fetch-fonts.js"`.
## Out of scope / do NOT touch
- Do not rename the font families or edit `styles.css`.
- Do not remove the fonts rule in `sw.js` fetch handler or `UTILITY_CACHES`.
- Do not subset further (e.g. glyph-level subsetting) — out of scope.
## Verification
```bash
cd /home/user/ytplayer && ls -la frontend/fonts
grep -c 'href="https://fonts.googleapis' frontend/index.html
grep -c "font-src 'self'" frontend/index.html
for f in $(ls frontend/fonts/*.woff2); do grep -c "/fonts/$(basename $f)" frontend/sw.js; done
node --test frontend/*.test.js 2>&1 | tail -3
cd server && [ -e public ] || ln -s ../frontend public; PORT=3998 bun server.js >/tmp/ytp003.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:3998/fonts/fonts.css
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3998/fonts/$(ls ../frontend/fonts | grep woff2 | head -1)
kill $SRV; true
```
Expected: 6 `.woff2` + `fonts.css`; google link grep `0`; CSP grep `1`; each sw.js grep `1`; tests `fail 0`;
`200 text/css…` and `200`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` plus unified diff of text files (not the woff2 binaries).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

85
plans/queue/004-coalesce-stream-resolves-a92d40.md Normal file
View File

@@ -0,0 +1,85 @@
---
id: 004-coalesce-stream-resolves-a92d40
title: Coalesce concurrent resolveStreams calls for the same video
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 1
---
# 004 — Coalesce concurrent resolveStreams calls
## Objective
`resolveStreams(videoId)` checks `streamCache` but has no in-flight map, so two
requests for the same id that arrive before the first finishes (a warm-up + the
real play, two devices, the media cache's `getInfo` + `/api/streams`) each spawn a
~6 s `yt-dlp -J`. After this plan, concurrent callers share ONE promise; a failure
is not cached (the next call retries).
## Context the executor must NOT rediscover
`server/server.js:457-494`:
```js
async function resolveStreams(videoId) {
const now = Date.now();
const cached = streamCache.get(videoId);
if (cached && now < cached.expiresAt) return cached;
const out = await runYtdlpResilient(['-J', '--no-warnings', `https://www.youtube.com/watch?v=${videoId}`]);
const info = JSON.parse(out);
...
streamCache.set(videoId, entry);
return entry;
}
```
`streamCache` and `STREAM_CACHE_MAX` are declared just above it.
## Steps
1. `server/server.js` — rename the existing function `resolveStreams` to
`resolveStreamsUncached` (definition only; body unchanged).
2. Directly after that function add:
```js
// One yt-dlp -J per video at a time: concurrent callers (warm-up + play,
// two devices, the media cache's getInfo) share the in-flight promise.
const inflightStreams = new Map(); // videoId -> Promise<entry>
function resolveStreams(videoId) {
const cached = streamCache.get(videoId);
if (cached && Date.now() < cached.expiresAt) return Promise.resolve(cached);
let p = inflightStreams.get(videoId);
if (!p) {
p = resolveStreamsUncached(videoId).finally(() => inflightStreams.delete(videoId));
inflightStreams.set(videoId, p);
}
return p;
}
```
All existing callers keep calling `resolveStreams` (it still returns a Promise).
3. Create `server/streams-inflight.test.js`? — NO. Keep it in-file; instead verify with the
script below.
## Out of scope / do NOT touch
- Cache TTL logic, `runYtdlpResilient`, format filtering.
## Verification
```bash
cd /home/user/ytplayer/server && grep -n "function resolveStreams\|function resolveStreamsUncached\|inflightStreams" server.js
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
```
Expected: both functions + map present, `SERVER_OK`, all server test files `0 fail`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

111
plans/queue/005-warm-streams-on-intent-47b3d3.md Normal file
View File

@@ -0,0 +1,111 @@
---
id: 005-warm-streams-on-intent-47b3d3
title: Warm the stream cache for likely next plays
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40]
est_files: 2
---
# 005 — Warm the stream cache for likely next plays
## Objective
A cold `/api/streams` costs ~6 s of yt-dlp; a cached one ~0 s (1.2 s total on prod,
all network). Start that work before the user taps:
- new `GET /api/streams/warm?v=<id>` → `204` immediately, resolves in the background
(shares the in-flight promise from plan 004), at most 2 warm resolves at a time,
skipped when the server already holds a ready media copy.
- the client warms: the top 3 search results after a search renders; a card on
`pointerdown` (touch/mouse down fires ~100–300 ms before `click`); the next 2 queue
items when a song starts playing.
## Context the executor must NOT rediscover
- `server/server.js` `resolveStreams(videoId)` returns a Promise and dedupes (plan 004).
- `media.getReady(videoId)` (server/media-cache.js) resolves the ready row or null.
- `server/server.js:532` — `app.get('/api/streams', async (c) => {` — register the new
route directly ABOVE it (Hono matches `/api/streams/warm` separately anyway).
- Client `frontend/app.js`:
- `const YT_ID_RE = /^[A-Za-z0-9_-]{11}$/;` at ~line 1202 (declared later in the file
than the helper you add — fine, it is only read at call time).
- `runSearchQuery` success path (~line 8603): `searchResults = results; … renderList(); RecentSearches.cacheResults(q, results);`
- `renderCard(v, index, list)` at ~line 7725; it has `card.addEventListener('click', (e) => {` at ~line 7781.
- globals `queue` (array of video objects) and `queueIndex` (~line 353).
- master `playing` listener at ~line 2287 (`el.addEventListener('playing', () => { if (!masterIs(el)) return; …`).
- `WEB` constant is true for the PWA; `cachedIds` is a Set of ids saved on this device.
## Steps
1. `server/server.js` — above `app.get('/api/streams', …)` add:
```js
// GET /api/streams/warm?v=<id> — fire-and-forget: resolve streams into
// streamCache so the real /api/streams a moment later is instant. Bounded
// so a scrolling user can't queue dozens of yt-dlp processes.
const WARM_MAX = 2;
let warmActive = 0;
app.get('/api/streams/warm', async (c) => {
const id = (c.req.query('v') || '').trim();
if (!/^[A-Za-z0-9_-]{11}$/.test(id)) return c.body(null, 204);
if (warmActive >= WARM_MAX) return c.body(null, 204);
try { if (await media.getReady(id)) return c.body(null, 204); } catch { /* fall through */ }
warmActive++;
resolveStreams(id).catch(() => {}).finally(() => { warmActive--; });
return c.body(null, 204);
});
```
2. `frontend/app.js` — directly after the `const API = { … };` object (~line 320) add:
```js
// Pre-resolve streams for videos the user is likely to play next (server
// /api/streams/warm). Each id is warmed at most once per 20 min per tab.
const warmedAt = new Map();
function warmStreams(ids) {
if (!WEB || navigator.onLine === false) return;
const now = Date.now();
for (const id of ids) {
if (!id || !/^[A-Za-z0-9_-]{11}$/.test(id) || cachedIds.has(id)) continue;
if (now - (warmedAt.get(id) || 0) < 20 * 60_000) continue;
warmedAt.set(id, now);
fetch(`/api/streams/warm?v=${encodeURIComponent(id)}`, { priority: 'low' }).catch(() => {});
}
}
```
3. `frontend/app.js` `runSearchQuery` — after `RecentSearches.cacheResults(q, results);` add
`warmStreams(results.slice(0, 3).map((r) => r.id));`
4. `frontend/app.js` `renderCard` — directly BEFORE `card.addEventListener('click', (e) => {` add:
```js
card.addEventListener('pointerdown', () => warmStreams([v.id]), { passive: true });
```
5. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` (and after
the plan-001 perf lines if present) add:
```js
if (Array.isArray(queue) && queueIndex >= 0) warmStreams(queue.slice(queueIndex + 1, queueIndex + 3).map((x) => x && x.id));
```
## Out of scope / do NOT touch
- Do not warm on hover/scroll, do not warm uploads (`upl_…`) — they need no yt-dlp.
- Do not change `/api/streams` itself.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && echo APP_OK
cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
[ -e public ] || ln -s ../frontend public; PORT=3997 bun server.js >/tmp/ytp005.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=bad'
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=dQw4w9WgXcQ'
kill $SRV; true
cd .. && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `APP_OK`, `SERVER_OK`, `204`, `204`, tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

212
plans/queue/006-innertube-search-48066b.md Normal file
View File

@@ -0,0 +1,212 @@
---
id: 006-innertube-search-48066b
title: Answer searches from YouTube InnerTube directly with yt-dlp fallback
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 4
---
# 006 — InnerTube search with yt-dlp fallback
## Objective
`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process
(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search
API returns the same data in a few hundred ms. After this plan `/api/search` tries
InnerTube first (6 s timeout), maps results to the existing card shape, and falls
back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos.
Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri
bridge contract are untouched.
## Context the executor must NOT rediscover
A trimmed real response is committed at `server/fixtures/innertube-search.json`
(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`,
and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29):
```
contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
.itemSectionRenderer.contents[].videoRenderer:
videoId -> id
title.runs[0].text -> title
ownerText.runs[0].text -> channel
ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId -> channelId (UC…)
ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl -> '/@handle' or '/channel/UC…'
lengthText.simpleText "5:43" | "1:59:47" | absent (live) -> duration seconds (0 if absent)
```
Request that works (no key needed):
```
POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
Content-Type: application/json
{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}
```
The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable.
Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first
cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the
fallback is for. Do not "fix" the 403 by adding cookies/keys.
Existing card shape — `server/server.js:293-305` `slimEntry`:
```js
return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };
```
`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`.
Current route `server/server.js:365-395` (inside it):
```js
let mine = [];
try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
try {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
const results = [...mine, ...parseCards(out)];
```
Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script
runs each file separately joined by `&&`.
## Steps
1. Create `server/innertube.js`:
```js
/* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
* parseSearch() is pure (tested against fixtures/innertube-search.json);
* search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
export function lengthToSeconds(s) {
if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
}
export function parseSearch(json) {
const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
?.sectionListRenderer?.contents;
if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
const out = [];
for (const s of sections) {
for (const it of s?.itemSectionRenderer?.contents || []) {
const v = it && it.videoRenderer;
if (!v || typeof v.videoId !== 'string') continue;
const owner = v.ownerText?.runs?.[0] || {};
const be = owner.navigationEndpoint?.browseEndpoint || {};
const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
out.push({
id: v.videoId,
title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
channel: owner.text || '',
channelId: be.browseId || '',
channelUrl: path ? `https://www.youtube.com${path}` : '',
duration: lengthToSeconds(v.lengthText?.simpleText),
thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
});
}
}
return out;
}
export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ context: { client: CLIENT }, query: q }),
signal: AbortSignal.timeout(timeoutMs),
});
if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
return parseSearch(await res.json());
}
```
2. Create `server/innertube.test.js`:
```js
import { test, expect } from 'bun:test';
import { readFileSync } from 'node:fs';
import { parseSearch, lengthToSeconds, search } from './innertube.js';
const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
test('parses video renderers into slim cards', () => {
const r = parseSearch(fixture);
expect(r.length).toBe(4);
expect(r[0]).toEqual({
id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
});
expect(r[1].duration).toBe(7187);
expect(r[3].duration).toBe(0); // live, no lengthText
});
test('lengthToSeconds', () => {
expect(lengthToSeconds('5:43')).toBe(343);
expect(lengthToSeconds('1:59:47')).toBe(7187);
expect(lengthToSeconds('LIVE')).toBe(0);
expect(lengthToSeconds(undefined)).toBe(0);
});
test('unexpected shape throws (caller falls back to yt-dlp)', () => {
expect(() => parseSearch({})).toThrow();
});
test('search() throws on HTTP error', async () => {
const fetchImpl = async () => new Response('no', { status: 429 });
await expect(search('x', { fetchImpl })).rejects.toThrow('429');
});
```
If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's
`lengthText` and use that value (the fixture is the source of truth).
3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`.
4. `server/server.js` — add import next to the other local imports:
`import * as innertube from './innertube.js';`
5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];`
head with:
```js
try {
let yt = [];
try {
yt = await innertube.search(q);
} catch (e) {
console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
}
if (!yt.length) {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
yt = parseCards(out);
}
const results = [...mine, ...yt];
```
Everything after (`searchCache.set`, return, catch) stays as is.
6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')`
skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented
line `# SEARCH_INNERTUBE: "0" # force yt-dlp search` next to the other commented env vars.
## Out of scope / do NOT touch
- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging.
- Card shape, `searchCache`, `app.js`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
kill $SRV; true
grep -c "innertube failed" /tmp/ytp006.log
```
Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns
`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container
has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

338
plans/queue/007-ytdlp-worker-045800.md Normal file
View File

@@ -0,0 +1,338 @@
---
id: 007-ytdlp-worker-045800
title: Keep one long-lived yt-dlp worker process instead of spawning per call
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b]
est_files: 5
---
# 007 — Long-lived yt-dlp worker pool
## Objective
Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with
yt-dlp 2026.08.19 (same host, same network):
| call | per-call spawn | warm worker |
|------|----------------|-------------|
| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s |
| `-J <video>` | 3.32 s | 2.29 s |
After this plan, the "read-only" yt-dlp calls (search fallback, channel, playlist
expand, `-J` stream resolve) go to a pool of 2 long-lived Python workers that import
`yt_dlp` once. Downloads (which write files and use an AbortSignal) keep spawning.
Any pool infrastructure problem falls back to spawning transparently. A bot-check answer
recycles the workers and retries that call with a fresh spawn; each worker is replaced after
100 requests. `YTDLP_WORKER=0` disables the pool.
Dry-run notes (2026-09-29): the worker calls yt-dlp's own CLI entry (`yt_dlp._real_main`) — a
first version using `parse_options()` + `YoutubeDL` directly behaved differently from the CLI.
During testing the container's IP got bot-checked by YouTube for plain CLI calls too, so the
pooled `-J` path could only be measured before that (2.29 s vs 3.32 s). Measure on prod with
plan 001's `[ytdlp] pooled …` log lines and keep `YTDLP_WORKER=0` as the escape hatch.
## Context the executor must NOT rediscover
- In Docker, yt-dlp is the release **zipapp** at `/usr/local/bin/yt-dlp`; Python can import
`yt_dlp` from it by putting that path on `sys.path`. `python3` is installed in the image
(Dockerfile apt line). `/etc/yt-dlp.conf` (the `--js-runtimes bun:…` line) is honoured
because `yt_dlp.parse_options()` reads config files like the CLI does.
- The Dockerfile copies `server/` to `/app`, so a file at `server/ytdlp-worker.py` ships
automatically.
- `server/server.js:143` — `function runYtdlp(args, { signal } = {})` spawns yt-dlp
(after plan 001 it also logs `[ytdlp] <kind> <ms>ms ok|fail`).
- `server/server.js:189` — `async function runYtdlpResilient(args, opts = {})` calls
`runYtdlp(withCookies(args), opts)` and, on a bot check, retries with
`runYtdlp(withCookies(['--extractor-args', …, ...args]), opts)`. `opts` is passed through.
- Read-only call sites to mark `{ pooled: true }` (line numbers from before plans 004/006;
search by the text):
1. `/api/search` fallback: `runYtdlpResilient([\`ytsearch${SEARCH_LIMIT}:${q}\`, …])`
2. `/api/channel`: `runYtdlpResilient([` … `'--playlist-end', String(CHANNEL_LIMIT),`
3. `resolveStreamsUncached`: `runYtdlpResilient(['-J', '--no-warnings', \`https://www.youtube.com/watch?v=${videoId}\`])`
4. `/api/playlist/expand` (~line 1712): `const out = await runYtdlpResilient([`
Do NOT mark the calls at ~861, ~932, ~988 (downloads) or `server/notes.js:362`.
- `const YTDLP = process.env.YTDLP_PATH || 'yt-dlp';` near the top of server.js.
- Expected noise on a machine WITHOUT yt-dlp (local dev before `npm run setup`): three
`ModuleNotFoundError: No module named 'yt_dlp'` lines, then
`[ytdlp-pool] disabled after 3 failed starts …` — the server keeps working via spawn. Not a bug.
## Steps
1. Create `server/ytdlp-worker.py` with exactly:
```python
#!/usr/bin/env python3
"""ytdlp-worker.py - one long-lived yt-dlp process answering many requests.
Spawning yt-dlp per call pays Python start-up + extractor import every time.
This worker imports yt_dlp ONCE and runs each request's argv in-process
through the CLI's own entry point (yt_dlp._real_main), so config files such
as /etc/yt-dlp.conf and everything else the command line sets up apply.
Protocol (newline-delimited JSON):
stdin : {"id": <int>, "args": [<yt-dlp argv>...]}
stdout: {"id": <int>, "code": <int>, "out": "<captured stdout>", "err": "<stderr tail>"}
Only for calls whose result is printed to stdout (-J, --dump-json, --version).
Usage: python3 ytdlp-worker.py <path-to-yt-dlp-zipapp-or-empty>
"""
import io, json, sys, contextlib
if len(sys.argv) > 1 and sys.argv[1]:
sys.path.insert(0, sys.argv[1]) # the yt-dlp release binary is a zipapp
import yt_dlp # noqa: E402
proto_out = sys.stdout
sys.stdout = io.StringIO() # nothing may leak onto the protocol pipe
def run(args):
out, err = io.StringIO(), io.StringIO()
code = 0
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
try:
# The CLI's own entry point (not parse_options + YoutubeDL): it also
# sets up what the command line does (JS challenge solving, plugins,
# post-processing defaults). A bare YoutubeDL got "Sign in to
# confirm you're not a bot" where the CLI succeeded.
ret = yt_dlp._real_main(args)
code = ret[0] if isinstance(ret, tuple) else (ret or 0)
except SystemExit as e:
code = e.code if isinstance(e.code, int) else 1
except Exception as e: # report, never die
err.write(f"ERROR: {e}\n")
code = 1
return code, out.getvalue(), err.getvalue()[-8000:]
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
code, out, err = run([str(a) for a in req.get("args", [])])
resp = {"id": req.get("id"), "code": code, "out": out, "err": err}
except Exception as e:
resp = {"id": None, "code": 1, "out": "", "err": f"worker: {e}"}
proto_out.write(json.dumps(resp) + "\n")
proto_out.flush()
```
2. Create `server/ytdlp-pool.js` with exactly:
```js
/* ytdlp-pool.js — a small pool of long-lived yt-dlp workers (ytdlp-worker.py).
*
* run(args) resolves stdout exactly like a spawned `yt-dlp <args>` would, or
* rejects with the captured stderr. Each worker handles one request at a time;
* extra requests queue. A worker that exits or exceeds the timeout is killed
* and replaced. Errors carrying `poolInfra: true` mean "the pool could not run
* this" — the caller falls back to a normal per-call spawn. Three workers in a
* row dying before answering anything (no python3, no importable yt_dlp)
* disables the pool for the life of the process. Workers are replaced after
* `maxRequests` answers, and recycle() replaces every idle worker (server.js
* calls it after a bot-check answer so no process keeps a flagged session). */
import { spawn } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const WORKER = join(dirname(fileURLToPath(import.meta.url)), 'ytdlp-worker.py');
const infra = (msg) => Object.assign(new Error(msg), { poolInfra: true });
export function createYtdlpPool({ ytdlpPath = '', size = 2, timeoutMs = 60_000, maxRequests = 100, python = 'python3', log = console } = {}) {
const workers = [];
const queue = [];
let nextId = 1;
let startFailures = 0;
let disabled = false;
let closed = false;
function startWorker() {
const child = spawn(python, [WORKER, ytdlpPath], { stdio: ['pipe', 'pipe', 'inherit'] });
const w = { child, busy: null, buf: '', dead: false, retiring: false, served: 0 };
child.stdout.setEncoding('utf8');
child.stdout.on('data', (d) => {
w.buf += d;
let nl;
while ((nl = w.buf.indexOf('\n')) >= 0) {
const line = w.buf.slice(0, nl);
w.buf = w.buf.slice(nl + 1);
let msg;
try { msg = JSON.parse(line); } catch { continue; }
const job = w.busy;
if (!job || msg.id !== job.id) continue;
clearTimeout(job.timer);
w.busy = null;
w.served++;
startFailures = 0;
if (msg.code === 0) job.resolve(msg.out);
else job.reject(new Error((msg.err || '').trim() || 'yt-dlp exited with code ' + msg.code));
if (w.served >= maxRequests) retire(w);
pump();
}
});
const onDead = (why) => {
if (w.dead) return;
w.dead = true;
const i = workers.indexOf(w);
if (i >= 0) workers.splice(i, 1);
if (w.busy) { clearTimeout(w.busy.timer); w.busy.reject(infra('yt-dlp worker ' + why)); w.busy = null; }
if (closed) return;
if (!w.served && ++startFailures >= 3) {
disabled = true;
log.warn?.(`[ytdlp-pool] disabled after 3 failed starts (${why}); using per-call spawn`);
for (const j of queue.splice(0)) j.reject(infra('pool disabled'));
return;
}
if (!w.retiring) log.warn?.(`[ytdlp-pool] worker ${why}; restarting`);
const t = setTimeout(() => { if (!closed && !disabled) { workers.push(startWorker()); pump(); } }, 1000);
t.unref?.();
};
child.on('exit', (code) => onDead('exited (' + code + ')'));
child.on('error', (e) => onDead('failed to start: ' + e.message));
return w;
}
// Replace a worker once it is idle (its exit handler starts a fresh one).
function retire(w) {
if (w.retiring || w.dead) return;
w.retiring = true;
try { w.child.kill(); } catch { /* gone */ }
}
for (let i = 0; i < size; i++) workers.push(startWorker());
function pump() {
for (const w of workers) {
if (w.dead || w.retiring || w.busy || !queue.length) continue;
const job = queue.shift();
w.busy = job;
job.timer = setTimeout(() => { try { w.child.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
try { w.child.stdin.write(JSON.stringify({ id: job.id, args: job.args }) + '\n'); }
catch (e) { clearTimeout(job.timer); w.busy = null; job.reject(infra(e.message)); }
}
}
function run(args) {
if (closed || disabled) return Promise.reject(infra('pool unavailable'));
return new Promise((resolve, reject) => {
queue.push({ id: nextId++, args: args.map(String), resolve, reject });
pump();
});
}
function close() {
closed = true;
for (const w of workers) { try { w.child.kill(); } catch { /* gone */ } }
for (const j of queue.splice(0)) j.reject(infra('pool closed'));
}
function recycle() {
for (const w of workers) if (!w.busy) retire(w);
}
return { run, close, recycle, get disabled() { return disabled; } };
}
```
3. Create `server/ytdlp-pool.test.js` with exactly:
```js
import { test, expect } from 'bun:test';
import { existsSync } from 'node:fs';
import { createYtdlpPool } from './ytdlp-pool.js';
// The worker imports yt_dlp from the release zipapp. Skip when none is around.
const YTDLP = [process.env.YTDLP_PATH, '../bin/yt-dlp', Bun.which('yt-dlp')].find((p) => p && existsSync(p)) || '';
const quiet = { warn() {}, info() {} };
test.skipIf(!YTDLP)('runs requests and reports yt-dlp errors without dying', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, log: quiet });
try {
const v = await pool.run(['--version']);
expect(v.trim()).toMatch(/^\d{4}\.\d{2}\.\d{2}/);
const err = await pool.run(['--definitely-not-a-flag']).catch((e) => e);
expect(err).toBeInstanceOf(Error);
expect(err.poolInfra).toBeFalsy();
expect((await pool.run(['--version'])).trim()).toBe(v.trim());
} finally { pool.close(); }
}, 30000);
test('a broken python disables the pool and rejects as infra', async () => {
const pool = createYtdlpPool({ python: '/nonexistent/python3', size: 1, log: quiet });
const err = await pool.run(['--version']).catch((e) => e);
expect(err.poolInfra).toBe(true);
await new Promise((r) => setTimeout(r, 3500));
expect(pool.disabled).toBe(true);
expect((await pool.run(['--version']).catch((e) => e)).poolInfra).toBe(true);
pool.close();
}, 15000);
test.skipIf(!YTDLP)('workers are replaced after maxRequests and by recycle(), without failing requests', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, maxRequests: 1, log: quiet });
try {
const a = await pool.run(['--version']);
const b = await pool.run(['--version']); // served by a fresh worker
expect(b).toBe(a);
pool.recycle();
expect(await pool.run(['--version'])).toBe(a);
expect(pool.disabled).toBe(false);
} finally { pool.close(); }
}, 30000);
```
4. `server/package.json` "test" script — append ` && bun test ./ytdlp-pool.test.js`.
5. `server/server.js`:
a. Add import: `import { createYtdlpPool } from './ytdlp-pool.js';`
b. Rename the existing `function runYtdlp(args, { signal } = {})` to
`function runYtdlpSpawn(args, { signal } = {})` (body unchanged).
c. Directly after it add:
```js
// Read-only calls (-J, --dump-json) go to long-lived workers that import
// yt_dlp once (~1 s saved per call). Downloads keep spawning. Pool trouble
// (not a yt-dlp error) falls back to a spawn. YTDLP_WORKER=0 disables it.
const ytdlpPool = process.env.YTDLP_WORKER === '0' ? null : createYtdlpPool({
ytdlpPath: Bun.which(YTDLP) || YTDLP,
size: Math.max(1, Number(process.env.YTDLP_WORKERS) || 2),
});
function runYtdlp(args, opts = {}) {
if (!opts.pooled || !ytdlpPool || opts.signal) return runYtdlpSpawn(args, opts);
const t0 = Date.now();
return ytdlpPool.run(args).then(
(out) => { console.log(`[ytdlp] pooled ${Date.now() - t0}ms ok`); return out; },
(err) => {
if (err.poolInfra) return runYtdlpSpawn(args, opts);
// A bot check can stick to a long-lived process: replace the workers and
// answer this call the old way, from a fresh process.
if (BOT_CHECK_RE.test(err.message)) { ytdlpPool.recycle(); return runYtdlpSpawn(args, opts); }
console.log(`[ytdlp] pooled ${Date.now() - t0}ms fail`);
throw err;
},
);
}
```
d. At the 4 read-only call sites listed in Context, add a second argument `{ pooled: true }`
to `runYtdlpResilient(...)`. Example: `runYtdlpResilient(['-J', '--no-warnings', url], { pooled: true })`.
6. `docker-compose.yml` — next to the commented yt-dlp env lines add:
`# YTDLP_WORKER: "0" # disable the long-lived yt-dlp worker pool`
`# YTDLP_WORKERS: "2" # pool size`
## Out of scope / do NOT touch
- Download paths (~861, ~932, ~988), `server/notes.js`, `withSaveSlot`, fallback-client logic.
- Dockerfile (python3 is already installed; the worker file ships with `server/`).
## Verification
```bash
cd /home/user/ytplayer && npm run setup >/dev/null 2>&1; ls -la bin/yt-dlp
cd server && bun test ./ytdlp-pool.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail|skip)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "pooled: true" server.js
```
Expected: pool tests `3 pass` (or `1 pass 2 skip` if `bin/yt-dlp` could not be downloaded —
say so in Findings); every file `0 fail`; `SERVER_OK`; grep prints `4`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

427
plans/queue/008-p2p-schema-and-config-127966.md Normal file
View File

@@ -0,0 +1,427 @@
---
id: 008-p2p-schema-and-config-127966
title: Add P2P tables, config flags and db helpers
created: 2026-09-29
depends_on: []
est_files: 5
---
# 008 — P2P tables, config flags and db helpers
## Objective
Lay the storage foundation for peer-to-peer sharing described in
`docs/p2p-architecture.md` (READ IT FIRST — it is short). After this plan:
- `server/p2p-config.js` exports `P2P` (config) and `loadP2pConfig(env)`.
Defaults: **P2P enabled**, **malware scan disabled**, stale after 7 days.
- `server/p2p-db.js` creates `p2p_content`, `p2p_devices`, `p2p_holders`,
`video_views` and adds `media_cache.sha256`, with query helpers.
- The server calls `initP2pSchema()` at boot. Nothing else changes behaviour yet.
## Context the executor must NOT rediscover
- `server/db.js` exports `db` (libsql client) and `initDb()`. Its `MEDIA_COLS` set
(~line 308) whitelists columns `upsertMedia()` may write:
```js
const MEDIA_COLS = new Set([
'status', 'gen', 'size', 'height', 'vcodec', 'acodec', 'duration', 'optimized',
'meta', 'priority', 'auto', 'attempts', 'error', 'retry_at', 'created_at',
'updated_at', 'last_access', 'hits',
]);
```
- `server/server.js` `main()` (~line 1916): `await initDb();` then `await media.init();`.
- Server test conventions: `bun:test`, a temp `DB_PATH` set BEFORE importing `./db.js`
(see `server/notes.test.js:1-12`), and each file runs in its own `bun test` process via the
`server/package.json` "test" script (db.js is a singleton).
## Steps
1. Create `server/p2p-config.js` with exactly:
```js
/* p2p-config.js — peer-to-peer settings (see docs/p2p-architecture.md).
* P2P is ON unless P2P_ENABLED=0. The malware scan is OFF unless
* P2P_MALWARE_SCAN=1. Hashing + validateMedia are never optional. */
import { dirname, join } from 'node:path';
const num = (v, d) => (Number.isFinite(Number(v)) && String(v).trim() !== '' ? Number(v) : d);
export function loadP2pConfig(env = process.env) {
const dbDir = dirname(env.DB_PATH || './data/ytplayer.db');
return {
enabled: env.P2P_ENABLED !== '0',
malwareScan: env.P2P_MALWARE_SCAN === '1',
scanCmd: (env.P2P_SCAN_CMD || 'clamscan --no-summary --infected').trim(),
staleDays: num(env.P2P_STALE_DAYS, 7),
keepMinViews: num(env.P2P_KEEP_MIN_VIEWS, 3),
keepDays: num(env.P2P_KEEP_DAYS, 30),
keepRecentDays: num(env.P2P_KEEP_RECENT_DAYS, 14),
intakeDir: env.P2P_INTAKE_DIR || join(dbDir, 'p2p-intake'),
intakeMaxBytes: num(env.P2P_INTAKE_MAX_BYTES, 3 * 1024 ** 3),
};
}
export const P2P = loadP2pConfig();
```
2. Create `server/p2p-db.js` — copy VERBATIM from the "p2p-db.js" appendix at the end of this plan.
3. Create `server/p2p-db.test.js` — copy VERBATIM from the "p2p-db.test.js" appendix.
4. `server/db.js` — add `'sha256'` to the end of `MEDIA_COLS` (after `'hits'`).
5. `server/package.json` "test" script — append ` && bun test ./p2p-db.test.js`.
6. `server/server.js` — add `import { initP2pSchema } from './p2p-db.js';` next to the other
local imports, and in `main()` directly after `await initDb();` add `await initP2pSchema();`.
7. `docker-compose.yml` — in the `ytplayer` service `environment:` block, after the
`LYRICS_WORKER_TOKEN` line, add:
```yaml
# Peer-to-peer sharing (docs/p2p-architecture.md). ON by default.
P2P_ENABLED: "${P2P_ENABLED:-1}"
# Malware scan before a file's hash is admitted. OFF by default; needs an
# image built with INSTALL_CLAMAV=1. Hashing + media validation always run.
P2P_MALWARE_SCAN: "${P2P_MALWARE_SCAN:-0}"
# P2P_STALE_DAYS: "7" # holder shown as stale after this many days unchecked
# P2P_KEEP_MIN_VIEWS: "3" # server keeps copies with ≥ this many views…
# P2P_KEEP_DAYS: "30" # …in this many days
# P2P_KEEP_RECENT_DAYS: "14" # …or played this recently
```
## Out of scope / do NOT touch
- No routes, no media-cache changes, no frontend changes (later plans).
- Do not edit `initDb()`'s SQL; the new column is added by `initP2pSchema()`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./p2p-db.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
```
Expected: `4 pass 0 fail`; every file `0 fail`; `SERVER_OK`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files in full).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — p2p-db.js
```js
/* ============================================================================
* p2p-db.js — tables + queries for peer-to-peer sharing
* (docs/p2p-architecture.md). Shares the libsql client from db.js.
*
* p2p_content one row per verified file (cid = sha256 of the bytes); never
* deleted, only revoked — the catalog grows over time
* p2p_devices registered devices (secret stored as sha256)
* p2p_holders which device holds which cid; PERSISTENT (no TTL) with
* last_verified_at — the UI decides what is "stale"
* video_views per-video per-day view counts (retention criteria)
* All timestamps are ms epochs.
* ========================================================================== */
import { db } from './db.js';
export async function initP2pSchema() {
await db.executeMultiple(`
CREATE TABLE IF NOT EXISTS p2p_content (
cid TEXT PRIMARY KEY,
video_id TEXT NOT NULL,
size INTEGER NOT NULL,
height INTEGER NOT NULL DEFAULT 0,
vcodec TEXT,
acodec TEXT,
duration REAL NOT NULL DEFAULT 0,
meta TEXT NOT NULL DEFAULT '{}',
origin TEXT NOT NULL, -- server | intake
status TEXT NOT NULL DEFAULT 'verified', -- verified | revoked
scan TEXT NOT NULL DEFAULT 'skipped', -- skipped | clean
created_at INTEGER NOT NULL,
verified_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_p2p_content_video ON p2p_content (video_id, created_at DESC);
CREATE TABLE IF NOT EXISTS p2p_devices (
device_id TEXT PRIMARY KEY,
secret_hash TEXT NOT NULL,
fingerprint TEXT,
profile TEXT,
share INTEGER NOT NULL DEFAULT 1,
created_at INTEGER NOT NULL,
last_seen_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS p2p_holders (
cid TEXT NOT NULL,
device_id TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active', -- active | removed
trust TEXT NOT NULL DEFAULT 'reported', -- reported | challenged
first_reported_at INTEGER NOT NULL,
last_verified_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (cid, device_id)
);
CREATE INDEX IF NOT EXISTS idx_p2p_holders_device ON p2p_holders (device_id, status);
CREATE TABLE IF NOT EXISTS video_views (
video_id TEXT NOT NULL,
day TEXT NOT NULL,
n INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (video_id, day)
);
`);
// media_cache.sha256 — the cid of the current <id>.<gen>.mp4 (plan 009).
try { await db.execute('ALTER TABLE media_cache ADD COLUMN sha256 TEXT'); }
catch (e) { if (!/duplicate column/i.test(String(e.message))) throw e; }
}
const rowsOf = (r) => r.rows.map((row) => {
const o = {};
r.columns.forEach((c, i) => { o[c] = row[i]; });
return o;
});
// ---- content ----------------------------------------------------------------
export async function upsertContent(c) {
await db.execute({
sql: `INSERT INTO p2p_content (cid, video_id, size, height, vcodec, acodec, duration, meta, origin, status, scan, created_at, verified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'verified', ?, ?, ?)
ON CONFLICT(cid) DO UPDATE SET verified_at = excluded.verified_at, scan = excluded.scan`,
args: [c.cid, c.videoId, c.size, c.height || 0, c.vcodec || null, c.acodec || null, c.duration || 0,
JSON.stringify(c.meta || {}), c.origin, c.scan || 'skipped', c.now, c.now],
});
}
export async function getContent(cid) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_content WHERE cid = ?', args: [cid] });
return rowsOf(r)[0] || null;
}
export async function listContentForVideo(videoId) {
const r = await db.execute({
sql: "SELECT * FROM p2p_content WHERE video_id = ? AND status = 'verified' ORDER BY created_at DESC LIMIT 10",
args: [videoId],
});
return rowsOf(r);
}
export async function knownCids(cids) {
if (!cids.length) return new Set();
const out = new Set();
for (let i = 0; i < cids.length; i += 200) {
const part = cids.slice(i, i + 200);
const r = await db.execute({
sql: `SELECT cid FROM p2p_content WHERE status = 'verified' AND cid IN (${part.map(() => '?').join(',')})`,
args: part,
});
for (const row of r.rows) out.add(row[0]);
}
return out;
}
export async function revokeContent(cid) {
await db.execute({ sql: "UPDATE p2p_content SET status = 'revoked' WHERE cid = ?", args: [cid] });
}
// ---- devices ------------------------------------------------------------------
export async function createDevice({ deviceId, secretHash, fingerprint, profile, now }) {
await db.execute({
sql: `INSERT INTO p2p_devices (device_id, secret_hash, fingerprint, profile, share, created_at, last_seen_at)
VALUES (?, ?, ?, ?, 1, ?, ?)`,
args: [deviceId, secretHash, fingerprint || null, profile || null, now, now],
});
}
export async function getDevice(deviceId) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_devices WHERE device_id = ?', args: [deviceId] });
return rowsOf(r)[0] || null;
}
export async function touchDevice(deviceId, { now, share, profile } = {}) {
await db.execute({
sql: `UPDATE p2p_devices SET last_seen_at = ?,
share = COALESCE(?, share), profile = COALESCE(?, profile)
WHERE device_id = ?`,
args: [now, share === undefined ? null : (share ? 1 : 0), profile || null, deviceId],
});
}
// ---- holders (persistent; never expired by time) ------------------------------
export async function upsertHolder({ cid, deviceId, trust = 'reported', now }) {
await db.execute({
sql: `INSERT INTO p2p_holders (cid, device_id, status, trust, first_reported_at, last_verified_at)
VALUES (?, ?, 'active', ?, ?, ?)
ON CONFLICT(cid, device_id) DO UPDATE SET
status = 'active', removed_at = NULL, last_verified_at = excluded.last_verified_at,
trust = CASE WHEN p2p_holders.trust = 'challenged' OR excluded.trust = 'challenged'
THEN 'challenged' ELSE 'reported' END`,
args: [cid, deviceId, trust, now, now],
});
}
export async function setHolderTrust({ cid, deviceId, trust, now }) {
await db.execute({
sql: 'UPDATE p2p_holders SET trust = ?, last_verified_at = ? WHERE cid = ? AND device_id = ?',
args: [trust, now, cid, deviceId],
});
}
export async function removeHolder({ cid, deviceId, now }) {
await db.execute({
sql: "UPDATE p2p_holders SET status = 'removed', removed_at = ? WHERE cid = ? AND device_id = ? AND status = 'active'",
args: [now, cid, deviceId],
});
}
// A full report: every active holding of this device NOT in `keep` is removed.
export async function removeHoldersExcept({ deviceId, keep, now }) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
const keepSet = new Set(keep);
let removed = 0;
for (const row of r.rows) {
if (keepSet.has(row[0])) continue;
await removeHolder({ cid: row[0], deviceId, now });
removed++;
}
return removed;
}
export async function activeHoldingsOf(deviceId) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
return r.rows.map((row) => row[0]);
}
// Holders of one cid, joined with the device's share flag. Newest check first.
export async function listHolders(cid, limit = 50) {
const r = await db.execute({
sql: `SELECT h.device_id, h.trust, h.first_reported_at, h.last_verified_at, d.share
FROM p2p_holders h JOIN p2p_devices d ON d.device_id = h.device_id
WHERE h.cid = ? AND h.status = 'active'
ORDER BY h.last_verified_at DESC LIMIT ?`,
args: [cid, limit],
});
return rowsOf(r);
}
// ---- views + stats --------------------------------------------------------------
export function dayKey(ms) {
return new Date(ms).toISOString().slice(0, 10);
}
export async function addView(videoId, now) {
await db.execute({
sql: `INSERT INTO video_views (video_id, day, n) VALUES (?, ?, 1)
ON CONFLICT(video_id, day) DO UPDATE SET n = n + 1`,
args: [videoId, dayKey(now)],
});
}
export async function viewsSince(videoId, sinceMs) {
const r = await db.execute({
sql: 'SELECT COALESCE(SUM(n), 0) FROM video_views WHERE video_id = ? AND day >= ?',
args: [videoId, dayKey(sinceMs)],
});
return Number(r.rows[0][0]) || 0;
}
export async function p2pStats() {
const one = async (sql) => Number((await db.execute(sql)).rows[0][0]) || 0;
return {
content: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'verified'"),
revoked: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'revoked'"),
devices: await one('SELECT COUNT(*) FROM p2p_devices'),
holders: await one("SELECT COUNT(*) FROM p2p_holders WHERE status = 'active'"),
heldCids: await one("SELECT COUNT(DISTINCT cid) FROM p2p_holders WHERE status = 'active'"),
};
}
```
## Appendix — p2p-db.test.js
```js
// P2P tables against a real temp libsql DB (docs/p2p-architecture.md).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const { loadP2pConfig } = await import('./p2p-config.js');
const CID = 'a'.repeat(64);
const CID2 = 'b'.repeat(64);
const T0 = Date.UTC(2026, 8, 29, 12);
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
await P.initP2pSchema(); // idempotent (ALTER TABLE duplicate column is ignored)
});
test('config defaults: P2P on, malware scan off', () => {
const c = loadP2pConfig({});
expect(c.enabled).toBe(true);
expect(c.malwareScan).toBe(false);
expect(c.staleDays).toBe(7);
expect(loadP2pConfig({ P2P_ENABLED: '0', P2P_MALWARE_SCAN: '1' })).toMatchObject({ enabled: false, malwareScan: true });
});
test('content upsert/get/known/revoke', async () => {
await P.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1000, height: 720, vcodec: 'h264', acodec: 'aac', duration: 212, meta: { title: 'x' }, origin: 'server', now: T0 });
const c = await P.getContent(CID);
expect(c.video_id).toBe('dQw4w9WgXcQ');
expect(c.status).toBe('verified');
expect([...(await P.knownCids([CID, CID2]))]).toEqual([CID]);
expect((await P.listContentForVideo('dQw4w9WgXcQ')).length).toBe(1);
await P.upsertContent({ cid: CID2, videoId: 'dQw4w9WgXcQ', size: 5, origin: 'intake', now: T0 });
await P.revokeContent(CID2);
expect([...(await P.knownCids([CID2]))]).toEqual([]);
});
test('holders persist, never expire, and a full report removes missing ones', async () => {
await P.createDevice({ deviceId: 'dev_1', secretHash: 'h', fingerprint: 'fp', now: T0 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 });
// 90 days later with no new report: still listed (UI marks it stale).
let hs = await P.listHolders(CID);
expect(hs.length).toBe(1);
expect(hs[0].last_verified_at).toBe(T0);
await P.setHolderTrust({ cid: CID, deviceId: 'dev_1', trust: 'challenged', now: T0 + 1000 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', trust: 'reported', now: T0 + 2000 });
hs = await P.listHolders(CID);
expect(hs[0].trust).toBe('challenged'); // a later plain report never downgrades trust
expect(hs[0].last_verified_at).toBe(T0 + 2000);
expect(await P.removeHoldersExcept({ deviceId: 'dev_1', keep: [], now: T0 + 3000 })).toBe(1);
expect((await P.listHolders(CID)).length).toBe(0);
expect(await P.activeHoldingsOf('dev_1')).toEqual([]);
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 + 4000 }); // re-added
expect(await P.activeHoldingsOf('dev_1')).toEqual([CID]);
});
test('views per day and window sums', async () => {
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0 - 40 * 86400_000);
expect(await P.viewsSince('vid00000001', T0 - 30 * 86400_000)).toBe(2);
expect(await P.viewsSince('vid00000001', T0 - 50 * 86400_000)).toBe(3);
const s = await P.p2pStats();
expect(s.content).toBe(1);
expect(s.devices).toBe(1);
});
```

273
plans/queue/009-server-content-hash-186e7f.md Normal file
View File

@@ -0,0 +1,273 @@
---
id: 009-server-content-hash-186e7f
title: Hash every validated server copy and register it as verified content
created: 2026-09-29
depends_on: [008-p2p-schema-and-config-127966]
est_files: 7
---
# 009 — Server content hashes → verified P2P content
## Objective
Implements flow 1 of `docs/p2p-architecture.md`. After this plan:
- Every copy the media cache promotes (fetch lane AND compression lane) gets a
server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are
hashed by a background backfill 30 s after boot.
- Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only
when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted.
- `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: <cid>`.
- `/api/streams`' cached payload gets an additive `data.cid` (web-only, like
`data.serverCached`).
The media-cache edits were written and tested ahead of time (25/25 media-cache
tests pass, incl. 2 new ones); they ship as patch files.
## Context the executor must NOT rediscover
- Patches (made against the current tree; `media-cache.js` and its test are untouched by
plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`.
They add to `createMediaCache()` the options `hashFile` (default `sha256File` from
`./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the
file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call
`onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export
`backfillHashes()`.
- `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last
property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`.
- `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers:
```js
headers: {
'Content-Type': 'video/mp4',
'Content-Length': String(file.size),
'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
'Cache-Control': 'no-store',
'Access-Control-Allow-Origin': '*',
},
```
- `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns
`{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`.
- Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`).
## Steps
1. Create `server/hash.js` with exactly:
```js
/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
* The hex digest of a validated file is its P2P content id (cid). */
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
export function sha256File(path) {
return new Promise((resolve, reject) => {
const h = createHash('sha256');
createReadStream(path, { highWaterMark: 1024 * 1024 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
// Hash of bytes [offset, offset+length) — used for holder range challenges.
export function sha256Range(path, offset, length) {
return new Promise((resolve, reject) => {
if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
const h = createHash('sha256');
createReadStream(path, { start: offset, end: offset + length - 1 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
```
2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B.
4. Apply the patches from the repo root:
```bash
git apply plans/patches/009-media-cache.diff
git apply plans/patches/009-media-cache-test.diff
```
If either fails, STOP and report the error (do not hand-edit).
5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`.
6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from
plan 008 may already exist; merge into it):
```js
import { admitFile } from './p2p-admit.js';
import { P2P } from './p2p-config.js';
import * as p2pDb from './p2p-db.js';
```
If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in
`main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import.
7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add:
```js
// P2P (docs/p2p-architecture.md): every validated copy's server-computed
// hash becomes verified content, after the optional malware scan.
onReady: (info) => admitFile(
{ ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
{ cfg: P2P, upsertContent: p2pDb.upsertContent },
),
```
8. `server/server.js` `cachedDownloadResponse` — add to the headers object:
```js
...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),
```
9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add
`cid: row.sha256 || null,` and add a comment above the return:
`// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.`
10. `Dockerfile` — optional ClamAV, off by default. After the existing
`RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add:
```dockerfile
# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
ARG INSTALL_CLAMAV=0
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
fi
```
## Out of scope / do NOT touch
- `validateMedia()` itself, the eviction logic, any frontend file.
- Never serve anything from the intake dir; no new routes in this plan.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
bun test ./p2p-admit.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js
```
Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail`
(media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-admit.js
```js
/* ============================================================================
* p2p-admit.js — the ONLY way a content id enters p2p_content
* (docs/p2p-architecture.md, "Security rules").
*
* Callers must already have: the complete file on the server's own disk, its
* SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
* optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
* row. The scan command gets the path as its last argument: exit 0 = clean,
* 1 = infected (rejected), anything else = scanner error (not admitted now).
* ========================================================================== */
import { spawn } from 'node:child_process';
export function scanFile(path, cmd) {
const parts = String(cmd).split(/\s+/).filter(Boolean);
return new Promise((resolve) => {
let child;
try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
catch (e) { resolve({ result: 'error', detail: e.message }); return; }
let out = '';
child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
child.on('close', (code) => resolve(
code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
));
});
}
const CID_RE = /^[0-9a-f]{64}$/;
// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
// → { ok: true, scan } | { ok: false, reason }
export async function admitFile(info, deps) {
const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
let scanResult = 'skipped';
if (cfg.malwareScan) {
const r = await scan(info.path, cfg.scanCmd);
if (r.result !== 'clean') {
log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
return { ok: false, reason: 'scan ' + r.result };
}
scanResult = 'clean';
}
await upsertContent({
cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
scan: scanResult, now: now(),
});
return { ok: true, scan: scanResult };
}
```
## Appendix B — server/p2p-admit.test.js
```js
import { test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { admitFile, scanFile } from './p2p-admit.js';
import { sha256File, sha256Range } from './hash.js';
const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
const file = join(dir, 'f.bin');
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const quiet = { warn() {}, info() {} };
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };
test('sha256File / sha256Range match node:crypto', async () => {
expect(await sha256File(file)).toBe(CID);
const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
expect(await sha256Range(file, 1000, 65536)).toBe(want);
});
test('scan off by default: admitted with scan=skipped', async () => {
const rows = [];
const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
expect(r).toEqual({ ok: true, scan: 'skipped' });
expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
});
test('scan on: clean admits, infected and scanner errors do not', async () => {
for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
const rows = [];
const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
expect(r.ok).toBe(ok);
expect(rows.length).toBe(ok ? 1 : 0);
}
});
test('disabled P2P or a malformed cid never admits', async () => {
const rows = [];
expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect(rows.length).toBe(0);
});
test('scanFile maps exit codes', async () => {
expect((await scanFile(file, 'true')).result).toBe('clean');
expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
});
```

168
plans/queue/010-views-and-retention-d0c6ca.md Normal file
View File

@@ -0,0 +1,168 @@
---
id: 010-views-and-retention-d0c6ca
title: Count views and evict server copies by retention criteria before LRU
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f]
est_files: 5
---
# 010 — Views + retention-ordered eviction
## Objective
Implements flow 9 of `docs/p2p-architecture.md`. After this plan:
- Every play (`GET /api/streams`) and save (`GET /api/download/:id`) counts one view in
`video_views` (deduped per client + video for 30 min; warm-ups don't count).
- When the media cache needs room it evicts copies that are neither **top**
(≥ `P2P_KEEP_MIN_VIEWS` views in `P2P_KEEP_DAYS`) nor **recent** (played within
`P2P_KEEP_RECENT_DAYS`) first — fewest views, then oldest — and only then qualifying
ones by LRU. Budget, disk guard and 10-min play protection are unchanged.
- Eviction never touches `p2p_content` / holders (the catalog keeps growing).
The query + media-cache change were tested ahead of time (retention test 1/1,
media-cache 26/26).
## Context the executor must NOT rediscover
- `plans/patches/010-retention-eviction.diff` patches `server/media-cache.js` (`makeRoom` uses
`db.listMediaEvictionOrder()` when present, else `db.listMediaLru()`) and adds one test to
`server/media-cache.test.js`. It applies on top of plan 009's patches.
- `server/p2p-db.js` (plan 008) has `dayKey`, `rowsOf`, `addView(videoId, now)`.
- `server/server.js` `createMediaCache({ dir: MEDIA_DIR, db: { getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats }, …` (~line 983).
- `app.get('/api/streams', async (c) => {` (~line 532): first lines validate `videoId` and
return 400 when empty. `app.get('/api/download/:videoId', async (c) => {` (~line 1311) does the same.
- Plan 009 imported `P2P` and `* as p2pDb` in server.js.
## Steps
1. `server/p2p-db.js` — append at the end of the file:
```js
// ---- retention (plan 010) -------------------------------------------------------
// Server copies in the order they should be evicted when the cache needs room:
// first the ones that are neither "top" (≥ keepMinViews views in keepDays) nor
// "recent" (played within keepRecentDays) — fewest views, then oldest — and
// only then the qualifying ones, least recently played first.
export async function listMediaEvictionOrder({ now, keepMinViews, keepDays, keepRecentDays }) {
const r = await db.execute({
sql: `SELECT m.video_id, m.size, m.last_access,
COALESCE((SELECT SUM(v.n) FROM video_views v
WHERE v.video_id = m.video_id AND v.day >= ?), 0) AS views
FROM media_cache m WHERE m.status = 'ready'`,
args: [dayKey(now - keepDays * 86400_000)],
});
const recentCut = now - keepRecentDays * 86400_000;
const rows = rowsOf(r).map((x) => ({
...x, views: Number(x.views) || 0,
qualifies: (Number(x.views) || 0) >= keepMinViews || Number(x.last_access) >= recentCut,
}));
rows.sort((a, b) => (a.qualifies - b.qualifies)
|| (a.qualifies ? 0 : a.views - b.views)
|| (a.last_access - b.last_access));
return rows;
}
```
2. Create `server/p2p-retention.test.js` — copy VERBATIM from the Appendix.
3. `server/package.json` "test" script — append ` && bun test ./p2p-retention.test.js`.
4. From the repo root: `git apply plans/patches/010-retention-eviction.diff` (STOP and report on failure).
5. `server/server.js` `createMediaCache({ … db: { … } …` — replace the `db:` line with:
```js
db: {
getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats,
// Retention (docs/p2p-architecture.md flow 9): cold copies go before popular ones.
listMediaEvictionOrder: () => p2pDb.listMediaEvictionOrder({
now: Date.now(), keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
}),
},
```
6. `server/server.js` — directly above `app.get('/api/streams', …)` (and above the plan-005
warm route if present) add:
```js
// One view per client per video per 30 min (a play and its save count once).
const viewSeen = new Map(); // `${who}|${id}` -> ms
function countView(c, videoId) {
const who = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || c.req.query('fp') || 'local';
const key = who + '|' + videoId;
const now = Date.now();
if (now - (viewSeen.get(key) || 0) < 30 * 60_000) return;
viewSeen.set(key, now);
if (viewSeen.size > 20000) viewSeen.clear();
p2pDb.addView(videoId, now).catch(() => {});
}
```
7. In `/api/streams`, directly after `if (!videoId) return c.json({ ok: false, error: 'missing videoId' }, 400);`
add `countView(c, videoId);`. Do the same in `/api/download/:videoId` after its
`missing videoId` guard.
## Out of scope / do NOT touch
- `MEDIA_CACHE_MAX_BYTES`, `EVICT_PROTECT_MS`, the disk guard, `listMediaLru` itself.
- Do not delete server files on a timer — eviction stays budget-driven.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-retention.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "countView(c, videoId)" server.js
```
Expected: `1 pass`; media-cache `26 pass 0 fail`; every file `0 fail`; `SERVER_OK`; grep `2`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — server/p2p-retention.test.js
```js
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-retention-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const DAY = 86400_000;
const NOW = Date.UTC(2026, 8, 29, 12);
const opts = { now: NOW, keepMinViews: 3, keepDays: 30, keepRecentDays: 14 };
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
const add = (id, lastAccessDaysAgo) => dbmod.upsertMedia(id, { status: 'ready', size: 100, last_access: NOW - lastAccessDaysAgo * DAY });
await add('topOld00001', 60); // 5 views in window → top, but old
await add('recent00001', 2); // 0 views, played 2 days ago → recent
await add('cold0000001', 40); // 1 view in window, old → evict first-ish
await add('cold0000002', 20); // 0 views, 20 days → evict first
await add('cold0000003', 50); // 0 views, 50 days → evict first (older than cold2)
for (let i = 0; i < 5; i++) await P.addView('topOld00001', NOW - 5 * DAY);
await P.addView('cold0000001', NOW - 3 * DAY);
await P.addView('cold0000002', NOW - 45 * DAY); // outside the 30-day window
});
test('non-qualifying copies go first (fewest views, then oldest); qualifying ones by LRU', async () => {
const order = (await P.listMediaEvictionOrder(opts)).map((r) => [r.video_id, r.qualifies]);
expect(order).toEqual([
['cold0000003', false],
['cold0000002', false],
['cold0000001', false],
['topOld00001', true],
['recent00001', true],
]);
});
```

242
plans/queue/011-browser-sha256-e1793d.md Normal file
View File

@@ -0,0 +1,242 @@
---
id: 011-browser-sha256-e1793d
title: Add an incremental SHA-256 library for the browser and node tests
created: 2026-09-29
depends_on: []
est_files: 5
---
# 011 — Incremental SHA-256 for the browser
## Objective
Devices must compute a file's content id (SHA-256) while bytes stream past —
during a download, a peer transfer, or a chunked OPFS read — without holding the
file in memory. WebCrypto's `digest()` needs the whole input at once, so add a
small pure-JS incremental hasher, usable from the page, from Web Workers
(`importScripts('/sha256.js')`) and from node tests. Pre-tested: all vectors and
every length 0–200 match `node:crypto`; random chunking of a 3 MiB buffer matches;
~116 MB/s in Node.
## Context the executor must NOT rediscover
- Module style to copy: `frontend/stats-core.js` — an IIFE `(function (root) { … })(typeof globalThis !== 'undefined' ? globalThis : this);`
that sets `module.exports` under node and `root.StatsCore` in the browser. Tests are
CommonJS `node:test` (`frontend/stats-core.test.js`).
- `frontend/index.html:562-569` loads scripts in this order:
```html
<script src="fingerprint.js"></script>
<script src="opfs.js"></script>
<script src="video-edit.js"></script>
<script src="lyrics-core.js"></script>
<script src="stats-core.js"></script>
<script src="async-guard.js"></script>
<script src="sw-update.js"></script>
<script src="app.js"></script>
```
- `frontend/sw.js` `SHELL` array (~line 55) must list every shell file.
## Steps
1. Create `frontend/sha256.js` — copy VERBATIM from Appendix A.
2. Create `frontend/sha256.test.js` — copy VERBATIM from Appendix B.
3. `frontend/index.html` — add ` <script src="sha256.js"></script>` directly after the
`stats-core.js` script line.
4. `frontend/sw.js` `SHELL` — add `'/sha256.js',` directly after `'/stats-core.js',`.
5. `CLAUDE.md` "Testing" section — in the list `(sw, sw-update, async-guard, video-edit, lyrics-core, stats-core)`
add `, sha256`.
## Out of scope / do NOT touch
- No callers yet (plans 012/013/017 use it). Do not touch `app.js`.
## Verification
```bash
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
node -e "const S=require('./frontend/sha256');console.log(S.hex(new TextEncoder().encode('abc')))"
grep -c "sha256.js" frontend/index.html frontend/sw.js
```
Expected: `# fail 0`; `ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad`; each grep `1`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — frontend/sha256.js
```js
/* ============================================================================
* sha256.js — incremental SHA-256 (pure JS; window.Sha256 / worker / node)
*
* WebCrypto's digest() needs the whole input at once, which would pull a
* multi-hundred-MB video into memory. This hasher takes chunks as they stream
* past (downloads, peer transfers, OPFS reads) and keeps ~100 bytes of state.
*
* const h = Sha256.create(); h.update(u8); …; const hex = h.hex();
* Sha256.hex(u8) // one-shot convenience
* ========================================================================== */
(function (root) {
'use strict';
const K = new Uint32Array([
0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2,
]);
function create() {
const H = new Uint32Array([
0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19,
]);
const W = new Uint32Array(64);
const block = new Uint8Array(64);
let blockLen = 0;
let total = 0; // bytes hashed so far (safe to 2^53)
let done = false;
function compress(buf, off) {
for (let i = 0; i < 16; i++) {
const j = off + i * 4;
W[i] = (buf[j] << 24) | (buf[j + 1] << 16) | (buf[j + 2] << 8) | buf[j + 3];
}
for (let i = 16; i < 64; i++) {
const w15 = W[i - 15], w2 = W[i - 2];
const s0 = ((w15 >>> 7) | (w15 << 25)) ^ ((w15 >>> 18) | (w15 << 14)) ^ (w15 >>> 3);
const s1 = ((w2 >>> 17) | (w2 << 15)) ^ ((w2 >>> 19) | (w2 << 13)) ^ (w2 >>> 10);
W[i] = (W[i - 16] + s0 + W[i - 7] + s1) | 0;
}
let a = H[0], b = H[1], c = H[2], d = H[3], e = H[4], f = H[5], g = H[6], h = H[7];
for (let i = 0; i < 64; i++) {
const S1 = ((e >>> 6) | (e << 26)) ^ ((e >>> 11) | (e << 21)) ^ ((e >>> 25) | (e << 7));
const ch = (e & f) ^ (~e & g);
const t1 = (h + S1 + ch + K[i] + W[i]) | 0;
const S0 = ((a >>> 2) | (a << 30)) ^ ((a >>> 13) | (a << 19)) ^ ((a >>> 22) | (a << 10));
const maj = (a & b) ^ (a & c) ^ (b & c);
const t2 = (S0 + maj) | 0;
h = g; g = f; f = e; e = (d + t1) | 0;
d = c; c = b; b = a; a = (t1 + t2) | 0;
}
H[0] = (H[0] + a) | 0; H[1] = (H[1] + b) | 0; H[2] = (H[2] + c) | 0; H[3] = (H[3] + d) | 0;
H[4] = (H[4] + e) | 0; H[5] = (H[5] + f) | 0; H[6] = (H[6] + g) | 0; H[7] = (H[7] + h) | 0;
}
function update(data) {
if (done) throw new Error('sha256: update() after digest');
const u8 = data instanceof Uint8Array ? data : new Uint8Array(data);
let i = 0;
total += u8.length;
if (blockLen) {
const take = Math.min(64 - blockLen, u8.length);
block.set(u8.subarray(0, take), blockLen);
blockLen += take;
i = take;
if (blockLen === 64) { compress(block, 0); blockLen = 0; }
}
for (; i + 64 <= u8.length; i += 64) compress(u8, i);
if (i < u8.length) { block.set(u8.subarray(i), 0); blockLen = u8.length - i; }
return api;
}
function digest() {
if (!done) {
done = true;
const bits = total * 8;
block[blockLen++] = 0x80;
if (blockLen > 56) { block.fill(0, blockLen); compress(block, 0); blockLen = 0; }
block.fill(0, blockLen, 56);
const hi = Math.floor(bits / 0x100000000), lo = bits >>> 0;
block[56] = hi >>> 24; block[57] = hi >>> 16; block[58] = hi >>> 8; block[59] = hi;
block[60] = lo >>> 24; block[61] = lo >>> 16; block[62] = lo >>> 8; block[63] = lo;
compress(block, 0);
}
const out = new Uint8Array(32);
for (let i = 0; i < 8; i++) {
out[i * 4] = H[i] >>> 24; out[i * 4 + 1] = H[i] >>> 16; out[i * 4 + 2] = H[i] >>> 8; out[i * 4 + 3] = H[i];
}
return out;
}
function hex() {
let s = '';
for (const b of digest()) s += (b < 16 ? '0' : '') + b.toString(16);
return s;
}
const api = { update, digest, hex, get bytes() { return total; } };
return api;
}
const Sha256 = {
create,
hex: (data) => create().update(data).hex(),
isHex: (s) => typeof s === 'string' && /^[0-9a-f]{64}$/.test(s),
};
if (typeof module !== 'undefined' && module.exports) module.exports = Sha256;
else root.Sha256 = Sha256;
})(typeof globalThis !== 'undefined' ? globalThis : this);
```
## Appendix B — frontend/sha256.test.js
```js
'use strict';
const { test } = require('node:test');
const assert = require('node:assert');
const crypto = require('node:crypto');
const Sha256 = require('./sha256');
const ref = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
test('known vectors', () => {
assert.strictEqual(Sha256.hex(new Uint8Array(0)), 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855');
assert.strictEqual(Sha256.hex(new TextEncoder().encode('abc')), 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad');
assert.strictEqual(
Sha256.hex(new TextEncoder().encode('abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq')),
'248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1');
});
test('every length around block boundaries matches node:crypto', () => {
for (let n = 0; n <= 200; n++) {
const buf = crypto.randomBytes(n);
assert.strictEqual(Sha256.hex(new Uint8Array(buf)), ref(buf), 'length ' + n);
}
});
test('random chunk boundaries give the same digest as one shot', () => {
const buf = crypto.randomBytes(3 * 1024 * 1024 + 17);
for (let round = 0; round < 5; round++) {
const h = Sha256.create();
let pos = 0;
while (pos < buf.length) {
const n = Math.min(buf.length - pos, 1 + Math.floor(Math.random() * 200000));
h.update(new Uint8Array(buf.buffer, buf.byteOffset + pos, n));
pos += n;
}
assert.strictEqual(h.hex(), ref(buf));
assert.strictEqual(h.bytes, buf.length);
}
});
test('update after digest throws; isHex', () => {
const h = Sha256.create();
h.hex();
assert.throws(() => h.update(new Uint8Array(1)));
assert.ok(Sha256.isHex(ref(Buffer.from('x'))));
assert.ok(!Sha256.isHex('ABC'));
});
```

215
plans/queue/012-device-file-registry-288d55.md Normal file
View File

@@ -0,0 +1,215 @@
---
id: 012-device-file-registry-288d55
title: Add the on-device IndexedDB file registry and hash saves while downloading
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f, 011-browser-sha256-e1793d]
est_files: 6
---
# 012 — On-device file registry + hash while saving
## Objective
Implements flow 2 of `docs/p2p-architecture.md`. After this plan:
- `frontend/device-db.js` (`window.DeviceDB`) keeps one IndexedDB record per saved
video: `{ videoId, cid, size, savedAt, lastCheckedAt, state }`.
- The OPFS download worker hashes bytes as it writes them. If the server sent
`X-Content-SHA256` (plan 009) and the hash differs, the save FAILS and the partial
file is deleted ("integrity check failed").
- `preload()` records the file (`state: 'verified'` when the hashes matched,
`'unverified'` when the server sent none, `'unhashed'` for the main-thread fallback);
deleting / clearing saved videos removes the records.
Pre-tested in Chromium with the harness in `plans/harness/` (good hash → saved and
`verified`; wrong hash → rejected, no file left; no header → saved with a local hash).
## Context the executor must NOT rediscover
- `plans/patches/012-opfs-hash.diff` patches `frontend/opfs-worker.js` (importScripts
`/sha256.js`, hash in the write loop, compare with `x-content-sha256`, post
`{ type:'done', ext, sha256, expectedSha, size }`) and `frontend/opfs.js`
(`downloadVideo` resolves `{ ok:true, sha256, expectedSha, size }`).
- `frontend/app.js:157-198` `async function opfsDownload(videoId, { mux = false } = {})` — worker path:
```js
if (typeof window.OPFS.downloadVideo === 'function' && typeof Worker !== 'undefined') {
const w = await window.OPFS.downloadVideo(videoId, url);
if (w.ok) return { ok: true, cached: true };
workerError = w.error || null;
}
```
- `frontend/app.js:258-268`:
```js
async function opfsDelete(videoId) {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.deleteVideo(videoId); } catch { /* ignore */ }
return { ok: true };
}
async function opfsClear() {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.clearAll(); } catch { /* ignore */ }
return { ok: true };
}
```
- `frontend/app.js` `async function preload(video, …)` (~line 1237), success branch:
```js
const res = await API.cacheDownload(id, { mux });
if (res && res.ok && res.cached) {
cachedIds.add(id);
cacheMutations++;
warmThumb(thumbUrlFor(id, video));
```
- Script order in `frontend/index.html` after plan 011: `… stats-core.js, sha256.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` after plan 011 contains `'/sha256.js',`.
## Steps
1. Create `frontend/device-db.js` — copy VERBATIM from the Appendix.
2. From the repo root: `git apply plans/patches/012-opfs-hash.diff` (STOP and report on failure).
3. `frontend/index.html` — add ` <script src="device-db.js"></script>` directly after the `sha256.js` line.
4. `frontend/sw.js` `SHELL` — add `'/device-db.js',` directly after `'/sha256.js',`.
5. `frontend/app.js` `opfsDownload` — the line `if (w.ok) return { ok: true, cached: true };` appears
TWICE in app.js; change ONLY the first one (inside `async function opfsDownload`, ~line 180),
NOT the one inside `opfsDownloadEdited` (edited cuts are never shared). Change it to:
```js
if (w.ok) return { ok: true, cached: true, sha256: w.sha256 || null, expectedSha: w.expectedSha || null, size: w.size || 0 };
```
(The main-thread fallback below keeps returning `{ ok: true, cached: true }` — no hash.)
6. `frontend/app.js` — directly ABOVE `async function preload(video, …)` add:
```js
// This device's record of what it holds and each file's content id
// (docs/p2p-architecture.md flow 2). The P2P client reports these.
function recordDeviceFile(id, res) {
if (!WEB || !window.DeviceDB) return;
const cid = res && window.Sha256 && window.Sha256.isHex(res.sha256) ? res.sha256 : null;
const state = cid ? (res.expectedSha === cid ? 'verified' : 'unverified') : 'unhashed';
const now = Date.now();
window.DeviceDB.putFile({ videoId: id, cid, size: (res && res.size) || 0, savedAt: now, lastCheckedAt: now, state })
.then(() => { if (window.P2PClient) window.P2PClient.changed(); })
.catch(() => {});
}
```
7. `frontend/app.js` `preload` success branch — after `cacheMutations++;` add
`recordDeviceFile(id, res);`
8. `frontend/app.js` `opfsDelete` — after the `try { await window.OPFS.deleteVideo(videoId); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.deleteFile(videoId); if (window.P2PClient) window.P2PClient.changed(); }
```
`opfsClear` — after its `try { await window.OPFS.clearAll(); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.clear(); if (window.P2PClient) window.P2PClient.changed(); }
```
## Out of scope / do NOT touch
- `opfsDownloadEdited` (edited cuts are device-only, never shared).
- No server calls here (plan 013 reports holdings). Do not hash old files here (plan 013).
- Do not migrate `_ytpdata` / playlists to IndexedDB.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/opfs.js frontend/opfs-worker.js frontend/device-db.js && echo SYNTAX_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "device-db.js" frontend/index.html frontend/sw.js
# Browser check (Chromium). Skip with a note in Findings if no Chromium/playwright is available.
cd plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1)
bun device-db-server.js >/tmp/ytp012.log 2>&1 & SRV=$!; sleep 2; timeout 60 node device-db-check.mjs; kill $SRV; true
```
Expected: `SYNTAX_OK`; `# fail 0`; grep `1` each; the browser check prints JSON containing
`"shaOk":true,"goodExpected":true`, `"bad":{"ok":false,"error":"integrity check failed (content hash mismatch)"}`,
`"list":["good","nohash"]`, `"rec":"verified"`, `"after":0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — frontend/device-db.js
```js
/* ============================================================================
* device-db.js — this device's own file registry (IndexedDB "ytp-device")
*
* One record per saved video, next to the bytes in OPFS:
* { videoId, cid, size, savedAt, lastCheckedAt, state }
* cid SHA-256 of the stored file (P2P content id) or null
* state 'verified' hash computed on save and equal to the server's
* 'unverified' hash computed, but the server sent none to compare
* 'unhashed' saved before hashing existed / main-thread fallback
* The P2P client (p2p-client.js) reads this to report holdings; OPFS stays the
* source of truth for what is playable (a record without a file is ignored).
* Every call resolves (null / [] on failure) — private windows can block IDB.
* See docs/p2p-architecture.md.
* ========================================================================== */
(function () {
'use strict';
const DB_NAME = 'ytp-device';
const VERSION = 1;
let _open = null;
function open() {
if (!_open) {
_open = new Promise((resolve, reject) => {
const r = indexedDB.open(DB_NAME, VERSION);
r.onupgradeneeded = () => {
const d = r.result;
if (!d.objectStoreNames.contains('files')) {
const s = d.createObjectStore('files', { keyPath: 'videoId' });
s.createIndex('cid', 'cid', { unique: false });
}
};
r.onsuccess = () => resolve(r.result);
r.onerror = () => reject(r.error);
r.onblocked = () => reject(new Error('device-db blocked'));
});
_open.catch(() => { _open = null; });
}
return _open;
}
const done = (req) => new Promise((resolve, reject) => {
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
async function store(mode) {
const d = await open();
return d.transaction('files', mode).objectStore('files');
}
async function safe(fn, fallback) {
try { return await fn(); } catch { return fallback; }
}
window.DeviceDB = {
isSupported: () => typeof indexedDB !== 'undefined',
putFile: (rec) => safe(async () => {
if (!rec || !rec.videoId) return null;
await done((await store('readwrite')).put({
videoId: String(rec.videoId),
cid: rec.cid || null,
size: Number(rec.size) || 0,
savedAt: Number(rec.savedAt) || Date.now(),
lastCheckedAt: Number(rec.lastCheckedAt) || Date.now(),
state: rec.state || 'unhashed',
}));
return true;
}, null),
getFile: (videoId) => safe(async () => (await done((await store('readonly')).get(String(videoId)))) || null, null),
getByCid: (cid) => safe(async () => (await done((await store('readonly')).index('cid').get(String(cid)))) || null, null),
listFiles: () => safe(async () => (await done((await store('readonly')).getAll())) || [], []),
deleteFile: (videoId) => safe(async () => { await done((await store('readwrite')).delete(String(videoId))); return true; }, null),
clear: () => safe(async () => { await done((await store('readwrite')).clear()); return true; }, null),
};
}());
```

622
plans/queue/013-device-identity-and-holdings-3ba493.md Normal file
View File

@@ -0,0 +1,622 @@
---
id: 013-device-identity-and-holdings-3ba493
title: Register devices and report verified holdings to the server
created: 2026-09-29
depends_on: [010-views-and-retention-d0c6ca, 012-device-file-registry-288d55]
est_files: 11
---
# 013 — Device identity + holdings sync
## Objective
Implements flow 3 of `docs/p2p-architecture.md`. After this plan, with default
settings (P2P ON):
- A device registers once (`POST /api/p2p/device` → `deviceId` + `secret`, kept in
`localStorage.ytpDevice`; only `sha256(secret)` is stored server-side).
- `P2PClient` reconciles `DeviceDB` with the files really in OPFS, hashes old/unhashed
saves in a worker, reports the FULL holdings list, answers the server's range
challenges, and marks accepted files `verified`.
- The server keeps holder rows forever (no TTL): refreshed `last_verified_at` on every
report, `removed` when a full report no longer lists them or a challenge fails.
- Turning sharing off (`data.settings.p2pShare === false`) withdraws every holding.
Pre-tested: server routes 6/6 (`bun:test`), and the whole client flow in Chromium via
`plans/harness/p2p-client-*.js` (ghost record dropped, legacy file hashed, verified
file accepted + challenge passed → `trust: challenged`, share off → 0 holders).
## Context the executor must NOT rediscover
- Plans 008–012 are applied: `server/p2p-db.js`, `p2p-config.js` (`P2P`), `hash.js`
(`sha256Range`), `p2p-admit.js`; `frontend/sha256.js`, `frontend/device-db.js`, and
`opfs.js`/`opfs-worker.js` hash on save. `server/server.js` imports `P2P` and `* as p2pDb`.
- `plans/patches/013-opfs-readrange.diff` adds `OPFS.readRange(videoId, offset, length)` to
`frontend/opfs.js` (applies on top of plan 012's patch).
- `server/server.js`: `const MEDIA_DIR = process.env.MEDIA_DIR || './data/media';` (~line 980).
Plan 009's `const notes = registerNoteRoutes(app, {…});` (~line 1823) is a good neighbourhood
for another `register…Routes(app, …)` call — add ours right after the uploads block
(`const isUpload = (id) => uploads.isUploadId(id);`).
- `frontend/app.js`:
- `const DEFAULT_SETTINGS = {` (~line 326), first line
`quality: 'auto', volume: 1, audioOnly: false, autoPreload: true,`.
- end of `async function boot()` (~line 9822):
```js
// Learn what's already cached, then top up any playlist videos that aren't.
await refreshCachedIds();
startCacheResyncWatch();
```
- `data.profile` is `{ name, syncedAt }` or null.
- Script order in `frontend/index.html` after plan 012:
`… stats-core.js, sha256.js, device-db.js, async-guard.js, sw-update.js, app.js`.
## Steps
1. `server/p2p-db.js`:
a. In `initP2pSchema()`, directly after the `ALTER TABLE media_cache ADD COLUMN sha256` try/catch add:
` await db.execute('CREATE INDEX IF NOT EXISTS idx_media_sha256 ON media_cache (sha256)');`
b. Append at the end of the file:
```js
// The server's own ready copy whose mp4 has this content id (or null).
export async function findMediaByCid(cid) {
const r = await db.execute({
sql: "SELECT video_id, gen FROM media_cache WHERE sha256 = ? AND status = 'ready' LIMIT 1",
args: [cid],
});
return rowsOf(r)[0] || null;
}
```
2. Create `server/p2p-routes.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-routes.test.js` — copy VERBATIM from Appendix B.
4. `server/package.json` "test" script — append ` && bun test ./p2p-routes.test.js`.
5. `server/server.js`:
a. imports: `import { registerP2pRoutes } from './p2p-routes.js';` and add `sha256Range` to an
import from `./hash.js`: `import { sha256Range } from './hash.js';`
b. after `const isUpload = (id) => uploads.isUploadId(id);` add:
```js
// ============================================================================
// Peer-to-peer sharing — devices + holdings (docs/p2p-architecture.md)
// ============================================================================
async function fileForCid(cid) {
const row = await p2pDb.findMediaByCid(cid);
if (!row) return null;
const path = `${MEDIA_DIR}/${row.video_id}.${row.gen}.mp4`;
const f = Bun.file(path);
return (await f.exists()) ? { path, size: f.size } : null;
}
const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });
```
(`p2p` is used by plan 014.)
6. Create `frontend/hash-worker.js` — copy VERBATIM from Appendix C.
7. Create `frontend/p2p-client.js` — copy VERBATIM from Appendix D.
8. From the repo root: `git apply plans/patches/013-opfs-readrange.diff` (STOP on failure).
9. `frontend/index.html` — add ` <script src="p2p-client.js"></script>` directly after the `device-db.js` line.
10. `frontend/sw.js` `SHELL` — add `'/hash-worker.js',` and `'/p2p-client.js',` directly after `'/device-db.js',`.
11. `frontend/app.js` `DEFAULT_SETTINGS` — add a new line after the first line:
```js
p2pShare: true, // P2P (docs/p2p-architecture.md): share my saved videos with other devices — ON by default
p2pReceive: true, // fetch from other devices when YouTube and the server can't — ON by default
```
12. `frontend/app.js` `boot()` — directly after `startCacheResyncWatch();` add:
```js
// Peer-to-peer: report what this device holds (on by default; settings.p2pShare).
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
## Out of scope / do NOT touch
- Presence/websocket, holders endpoint, UI (plans 014/015). Intake of unknown files (016).
- Do not change how saves download or play.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-routes.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js frontend/hash-worker.js frontend/opfs.js frontend/app.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
# Browser check (Chromium + playwright; see plans/harness/device-db-check.mjs header).
cd server
bun ../plans/harness/p2p-client-server.js >/tmp/ytp013.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node p2p-client-check.mjs
kill $SRV; true
```
Expected: routes `6 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; `# fail 0`; browser JSON
`{"accepted":1,"unknown":1,"challenges":1,"recs":[["goodAAAAAAA","verified",true],["legacyAAAAA","unverified",true]],"trust":["challenged"],"afterShareOff":0,"device":true}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files may be summarised as "verbatim from appendix").
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-routes.js
```js
/* ============================================================================
* p2p-routes.js — device registration + holdings (docs/p2p-architecture.md
* flows 3 and 5). Mounted by server.js; every route answers 404 when
* P2P_ENABLED=0.
*
* GET /api/p2p/config { ok, enabled, staleDays }
* POST /api/p2p/device { fingerprint?, profile? } → { ok, deviceId, secret }
* POST /api/p2p/holdings (device) { items:[{cid,size,videoId}], share } → { ok, accepted, unknown, challenges }
* POST /api/p2p/challenge (device) { answers:[{cid,offset,length,sha256}] } → { ok, passed, failed }
* GET /api/p2p/holders?v=<id>|cid= availability — added by plan 014
*
* Device auth: header `X-Device: <deviceId>.<secret>`; only sha256(secret) is
* stored. A holdings report is always the device's FULL list: anything it held
* before and no longer lists is marked removed. Holder rows never expire by
* time — last_verified_at is refreshed by each report (see the architecture doc).
* ========================================================================== */
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto';
const CID_RE = /^[0-9a-f]{64}$/;
const DEV_RE = /^dev_[0-9a-f]{16}$/;
const MAX_ITEMS = 5000;
const MAX_CHALLENGES = 5;
const CHALLENGE_LEN = 64 * 1024;
const sha = (s) => createHash('sha256').update(String(s)).digest('hex');
const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); };
export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12);
export function registerP2pRoutes(app, deps) {
const { cfg, p2pDb, fileForCid, sha256Range, now = () => Date.now(), log = console } = deps;
const pending = new Map(); // `${deviceId}|${cid}` -> { offset, length, at }
const regLog = new Map(); // ip -> [ms]
const gate = async (c, next) => {
if (!cfg.enabled) return c.json({ ok: false, error: 'p2p disabled' }, 404);
await next();
};
async function deviceOf(c) {
const m = String(c.req.header('x-device') || '').match(/^(dev_[0-9a-f]{16})\.([0-9a-f]{64})$/);
if (!m) return null;
const d = await p2pDb.getDevice(m[1]);
if (!d || !same(d.secret_hash, sha(m[2]))) return null;
return d;
}
const requireDevice = async (c, next) => {
const d = await deviceOf(c);
if (!d) return c.json({ ok: false, error: 'unknown device' }, 401);
c.set('device', d);
await next();
};
app.get('/api/p2p/config', (c) => c.json({ ok: true, enabled: cfg.enabled, staleDays: cfg.staleDays }));
app.post('/api/p2p/device', gate, async (c) => {
const ip = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || 'local';
const t = now();
const recent = (regLog.get(ip) || []).filter((x) => t - x < 3600_000);
if (recent.length >= 20) return c.json({ ok: false, error: 'too many registrations' }, 429);
recent.push(t);
regLog.set(ip, recent);
if (regLog.size > 10000) regLog.clear();
const body = await c.req.json().catch(() => ({}));
const deviceId = 'dev_' + randomBytes(8).toString('hex');
const secret = randomBytes(32).toString('hex');
await p2pDb.createDevice({
deviceId, secretHash: sha(secret),
fingerprint: String(body.fingerprint || '').slice(0, 128) || null,
profile: String(body.profile || '').slice(0, 64) || null,
now: t,
});
return c.json({ ok: true, deviceId, secret });
});
app.post('/api/p2p/holdings', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const share = body.share !== false;
await p2pDb.touchDevice(d.device_id, { now: t, share, profile: String(body.profile || '').slice(0, 64) || undefined });
const raw = Array.isArray(body.items) ? body.items.slice(0, MAX_ITEMS) : [];
const items = share ? raw.filter((x) => x && CID_RE.test(String(x.cid))) : [];
const known = await p2pDb.knownCids(items.map((x) => x.cid));
const accepted = [];
const unknown = [];
for (const it of items) {
if (!known.has(it.cid)) { unknown.push(it.cid); continue; }
await p2pDb.upsertHolder({ cid: it.cid, deviceId: d.device_id, now: t });
accepted.push(it.cid);
}
await p2pDb.removeHoldersExcept({ deviceId: d.device_id, keep: accepted, now: t });
// Spot-check a few holdings against the server's own copy when it has one.
const challenges = [];
for (const cid of accepted) {
if (challenges.length >= MAX_CHALLENGES) break;
const f = await fileForCid(cid);
if (!f || !(f.size > CHALLENGE_LEN)) continue;
const offset = Math.floor(Math.random() * (f.size - CHALLENGE_LEN));
pending.set(d.device_id + '|' + cid, { offset, length: CHALLENGE_LEN, at: t });
challenges.push({ cid, offset, length: CHALLENGE_LEN });
}
if (pending.size > 50000) pending.clear();
return c.json({ ok: true, accepted, unknown, challenges });
});
app.post('/api/p2p/challenge', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const passed = [];
const failed = [];
for (const a of (Array.isArray(body.answers) ? body.answers : []).slice(0, MAX_CHALLENGES)) {
const key = d.device_id + '|' + a.cid;
const p = pending.get(key);
if (!p || p.offset !== a.offset || p.length !== a.length || t - p.at > 10 * 60_000) continue;
pending.delete(key);
const f = await fileForCid(a.cid);
if (!f) continue;
const want = await sha256Range(f.path, p.offset, p.length);
if (String(a.sha256) === want) {
await p2pDb.setHolderTrust({ cid: a.cid, deviceId: d.device_id, trust: 'challenged', now: t });
passed.push(a.cid);
} else {
await p2pDb.removeHolder({ cid: a.cid, deviceId: d.device_id, now: t });
failed.push(a.cid);
log.warn?.(`[p2p] ${d.device_id} failed challenge for ${a.cid.slice(0, 12)}`);
}
}
return c.json({ ok: true, passed, failed });
});
return { deviceOf, peerIdOf, requireDevice, gate };
}
```
## Appendix B — server/p2p-routes.test.js
```js
// Device registration, holdings reports and range challenges (plan 013).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { Hono } from 'hono';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-routes-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const p2pDb = await import('./p2p-db.js');
const { registerP2pRoutes } = await import('./p2p-routes.js');
const { sha256Range } = await import('./hash.js');
const file = join(root, 'copy.mp4');
const bytes = Buffer.from(Array.from({ length: 400000 }, (_, i) => (i * 13) % 256));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const OTHER = 'c'.repeat(64); // verified but the server has no file
const UNKNOWN = 'd'.repeat(64); // not in p2p_content
const quiet = { warn() {}, info() {} };
let app;
let enabled = true;
const req = (path, { method = 'GET', body, dev } = {}) => app.request(path, {
method,
headers: { 'Content-Type': 'application/json', ...(dev ? { 'X-Device': dev.deviceId + '.' + dev.secret } : {}) },
body: body ? JSON.stringify(body) : undefined,
});
beforeAll(async () => {
await dbmod.initDb();
await p2pDb.initP2pSchema();
await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server', now: 1 });
await p2pDb.upsertContent({ cid: OTHER, videoId: 'dQw4w9WgXcQ', size: 10, origin: 'server', now: 1 });
app = new Hono();
registerP2pRoutes(app, {
cfg: { get enabled() { return enabled; }, staleDays: 7 },
p2pDb,
fileForCid: async (cid) => (cid === CID ? { path: file, size: bytes.length } : null),
sha256Range,
log: quiet,
});
});
async function newDevice() {
const r = await req('/api/p2p/device', { method: 'POST', body: { fingerprint: 'fp1' } });
expect(r.status).toBe(200);
const j = await r.json();
expect(j.deviceId).toMatch(/^dev_[0-9a-f]{16}$/);
expect(j.secret).toMatch(/^[0-9a-f]{64}$/);
return j;
}
test('config is public and says enabled + staleDays', async () => {
expect(await (await req('/api/p2p/config')).json()).toEqual({ ok: true, enabled: true, staleDays: 7 });
});
test('holdings need a valid device secret', async () => {
const dev = await newDevice();
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] } })).status).toBe(401);
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] }, dev: { ...dev, secret: 'e'.repeat(64) } })).status).toBe(401);
});
test('report accepts verified cids, returns unknown ones, and challenges the server-held file', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: true, items: [
{ cid: CID, size: bytes.length, videoId: 'dQw4w9WgXcQ' }, { cid: OTHER, size: 10 }, { cid: UNKNOWN, size: 5 }, { cid: 'nothex' },
] } })).json();
expect(r.accepted.sort()).toEqual([CID, OTHER].sort());
expect(r.unknown).toEqual([UNKNOWN]);
expect(r.challenges.length).toBe(1);
const ch = r.challenges[0];
expect(ch.cid).toBe(CID);
const good = createHash('sha256').update(bytes.subarray(ch.offset, ch.offset + ch.length)).digest('hex');
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: good }] } })).json();
expect(a).toEqual({ ok: true, passed: [CID], failed: [] });
const hs = await p2pDb.listHolders(CID);
expect(hs.find((h) => h.device_id === dev.deviceId).trust).toBe('challenged');
});
test('a wrong challenge answer removes the holder; a replayed answer is ignored', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID, size: bytes.length }] } })).json();
const ch = r.challenges[0];
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(a.failed).toEqual([CID]);
expect((await p2pDb.listHolders(CID)).some((h) => h.device_id === dev.deviceId)).toBe(false);
const again = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(again).toEqual({ ok: true, passed: [], failed: [] });
});
test('a full report without an item removes it; share=false withdraws everything', async () => {
const dev = await newDevice();
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID }, { cid: OTHER }] } });
expect((await p2pDb.activeHoldingsOf(dev.deviceId)).sort()).toEqual([CID, OTHER].sort());
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([OTHER]);
await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: false, items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([]);
expect((await p2pDb.getDevice(dev.deviceId)).share).toBe(0);
});
test('P2P_ENABLED=0 → routes answer 404 (config still says disabled)', async () => {
enabled = false;
try {
expect((await req('/api/p2p/device', { method: 'POST', body: {} })).status).toBe(404);
expect((await (await req('/api/p2p/config')).json()).enabled).toBe(false);
} finally { enabled = true; }
});
```
## Appendix C — frontend/hash-worker.js
```js
/* ============================================================================
* hash-worker.js — SHA-256 of a file already saved in OPFS, off the main
* thread, read in 4 MiB slices (never the whole video in memory).
*
* In: { name } file name under OPFS videos/ (e.g. "abc.mp4")
* Out: { ok: true, sha256, size } | { ok: false, error }
* ========================================================================== */
'use strict';
importScripts('/sha256.js');
self.onmessage = async (e) => {
const { name } = e.data || {};
try {
const root = await navigator.storage.getDirectory();
const dir = await root.getDirectoryHandle('videos');
const file = await (await dir.getFileHandle(String(name))).getFile();
const h = self.Sha256.create();
const STEP = 4 * 1024 * 1024;
for (let pos = 0; pos < file.size; pos += STEP) {
h.update(new Uint8Array(await file.slice(pos, pos + STEP).arrayBuffer()));
}
self.postMessage({ ok: true, sha256: h.hex(), size: file.size });
} catch (err) {
self.postMessage({ ok: false, error: err && err.message ? err.message : String(err) });
}
};
```
## Appendix D — frontend/p2p-client.js
```js
/* ============================================================================
* p2p-client.js — this device's side of peer-to-peer sharing
* (docs/p2p-architecture.md flows 2–3). window.P2PClient.
*
* start({ getSettings, getProfile }) called once from app.js boot()
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
*
* What it does, in order, each sync:
* 1. registers the device once (localStorage ytpDevice)
* 2. reconciles DeviceDB with the files really in OPFS (a record without a
* file is dropped; a file without a record is added as 'unhashed')
* 3. hashes 'unhashed' files and files not re-checked for 30 days, one at a
* time in hash-worker.js
* 4. reports the FULL holdings list (empty when sharing is off), answers the
* server's range challenges, and marks accepted files 'verified'
* P2P is ON by default: sharing runs unless settings.p2pShare === false or the
* server says P2P is disabled.
* ========================================================================== */
(function () {
'use strict';
const KEY = 'ytpDevice';
const REHASH_MS = 30 * 24 * 3600_000;
const RESYNC_MS = 6 * 3600_000;
let hooks = { getSettings: () => ({}), getProfile: () => '' };
let serverCfg = null;
let running = null;
let again = false;
let timer = null;
let lastSync = 0;
function device() {
try {
const d = JSON.parse(localStorage.getItem(KEY) || 'null');
return d && /^dev_[0-9a-f]{16}$/.test(d.deviceId) && /^[0-9a-f]{64}$/.test(d.secret) ? d : null;
} catch { return null; }
}
const authHeaders = () => { const d = device(); return d ? { 'X-Device': d.deviceId + '.' + d.secret } : {}; };
async function config() {
if (serverCfg) return serverCfg;
try {
const j = await (await fetch('/api/p2p/config')).json();
if (j && j.ok) serverCfg = j;
} catch { /* offline — try again next sync */ }
return serverCfg;
}
async function ensureDevice() {
const have = device();
if (have) return have;
const r = await fetch('/api/p2p/device', {
method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ fingerprint: window.getFingerprint ? window.getFingerprint() : '', profile: hooks.getProfile() || '' }),
});
const j = await r.json();
if (!j || !j.ok) throw new Error((j && j.error) || 'device registration failed');
const d = { deviceId: j.deviceId, secret: j.secret };
try { localStorage.setItem(KEY, JSON.stringify(d)); } catch { /* storage blocked */ }
return d;
}
function hashInWorker(name) {
return new Promise((resolve) => {
let w;
try { w = new Worker('/hash-worker.js'); } catch { resolve(null); return; }
w.onmessage = (e) => { w.terminate(); resolve(e.data && e.data.ok ? e.data : null); };
w.onerror = () => { w.terminate(); resolve(null); };
w.postMessage({ name });
});
}
async function sha256Range(videoId, offset, length) {
const bytes = window.OPFS && window.OPFS.readRange ? await window.OPFS.readRange(videoId, offset, length) : null;
if (!bytes) return null;
return window.Sha256.hex(bytes);
}
// DeviceDB ⇄ OPFS. Returns the records that describe a real file.
async function reconcile() {
const files = await window.OPFS.listVideos({ strict: true });
const byId = new Map(files.map((f) => [f.id, f]));
const recs = await window.DeviceDB.listFiles();
const out = [];
for (const r of recs) {
const f = byId.get(r.videoId);
if (!f) { await window.DeviceDB.deleteFile(r.videoId); continue; }
if (f.size !== r.size && r.size) { r.cid = null; r.state = 'unhashed'; r.size = f.size; await window.DeviceDB.putFile(r); }
r.name = f.name;
out.push(r);
byId.delete(r.videoId);
}
for (const f of byId.values()) {
if (String(f.id).startsWith('edit_')) continue; // edited cuts are never shared
const r = { videoId: f.id, cid: null, size: f.size, savedAt: Date.now(), lastCheckedAt: 0, state: 'unhashed' };
await window.DeviceDB.putFile(r);
out.push({ ...r, name: f.name });
}
return out;
}
async function hashPending(recs) {
const now = Date.now();
for (const r of recs) {
if (r.state !== 'unhashed' && now - (r.lastCheckedAt || 0) < REHASH_MS) continue;
const h = await hashInWorker(r.name);
if (!h) continue;
const changedCid = h.sha256 !== r.cid;
r.cid = h.sha256;
r.size = h.size;
r.lastCheckedAt = Date.now();
if (changedCid || r.state === 'unhashed') r.state = 'unverified';
await window.DeviceDB.putFile(r);
}
}
async function report(recs, dev) {
const share = hooks.getSettings().p2pShare !== false;
const items = recs.filter((r) => r.cid).map((r) => ({ cid: r.cid, size: r.size, videoId: r.videoId }));
const r = await fetch('/api/p2p/holdings', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ share, items: share ? items : [], profile: hooks.getProfile() || '' }),
});
if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; }
const j = await r.json();
if (!j || !j.ok) return null;
const accepted = new Set(j.accepted || []);
for (const rec of recs) {
if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); }
}
const answers = [];
for (const ch of j.challenges || []) {
const rec = recs.find((x) => x.cid === ch.cid);
if (!rec) continue;
const hex = await sha256Range(rec.videoId, ch.offset, ch.length);
if (hex) answers.push({ ...ch, sha256: hex });
}
if (answers.length) {
await fetch('/api/p2p/challenge', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ answers }),
}).catch(() => {});
}
return j;
}
async function syncOnce() {
if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null;
if (navigator.onLine === false) return null;
const cfg = await config();
if (!cfg || !cfg.enabled) return null;
const dev = await ensureDevice();
const recs = await reconcile();
await hashPending(recs);
const res = await report(recs, dev);
lastSync = Date.now();
return res;
}
// One sync at a time; a request during a sync schedules exactly one more.
function sync() {
if (running) { again = true; return running; }
running = syncOnce().catch(() => null).finally(() => {
running = null;
if (again) { again = false; sync(); }
});
return running;
}
function changed() {
clearTimeout(timer);
timer = setTimeout(sync, 5000);
}
function start(h) {
hooks = { ...hooks, ...(h || {}) };
const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1));
setTimeout(() => idle(() => sync()), 8000);
document.addEventListener('visibilitychange', () => {
if (document.visibilityState === 'visible' && Date.now() - lastSync > RESYNC_MS) sync();
});
}
window.P2PClient = { start, changed, sync, device, authHeaders, config };
}());
```

122
plans/queue/014-p2p-presence-hub-ceced8.md Normal file
View File

@@ -0,0 +1,122 @@
---
id: 014-p2p-presence-hub-ceced8
title: Add the /ws/p2p presence and signalling hub and the holders endpoint
created: 2026-09-29
depends_on: [013-device-identity-and-holdings-3ba493]
est_files: 6
---
# 014 — Presence + signalling hub, holders endpoint
## Objective
Implements flows 4–5 of `docs/p2p-architecture.md`. After this plan:
- `/ws/p2p` is a websocket hub (`server/p2p-hub.js`). A device authenticates with its
FIRST message `{type:'auth', device, secret}` (never in the URL — proxies log URLs),
gets `{type:'hello', peer}`, and is **online** while the socket is open (memory only).
It relays `{type:'signal', to, data}` between two authenticated online devices.
- `GET /api/p2p/holders?v=<videoId>` (or `?cid=`) lists each verified cid with its
holders: opaque `peer`, `online`, `lastVerifiedAt`, `stale`, `trust`, plus `serverHas`
and counts. Holder rows are NEVER hidden for age — `stale` only flags them.
Devices with sharing off are not listed. Device ids never appear in the payload.
- `P2PClient` keeps the socket open while sharing or receiving is on (both default ON),
with a 50 s keepalive (the server closes idle sockets after 120 s), and exposes
`onMessage(type, fn)`, `signal(to, data)`, `peer()`, `isConnected()`.
Pre-tested: hub/payload 3/3 (`bun:test`); two real browsers exchanged a signal through a
real Bun server (`plans/harness/presence-*.js`).
## Context the executor must NOT rediscover
- `plans/patches/014-p2p-hub-new.diff` creates `server/p2p-hub.js` (exports `createP2pHub`,
`holdersPayload`, `peerIdOf`) and `server/p2p-hub.test.js`.
- `plans/patches/014-p2p-client-presence.diff` patches `frontend/p2p-client.js` (from plan 013).
- `server/server.js` (~line 1778-1790):
```js
const remote = createRemoteHub({ requireSameNetwork: process.env.REMOTE_SAME_NETWORK === '1' });
const party = createPartyHub();
// Bun allows ONE websocket handler per server: party sockets are tagged
// (ws.data.hub === 'party'), everything else belongs to the remote relay.
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket : remote.websocket);
```
- `server/server.js` `Bun.serve({ … fetch(req, server) {` (~line 1926):
```js
const path = new URL(req.url).pathname;
if (path === '/ws/remote') return remote.upgrade(req, server, clientIpOf(req, server));
if (path === '/ws/party') return party.upgrade(req, server, clientIpOf(req, server));
return app.fetch(req, server);
```
- Plan 013 added in server.js: `async function fileForCid(cid)` and
`const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });`
(`p2p.gate` is a Hono middleware answering 404 when P2P is disabled).
## Steps
1. From the repo root:
```bash
git apply plans/patches/014-p2p-hub-new.diff
git apply plans/patches/014-p2p-client-presence.diff
```
STOP and report if either fails.
2. `server/package.json` "test" script — append ` && bun test ./p2p-hub.test.js`.
3. `server/server.js` — add import `import { createP2pHub, holdersPayload } from './p2p-hub.js';`
4. `server/server.js` — directly after `const p2p = registerP2pRoutes(app, { … });` add:
```js
const p2pHub = createP2pHub({ getDevice: p2pDb.getDevice, enabled: () => P2P.enabled });
// GET /api/p2p/holders?v=<videoId>|cid=<sha256> — who holds a copy. Rows are
// persistent; `stale` flags a holder not re-verified for P2P_STALE_DAYS.
app.get('/api/p2p/holders', p2p.gate, async (c) => {
const v = (c.req.query('v') || '').trim();
const cid = (c.req.query('cid') || '').trim().toLowerCase();
const hasCid = /^[0-9a-f]{64}$/.test(cid);
if (!hasCid && !/^[A-Za-z0-9_-]{6,64}$/.test(v)) return c.json({ ok: false, error: 'missing v or cid' }, 400);
const payload = await holdersPayload({
videoId: v, cid: hasCid ? cid : null, p2pDb, isOnline: p2pHub.isOnline,
staleDays: P2P.staleDays, serverHas: fileForCid,
});
return c.json(payload, 200, { 'Cache-Control': 'no-store' });
});
```
5. `server/server.js` — replace the `pickHub` line with:
```js
// …and P2P sockets are tagged ws.data.hub === 'p2p' (p2p-hub.js).
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket
: ws.data && ws.data.hub === 'p2p' ? p2pHub.websocket : remote.websocket);
```
(`p2pHub` is declared later in the file; that is fine — `pickHub` only runs once sockets exist.)
6. `server/server.js` `Bun.serve` fetch — after the `/ws/party` line add:
` if (path === '/ws/p2p') return p2pHub.upgrade(req, server);`
7. Copy nothing else; `frontend/index.html` / `sw.js` need no change (no new frontend files).
## Out of scope / do NOT touch
- `remote.js`, `party.js` and their behaviour. No UI (plan 015). No file transfer (plan 017).
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/presence-server.js >/tmp/ytp014.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node presence-check.mjs
kill $SRV; true
```
Expected: hub `3 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"peers":[true,true],"distinct":true,"sent":true,"got":[{"from":true,"data":{"hi":1}}]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

184
plans/queue/015-availability-ui-and-settings-3b9397.md Normal file
View File

@@ -0,0 +1,184 @@
---
id: 015-availability-ui-and-settings-3b9397
title: Show peer availability with stale markers and add Sharing settings
created: 2026-09-29
depends_on: [014-p2p-presence-hub-ceced8]
est_files: 6
---
# 015 — Availability line + Sharing settings
## Objective
Implements the UI half of flow 5 and the settings of `docs/p2p-architecture.md`.
After this plan:
- Under the now-playing title a small line shows peer availability, e.g.
`📡 On 3 devices · 1 online now · last checked 2d ago`. Hovering lists each holder
(`peer · online/offline · checked Xd ago (stale) · spot-checked`). When EVERY holder
is older than the server's `staleDays`, the line says `(not checked recently)` and is
styled dimmer — holders are **never hidden for age**. No holders → no line.
- Settings gets a "Sharing (peer-to-peer)" group: **Share my saved videos** and
**Get videos from other devices**, both ON by default, plus a status row
(registered / online peer id, verified/saved counts).
Pre-tested: `frontend/p2p-core.test.js` 5/5.
## Context the executor must NOT rediscover
- `plans/patches/015-p2p-core-new.diff` creates `frontend/p2p-core.js` (`window.P2PCore`:
`ago`, `formatAvailability(payload, now)`, `pickPeers(payload, cid)`) and its node test.
- `frontend/index.html:223-227`:
```html
<div id="nowPlayingMeta" class="now-meta hidden">
<div class="np-text">
<div class="np-title" id="npTitle"></div>
<div class="np-channel" id="npChannel"></div>
</div>
```
- `frontend/app.js` `function updateNowPlayingActions()` (~line 2389) begins
`if (!current || !current.meta) return; const id = current.meta.id;` and ends with the
save-button `if/else` chain followed by `}`.
- `frontend/app.js` `async function renderSettings()` (~line 7321). Its template contains:
```html
<div class="set-group">
<div class="set-group-title">Offline cache</div>
```
and further down the handlers:
```js
$('setAutoPreload').addEventListener('change', (e) => {
data.settings.autoPreload = e.target.checked;
persist();
if (e.target.checked) data.playlists.forEach(preloadPlaylist);
});
```
- `frontend/styles.css:737` — `.np-actions { display: flex; gap: 8px; flex-shrink: 0; }`;
the dim text colour token is `var(--text-dim)`.
- `window.P2PClient` (plans 013/014): `device()`, `isConnected()`, `peer()`, `sync()`.
`window.DeviceDB.listFiles()` (plan 012).
## Steps
1. From the repo root: `git apply plans/patches/015-p2p-core-new.diff` (STOP on failure).
2. `frontend/index.html` — after `<div class="np-channel" id="npChannel"></div>` add:
` <div class="np-avail hidden" id="npAvail" aria-live="polite"></div>`
and add ` <script src="p2p-core.js"></script>` directly after the `p2p-client.js` script line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-core.js',` directly after `'/p2p-client.js',`.
4. `frontend/styles.css` — directly after the `.np-actions { … }` line add:
```css
.np-avail { font-size: 12px; color: var(--text-dim); margin-top: 4px; cursor: default; }
.np-avail.stale { opacity: 0.6; font-style: italic; }
```
5. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` add:
```js
// P2P availability under the title (docs/p2p-architecture.md flow 5).
// Cached 60 s per video so the frequent updateNowPlayingActions() calls
// don't hammer the server.
const availCache = new Map(); // videoId -> { at, payload }
async function renderAvailability(id) {
const el = $('npAvail');
if (!el) return;
const off = data.settings.p2pShare === false && data.settings.p2pReceive === false;
if (!WEB || !window.P2PCore || off || navigator.onLine === false || String(id).startsWith('edit_')) {
el.classList.add('hidden');
return;
}
let hit = availCache.get(id);
if (!hit || Date.now() - hit.at > 60_000) {
hit = { at: Date.now(), payload: null };
availCache.set(id, hit);
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
hit.payload = r.ok ? await r.json() : null;
} catch { /* offline / disabled — no line */ }
if (availCache.size > 200) availCache.delete(availCache.keys().next().value);
}
if (!current || !current.meta || current.meta.id !== id) return; // moved on meanwhile
const f = window.P2PCore.formatAvailability(hit.payload, Date.now());
if (!f) { el.classList.add('hidden'); return; }
el.textContent = f.text;
el.title = f.title;
el.classList.toggle('stale', f.stale);
el.classList.remove('hidden');
}
```
6. `frontend/app.js` `updateNowPlayingActions` — directly after `const id = current.meta.id;` add
`renderAvailability(id);`
7. `frontend/app.js` `renderSettings` template — directly BEFORE the
`<div class="set-group">` whose title is `Offline cache`, insert:
```html
<div class="set-group">
<div class="set-group-title">Sharing (peer-to-peer)</div>
<label class="set-row">
<span>
Share my saved videos
<small>Other devices can download videos you saved when YouTube and the server can't provide them. Files are verified by their hash. On by default.</small>
</span>
<input id="setP2pShare" type="checkbox" ${data.settings.p2pShare !== false ? 'checked' : ''} />
</label>
<label class="set-row">
<span>
Get videos from other devices
<small>When a video is gone from YouTube and the server, download a verified copy from a device that has it. On by default.</small>
</span>
<input id="setP2pReceive" type="checkbox" ${data.settings.p2pReceive !== false ? 'checked' : ''} />
</label>
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
8. `frontend/app.js` `renderSettings` — directly after the `$('setAutoPreload').addEventListener(…);`
block add:
```js
$('setP2pShare').addEventListener('change', (e) => {
data.settings.p2pShare = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync(); // share off → withdraws every holding
});
$('setP2pReceive').addEventListener('change', (e) => {
data.settings.p2pReceive = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync();
});
(async () => {
const info = $('p2pDeviceInfo');
const cnt = $('p2pHoldingCount');
if (!info || !cnt) return;
const P = window.P2PClient;
const d = P && P.device();
info.textContent = !d ? 'Not registered yet'
: P.isConnected() ? `Online · peer ${P.peer()}` : 'Registered · not connected';
const files = window.DeviceDB ? await window.DeviceDB.listFiles() : [];
cnt.textContent = `${files.filter((f) => f.state === 'verified').length} verified / ${files.length} saved`;
})();
```
## Out of scope / do NOT touch
- No download button yet (plan 017 adds "Get from a device"). Do not change `p2p-client.js`.
- Do not show availability for edited cuts (`edit_…`).
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-core.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-core.js" frontend/index.html frontend/sw.js
grep -c "setP2pShare\|setP2pReceive\|renderAvailability(id)" frontend/app.js
```
Expected: `FRONT_OK`; `# fail 0`; `1` and `1`; app.js grep ≥ `5`.
Manual (optional, needs a running server with P2P data): open the app, play a video that has holders,
see the 📡 line; Settings shows the Sharing group with both boxes ticked.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

150
plans/queue/016-intake-and-server-verification-cfe031.md Normal file
View File

@@ -0,0 +1,150 @@
---
id: 016-intake-and-server-verification-cfe031
title: Let a device hand a file to the server for hashing and validation
created: 2026-09-29
depends_on: [015-availability-ui-and-settings-3b9397]
est_files: 9
---
# 016 — Intake: server-side verification of device files
## Objective
Implements flow 7 of `docs/p2p-architecture.md` — the owner's rule "a file must first
be downloaded by the server and checked before its hash is added to the server DB".
After this plan:
- `POST /api/p2p/intake` (device auth) opens a 30-min ticket; `PUT /api/p2p/intake/:ticket`
streams the bytes into `P2P_INTAKE_DIR` (never served). The SERVER hashes while writing,
a claimed cid must match, `validateMedia()` must pass, then `admitFile()` (malware scan
only when `P2P_MALWARE_SCAN=1`, off by default). Only then does the cid enter
`p2p_content` (`origin 'intake'`), and the uploader becomes a holder (`trust 'challenged'`).
- If the server has no copy of that video, the file is adopted into the media cache
byte-for-byte (`media.adoptFile`, new) so it can be streamed again. Otherwise deleted.
- Settings → Sharing shows "Verify & share N saved videos" for saves whose hash the server
doesn't know yet (old saves, fallback-path saves); one tap uploads them one by one.
Nothing uploads automatically in this plan.
Pre-tested: intake 4/4 (valid, hash-mismatch, truncated, non-media, size limits, ticket
reuse, scan-infected), media-cache 27/27 (incl. `adoptFile`), and the browser flow
(unknown → contribute → accepted) in Chromium.
## Context the executor must NOT rediscover
- Patches (apply in this order from the repo root):
- `plans/patches/016-p2p-intake-new.diff` → new `server/p2p-intake.js` (`registerIntakeRoutes`) + `server/p2p-intake.test.js`
- `plans/patches/016-media-cache-adopt.diff` → `adoptFile(id, src, { sha256, probe, meta })` in `server/media-cache.js` + a test
- `plans/patches/016-client-intake.diff` → `P2PClient.contribute(videoId, { cid, title, channel })`,
`P2PClient.unknownVideos()` in `frontend/p2p-client.js`; `OPFS.getFileObject(videoId)` in `frontend/opfs.js`
- `server/server.js` imports `{ createMediaCache, HIGH, LOW } from './media-cache.js'`; `FFMPEG`
const exists; ffprobe path is `process.env.FFPROBE_PATH || 'ffprobe'` (used in createMediaCache).
- Plan 013/014 added in server.js: `const p2p = registerP2pRoutes(app, …)` (exposes `gate`,
`requireDevice`) and `const p2pHub = createP2pHub(…)`. Plan 009 imported `admitFile`, `P2P`, `* as p2pDb`.
- Plan 015 added to `renderSettings()` the Sharing group ending with:
```html
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
and an `(async () => { const info = $('p2pDeviceInfo'); … cnt.textContent = …; })();` block.
- Settings buttons use `class="btn"`. `videoById(id)` (app.js ~7301) returns the known video object
(title/channel) or undefined. `toast(msg)` shows a toast.
## Steps
1. Apply the three patches (STOP and report on any failure):
```bash
git apply plans/patches/016-p2p-intake-new.diff
git apply plans/patches/016-media-cache-adopt.diff
git apply plans/patches/016-client-intake.diff
```
2. `server/package.json` "test" script — append ` && bun test --timeout 60000 ./p2p-intake.test.js`.
3. `server/server.js`:
a. change the media-cache import to `import { createMediaCache, HIGH, LOW, validateMedia } from './media-cache.js';`
b. add `import { registerIntakeRoutes } from './p2p-intake.js';`
c. directly after the `/api/p2p/holders` route (plan 014) add:
```js
// Device → server intake: the server hashes, validates (and scans when
// P2P_MALWARE_SCAN=1) before a cid is admitted. docs/p2p-architecture.md flow 7.
const intake = registerIntakeRoutes(app, {
cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile,
validateMedia: (path, expected, opts) => validateMedia(path, expected,
{ ...opts, ffmpeg: FFMPEG, ffprobe: process.env.FFPROBE_PATH || 'ffprobe' }),
adopt: (videoId, path, info) => media.adoptFile(videoId, path, info),
});
```
4. `docker-compose.yml` — under the P2P env lines from plan 008 add the commented lines:
```yaml
# P2P_INTAKE_DIR: "/app/data/p2p-intake" # quarantine for device uploads (never served)
# P2P_INTAKE_MAX_BYTES: "3221225472" # 3 GiB
```
5. `frontend/app.js` `renderSettings` template — directly BEFORE the closing `</div>` of the
Sharing group (i.e. after the `This device` row), insert:
```html
<div class="set-row hidden" id="p2pContributeRow">
<span>
Saved videos the server can't verify yet
<small>Uploading lets the server check them (hash + media validation) so other devices can get them. Uses your upload bandwidth.</small>
</span>
<button id="p2pContributeBtn" class="btn">Verify &amp; share</button>
</div>
```
6. `frontend/app.js` `renderSettings` — inside the plan-015 `(async () => { … })();` block, after the
`cnt.textContent = …;` line, add:
```js
const unknown = window.P2PClient ? await window.P2PClient.unknownVideos() : [];
const row = $('p2pContributeRow');
const btn = $('p2pContributeBtn');
if (row && btn && unknown.length) {
row.classList.remove('hidden');
btn.textContent = `Verify & share ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`;
btn.onclick = async () => {
btn.disabled = true;
let ok = 0;
for (let i = 0; i < unknown.length; i++) {
const v = videoById(unknown[i]) || {};
btn.textContent = `Uploading ${i + 1} of ${unknown.length}…`;
const r = await window.P2PClient.contribute(unknown[i], { title: v.title || '', channel: v.channel || '' });
if (r && r.ok) ok++;
}
toast(`Verified ${ok} of ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`);
btn.disabled = false;
row.classList.add('hidden');
};
}
```
## Out of scope / do NOT touch
- No automatic uploads (plan 018 adds server-requested ones). No UI for the admin (plan 019).
- Never serve files from `P2P_INTAKE_DIR`; never add a `/api/p2p/intake` GET.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — install it (apt-get install -y ffmpeg); intake/media tests need it"
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/app.js frontend/p2p-client.js frontend/opfs.js && echo FRONT_OK
cd server
bun ../plans/harness/intake-server.js >/tmp/ytp016.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node intake-check.mjs
kill $SRV; true
```
Expected: intake `4 pass`; media-cache `27 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`;
browser JSON `{"firstUnknown":1,"unknown":["upAAAAAAAA9"],"contribute":{"ok":true,"adopted":false,"cidOk":true},"secondAccepted":1,"secondUnknown":0}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

179
plans/queue/017-peer-transfer-1faaa7.md Normal file
View File

@@ -0,0 +1,179 @@
---
id: 017-peer-transfer-1faaa7
title: Download a verified file from another device over WebRTC
created: 2026-09-29
depends_on: [016-intake-and-server-verification-cfe031]
est_files: 6
---
# 017 — Peer download over WebRTC
## Objective
Implements flow 6 of `docs/p2p-architecture.md`. After this plan:
- `window.P2PTransfer` serves this device's saved files to other devices (only while
`settings.p2pShare !== false`, one upload at a time) and downloads from them over a
WebRTC data channel, signalled through `/ws/p2p` (plan 014). STUN only.
- The receiver writes through `p2p-recv-worker.js` into `videos/<id>.p2p.part` while
hashing, and renames to `<id>.mp4` ONLY when the SHA-256 equals the cid. A wrong,
corrupted or short file leaves nothing behind.
- When a video fails to load (YouTube and the server both failing), and
`settings.p2pReceive !== false`, the player shows **📡 Get it from a device (N online)**
next to **↻ Retry**. On success the video is saved offline, registered as `verified`
(so this device becomes a holder at its next report) and plays.
Pre-tested in Chromium with two browser contexts and a real hub
(`plans/harness/transfer-*.js`): 5 MiB file delivered and verified in ~2.5 s; unknown cid →
`peer declined: not here`; holder with a corrupted same-size file → `content hash mismatch`
and no file kept.
## Context the executor must NOT rediscover
- `plans/patches/017-p2p-transfer-new.diff` creates `frontend/p2p-transfer.js` and
`frontend/p2p-recv-worker.js`.
- Script order in `frontend/index.html` after plan 015:
`… sha256.js, device-db.js, p2p-client.js, p2p-core.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` has `'/p2p-core.js',` (plan 015).
- `frontend/app.js`:
- `Player.loadVideo` catch block (~line 1735):
```js
} catch (err) {
showSpinner(false);
toast('⚠ ' + err.message);
// Show retry button in the player pane
const retry = els.playerPane.querySelector('.retry-btn');
if (retry) retry.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn';
btn.textContent = '↻ Retry';
btn.addEventListener('click', () => {
btn.remove();
Player.loadVideo(videoObj, { preferStream, resume, reveal, nocache });
});
els.playerPane.appendChild(btn);
}
```
- globals/helpers: `downloading` (Set, ~line 372), `cachedIds`, `cacheMutations`,
`markCardCacheState(id, state)` (~1351), `updateDownloadBadge()` (~1313), `renderSidebar()`,
`recordDeviceFile(id, res)` (plan 012), `toast()`.
- `boot()` has (plan 013):
```js
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
- `window.P2PCore.pickPeers(payload, cid)` → `[{ peer, cid, size }]` online holders (plan 015).
## Steps
1. From the repo root: `git apply plans/patches/017-p2p-transfer-new.diff` (STOP on failure).
2. `frontend/index.html` — add ` <script src="p2p-transfer.js"></script>` directly after the `p2p-core.js` line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-transfer.js',` and `'/p2p-recv-worker.js',` directly after `'/p2p-core.js',`.
4. `frontend/styles.css` — append at the end of the file:
```css
/* P2P: "Get it from a device" next to Retry (plan 017) */
.peer-btn { margin-left: 8px; }
```
5. `frontend/app.js` `boot()` — inside the `if (WEB && window.P2PClient) { … }` block, after the
`start(...)` call add:
```js
if (window.P2PTransfer) window.P2PTransfer.start({ canShare: () => data.settings.p2pShare !== false });
```
6. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` (next to plan 015's
`renderAvailability`) add:
```js
// Fetch a verified copy from another device (docs/p2p-architecture.md flow 6).
// Download-then-play: on success the file is saved offline like any save.
async function getFromPeers(videoObj, c, peers, onProgress) {
const id = videoObj.id;
downloading.add(id);
markCardCacheState(id, 'downloading');
updateDownloadBadge();
try {
const res = await window.P2PTransfer.download({ videoId: id, cid: c.cid, size: c.size, peers, onProgress });
if (res.ok) {
cachedIds.add(id);
cacheMutations++;
recordDeviceFile(id, { sha256: res.sha256, expectedSha: c.cid, size: res.size });
toast(`Saved “${videoObj.title || id}” from another device ✓`);
}
return res;
} finally {
downloading.delete(id);
markCardCacheState(id, cachedIds.has(id) ? 'cached' : 'none');
updateDownloadBadge();
renderSidebar();
}
}
// Offered when a video won't load: only if some device holding it is online now.
async function offerPeerDownload(videoObj) {
if (!WEB || !window.P2PTransfer || !window.P2PCore || data.settings.p2pReceive === false) return;
const id = videoObj && videoObj.id;
if (!id || videoObj.custom || cachedIds.has(id)) return;
let payload = null;
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
payload = r.ok ? await r.json() : null;
} catch { return; }
const c = payload && Array.isArray(payload.cids) ? payload.cids[0] : null;
const peers = c ? window.P2PCore.pickPeers(payload, c.cid) : [];
if (!peers.length || !els.playerPane.querySelector('.retry-btn')) return; // nobody online / user moved on
const old = els.playerPane.querySelector('.peer-btn');
if (old) old.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn peer-btn';
btn.textContent = `📡 Get it from a device (${peers.length} online)`;
btn.addEventListener('click', async () => {
btn.disabled = true;
const res = await getFromPeers(videoObj, c, peers, (got, total) => {
btn.textContent = `📡 ${Math.min(99, Math.round((got / total) * 100))}%…`;
});
if (res.ok) {
els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());
Player.loadVideo(videoObj);
} else {
btn.disabled = false;
btn.textContent = '📡 Try again';
toast('⚠ ' + res.error);
}
});
els.playerPane.appendChild(btn);
}
```
7. `frontend/app.js` `Player.loadVideo` catch block — directly after `els.playerPane.appendChild(btn);`
add `offerPeerDownload(videoObj);`. Also, in the Retry button's click handler, change
`btn.remove();` to
`els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());` so the peer button
goes away with it.
## Out of scope / do NOT touch
- No TURN server, no progressive (streaming) playback from peers, no partial seeding.
- Don't change `p2p-client.js` or the server.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-transfer.js frontend/p2p-recv-worker.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-transfer.js\|p2p-recv-worker.js" frontend/index.html frontend/sw.js
cd server && bun install >/dev/null 2>&1
bun ../plans/harness/transfer-server.js >/tmp/ytp017.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node transfer-check.mjs 2>&1 | grep -v "status of 500"
kill $SRV; true
```
Expected: `FRONT_OK`; `# fail 0`; index.html `1`, sw.js `2`; browser JSON containing
`"good":{"ok":true`, `"list":[["gotAAAAAAAA",5243657]]`, `"bad":{"ok":false,"error":"peer declined: not here"}`,
`"corrupt":{"r":{"ok":false,"error":"content hash mismatch"},"names":["gotAAAAAAAA.mp4"]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

120
plans/queue/018-server-rehydrate-from-peer-4fb8bd.md Normal file
View File

@@ -0,0 +1,120 @@
---
id: 018-server-rehydrate-from-peer-4fb8bd
title: Restore an evicted server copy from an online holder
created: 2026-09-29
depends_on: [017-peer-transfer-1faaa7]
est_files: 6
---
# 018 — Rehydrate the server from a device
## Objective
Implements flow 8 of `docs/p2p-architecture.md`: the owner's "the original source can go
offline, the video stays reachable from devices". After this plan, when `/api/streams`
fails at the source (removed/private video, yt-dlp failing) AND the server has no ready
copy AND a device holding a verified cid of that video is online with sharing on:
- the server sends that ONE device `{type:'upload-request', videoId, cid}` over `/ws/p2p`
(at most once per video per 10 min) and answers
`503 { ok:false, restoring:true, error:'…try again in a minute.' }`;
- the device uploads the file through intake with `restore: true`; intake accepts a known
cid again ONLY when the server no longer holds those bytes, re-hashes + re-validates,
and adopts it into the media cache — the next play is served from the server again.
Pre-tested: rehydrator + restore intake (`bun:test` hub 4/4, intake 5/5) and the whole loop in
Chromium (`plans/harness/rehydrate-*.js`: first request → `restoring:true`, device uploads,
server adopts the exact cid).
## Context the executor must NOT rediscover
- Patches (apply in order):
- `plans/patches/018-server-rehydrate.diff` — `server/p2p-hub.js` gains
`createRehydrator({ p2pDb, hub, hasServerCopy, enabled, now })`; `server/p2p-intake.js` gains the
`restore` body flag and a `serverHasCid(cid)` dep; tests for both.
- `plans/patches/018-client-restore.diff` — `frontend/p2p-client.js` answers `upload-request`
(only when sharing is on, only for the exact cid it holds, one at a time) and `contribute()`
accepts `restore`.
- `server/server.js` `/api/streams` handler ends (~line 623):
```js
} catch (err) {
return c.json({ ok: false, error: err.message }, 500);
}
});
```
(this catch covers the yt-dlp resolve path; the cached-copy path returned earlier).
- Plans 013–016 created in server.js: `fileForCid(cid)`, `const p2p = registerP2pRoutes(…)`,
`const p2pHub = createP2pHub(…)`, the `/api/p2p/holders` route, and
`const intake = registerIntakeRoutes(app, { cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile, validateMedia: …, adopt: … });`
- `media.getReady(videoId)` resolves the ready row or null.
## Steps
1. From the repo root (STOP on failure):
```bash
git apply plans/patches/018-server-rehydrate.diff
git apply plans/patches/018-client-restore.diff
```
2. `server/server.js` — change the import from `./p2p-hub.js` to
`import { createP2pHub, holdersPayload, createRehydrator } from './p2p-hub.js';`
3. `server/server.js` — in the `registerIntakeRoutes(app, { … })` options add
`serverHasCid: async (cid) => !!(await fileForCid(cid)),`
4. `server/server.js` — directly after the `const intake = registerIntakeRoutes(…);` statement add:
```js
// Source gone + server copy evicted → ask one online holder to send it back
// through intake (docs/p2p-architecture.md flow 8).
const p2pRehydrate = createRehydrator({
p2pDb, hub: p2pHub, enabled: () => P2P.enabled,
hasServerCopy: async (id) => !!(await media.getReady(id).catch(() => null)),
});
```
5. `server/server.js` `/api/streams` — replace the final catch block shown in Context with:
```js
} catch (err) {
// The source failed. If a device holds a verified copy, ask it to send one
// to the server so the video comes back (P2P flow 8).
let restoring = false;
try { restoring = await p2pRehydrate(videoId); } catch { /* best effort */ }
if (restoring) {
return c.json({
ok: false, restoring: true,
error: 'This video is unavailable at the source — a device that has it is sending a copy to the server. Try again in a minute.',
}, 503);
}
return c.json({ ok: false, error: err.message }, 500);
}
```
(`p2pRehydrate` is declared further down the file; it is only called at request time, after startup.)
## Out of scope / do NOT touch
- The client error UI (the existing toast + Retry + plan-017 peer button already cover it).
- Do not ask more than one device per video per 10 minutes; do not upload without sharing on.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -3
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -3
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/rehydrate-server.js >/tmp/ytp018.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node rehydrate-check.mjs
kill $SRV; true
```
Expected: hub `4 pass`; intake `5 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"accepted":1,"first":{"ok":false,"restoring":true,"error":"restoring from a device"},"adopted":[["goneAAAAAAA",true]],"second":{"ok":false,"error":"source unavailable"}}`
(the harness's stand-in `/api/streams` has no media cache, so the second answer is expected).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

169
plans/queue/019-admin-p2p-panel-4dc623.md Normal file
View File

@@ -0,0 +1,169 @@
---
id: 019-admin-p2p-panel-4dc623
title: Add a P2P panel to the admin page
created: 2026-09-29
depends_on: [018-server-rehydrate-from-peer-4fb8bd]
est_files: 3
---
# 019 — Admin P2P panel
## Objective
Give the admin one place to see and steer peer-to-peer sharing
(`docs/p2p-architecture.md`). After this plan `/admin` has a **Peer-to-peer** section showing:
- config: P2P on/off, malware scan on/off (with a note that it is OFF by default and hashing +
validation always run), stale-after days, retention thresholds;
- counts: verified content, revoked, devices, active holder rows, cids with ≥1 holder,
devices online now, open/active intake uploads;
- the 30 most recently verified files (video, cid prefix, origin, scan, size, verified at,
holder count) with a **Revoke** button (revoked cids are never offered to or accepted
from devices again — plan 013 only accepts `verified` cids).
## Context the executor must NOT rediscover
- Admin auth middleware: `notes.requireAdminOrToken` (returned by `registerNoteRoutes`,
`server/server.js` ~line 1823). Used like `app.get('/api/admin/media', requireAdminOrToken, async (c) => …)`.
- In server.js (plans 008–018): `P2P` (config), `p2pDb` (module), `p2pHub.onlineCount()`,
`intake.openTickets()`, `intake.active()`.
- `server/p2p-db.js` has `rowsOf`, `p2pStats()`, `revokeContent(cid)`.
- `frontend/admin.html`:
- sections are `<section>…</section>` inside `<div id="app">`; the "Recent edits" section
(~line 190) starts with
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Recent edits</h2>
```
- helpers inside the script IIFE: `$(id)`, `esc(s)`, `api(path, opts)` (JSON in/out,
`opts.body` is JSON-encoded), classes `tbl`, `scroll`, `muted`, `row`, `spacer`.
- `async function boot()` calls `loadTokens(); loadRevs(); loadUploads();`.
## Steps
1. `server/p2p-db.js` — append:
```js
// Admin panel (plan 019): newest verified/revoked files with their holder counts.
export async function recentContent(limit = 30) {
const r = await db.execute({
sql: `SELECT c.cid, c.video_id, c.size, c.height, c.vcodec, c.origin, c.status, c.scan, c.verified_at, c.meta,
(SELECT COUNT(*) FROM p2p_holders h WHERE h.cid = c.cid AND h.status = 'active') AS holders
FROM p2p_content c ORDER BY c.verified_at DESC LIMIT ?`,
args: [limit],
});
return rowsOf(r).map((x) => ({ ...x, holders: Number(x.holders) || 0 }));
}
```
2. `server/server.js` — directly after the `const p2pRehydrate = createRehydrator({…});` statement add:
```js
// Admin: P2P overview + revoke (frontend/admin.html → Peer-to-peer).
app.get('/api/admin/p2p', notes.requireAdminOrToken, async (c) => c.json({
ok: true,
config: {
enabled: P2P.enabled, malwareScan: P2P.malwareScan, staleDays: P2P.staleDays,
keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
},
stats: { ...(await p2pDb.p2pStats()), online: p2pHub.onlineCount(), intakeOpen: intake.openTickets(), intakeActive: intake.active() },
recent: await p2pDb.recentContent(30),
}, 200, { 'Cache-Control': 'no-store' }));
app.post('/api/admin/p2p/revoke', notes.requireAdminOrToken, async (c) => {
const body = await c.req.json().catch(() => ({}));
const cid = String(body.cid || '').toLowerCase();
if (!/^[0-9a-f]{64}$/.test(cid)) return c.json({ ok: false, error: 'bad cid' }, 400);
await p2pDb.revokeContent(cid);
console.warn(`[p2p] admin revoked ${cid.slice(0, 12)}`);
return c.json({ ok: true });
});
```
3. `frontend/admin.html` — directly BEFORE the "Recent edits" `<section>` insert:
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Peer-to-peer</h2>
<span class="spacer"></span>
<button id="p2pRefreshBtn">↻ Refresh</button>
</div>
<p class="muted">Devices share verified copies with each other (docs/p2p-architecture.md). A file's hash is only
added after the server itself hashed and validated it. The malware scan is off by default
(<code>P2P_MALWARE_SCAN=1</code> to enable). Holders are never expired — ones not re-checked for the
stale period are only marked stale.</p>
<div id="p2pSummary" class="msg"></div>
<div class="scroll"><table class="tbl" id="p2pTable"></table></div>
</section>
```
4. `frontend/admin.html` script — directly ABOVE `async function boot() {` add:
```js
async function loadP2p() {
const j = await api('/api/admin/p2p');
if (!j.ok) { $('p2pSummary').textContent = j.error || 'P2P unavailable'; return; }
const c = j.config, s = j.stats;
$('p2pSummary').textContent =
`P2P ${c.enabled ? 'ON' : 'OFF'} · malware scan ${c.malwareScan ? 'ON' : 'off'} · stale after ${c.staleDays} d · ` +
`keep ≥${c.keepMinViews} views/${c.keepDays} d or played in ${c.keepRecentDays} d — ` +
`${s.content} verified (${s.revoked} revoked) · ${s.devices} devices, ${s.online} online · ` +
`${s.holders} holdings over ${s.heldCids} files · intake ${s.intakeActive} running / ${s.intakeOpen} open`;
const mb = (b) => (Number(b) / 1048576).toFixed(1) + ' MB';
$('p2pTable').innerHTML = '<tr><th>Video</th><th>cid</th><th>Origin</th><th>Scan</th><th>Size</th><th>Verified</th><th>Holders</th><th></th></tr>' +
j.recent.map((r) => {
let title = '';
try { title = JSON.parse(r.meta || '{}').title || ''; } catch { /* no meta */ }
return `<tr${r.status === 'revoked' ? ' class="muted"' : ''}><td>${esc(title || r.video_id)}<br><small>${esc(r.video_id)}</small></td>` +
`<td><code>${esc(r.cid.slice(0, 12))}</code></td><td>${esc(r.origin)}</td><td>${esc(r.scan)}</td><td>${mb(r.size)}</td>` +
`<td>${esc(new Date(Number(r.verified_at)).toLocaleString())}</td><td>${r.holders}</td>` +
`<td>${r.status === 'verified' ? `<button data-revoke="${esc(r.cid)}">Revoke</button>` : 'revoked'}</td></tr>`;
}).join('');
}
$('p2pRefreshBtn').addEventListener('click', loadP2p);
$('p2pTable').addEventListener('click', async (e) => {
const b = e.target.closest('[data-revoke]');
if (!b || !confirm('Revoke this file? Devices will stop sharing it and the server will never accept it again.')) return;
const j = await api('/api/admin/p2p/revoke', { method: 'POST', body: { cid: b.dataset.revoke } });
if (!j.ok) alert(j.error || 'failed');
loadP2p();
});
```
5. `frontend/admin.html` `boot()` — `loadUploads();` appears 3 times in the file; use ONLY the one
inside `async function boot()`, i.e. this exact pair of lines:
```js
loadUploads();
const want = videoIdFrom(new URLSearchParams(location.search).get('v') || '');
```
and insert ` loadP2p();` between them.
## Out of scope / do NOT touch
- No editing of config from the page (env vars stay the source of truth).
- `admin.html` is never cached by the SW — no `sw.js` change.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1 && [ -e public ] || ln -s ../frontend public
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
DBDIR=$(mktemp -d); DB_PATH=$DBDIR/t.db MEDIA_DIR=$DBDIR/media ADMIN_PASSWORD=test-pass PORT=3995 bun server.js >/tmp/ytp019.log 2>&1 & SRV=$!; sleep 4
curl -s -c /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"password":"test-pass"}' http://localhost:3995/api/admin/login
echo
curl -s -b /tmp/ytp019.jar http://localhost:3995/api/admin/p2p | head -c 400; echo
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3995/api/admin/p2p
curl -s -b /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"cid":"nope"}' http://localhost:3995/api/admin/p2p/revoke
echo
kill $SRV; true
grep -c "loadP2p" ../frontend/admin.html
```
Expected: `SERVER_OK`; every test file `0 fail`; login `{"ok":true…}`; the p2p call returns
`{"ok":true,"config":{"enabled":true,"malwareScan":false,"staleDays":7,…},"stats":{"content":0,…},"recent":[]}`;
without the cookie `401`; bad cid `{"ok":false,"error":"bad cid"}`; grep ≥ `3`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

166
server/fixtures/innertube-search.json Normal file
View File

@@ -0,0 +1,166 @@
{
"contents": {
"twoColumnSearchResultsRenderer": {
"primaryContents": {
"sectionListRenderer": {
"contents": [
{
"itemSectionRenderer": {
"contents": [
{
"videoRenderer": {
"videoId": "nQWFzMvCfLE",
"title": {
"runs": [
{
"text": "What A Beautiful Name - Hillsong Worship"
}
],
"accessibility": {
"accessibilityData": {
"label": "What A Beautiful Name - Hillsong Worship 5 minutes, 43 seconds"
}
}
},
"ownerText": {
"runs": [
{
"text": "Hillsong Worship",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UC4q12NoPNySbVqwpw4iO5Vg",
"canonicalBaseUrl": "/channel/UC4q12NoPNySbVqwpw4iO5Vg"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "5 minutes, 43 seconds"
}
},
"simpleText": "5:43"
}
}
},
{
"shelfRenderer": {
"title": {
"simpleText": "ignored"
}
}
},
{
"videoRenderer": {
"videoId": "8YKJVx4qxcs",
"title": {
"runs": [
{
"text": "Best Hillsong Worship Songs of All Time | Nonstop Christian Music Playlist"
}
],
"accessibility": {
"accessibilityData": {
"label": "Best Hillsong Worship Songs of All Time | Nonstop Christian Music Playlist 1 hour, 59 minutes"
}
}
},
"ownerText": {
"runs": [
{
"text": "Top Praise and Worship Songs",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UCNp42By0x1viJ1XSCHYV38g",
"canonicalBaseUrl": "/@TopPraiseAndWorshipSongs7"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "1 hour, 59 minutes, 47 seconds"
}
},
"simpleText": "1:59:47"
}
}
},
{
"videoRenderer": {
"videoId": "_1HGZ_9aRhI",
"title": {
"runs": [
{
"text": "Hillsong Worship Best Praise Songs Collection 2023 \u2013 Gospel Christian Songs Of Hillsong Worship"
}
],
"accessibility": {
"accessibilityData": {
"label": "Hillsong Worship Best Praise Songs Collection 2023 \u2013 Gospel Christian Songs Of Hillsong Worship 1 hour, 17 minutes"
}
}
},
"ownerText": {
"runs": [
{
"text": "Hallelujah Harmony",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UC1dEcyKOYiqdHwcVUmbPudw",
"canonicalBaseUrl": "/@HallelujahHarmony68"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "1 hour, 17 minutes, 50 seconds"
}
},
"simpleText": "1:17:50"
}
}
},
{
"videoRenderer": {
"videoId": "LIVEid12345",
"title": {
"runs": [
{
"text": "Live now"
}
]
},
"ownerText": {
"runs": [
{
"text": "Some Church",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UCxxxxxxxxxxxxxxxxxxxxxx"
}
}
}
]
}
}
}
]
}
},
{
"continuationItemRenderer": {}
}
]
}
}
}
}
}