josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Queue executable speed and peer-to-peer sharing plans with tested patches and harnesses

Browse Source
This commit is contained in:
Claude
2026-09-29 19:03:12 +00:00
parent a0ffc1b493
commit c17bd4c9ac
54 changed files with 6369 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
docs/improvement-plan.md
View File

@@ -1,7 +1,25 @@
# ytplayer — Speed & Features Plan (2026-09-29)
Scope: **make the app faster** (startup, search, pressing play, moving around)
and **add new features**. Hardening/refactor work is intentionally out of scope.
and **add new features**, led by **peer-to-peer video sharing**. Hardening/refactor
work is intentionally out of scope.
## Integrated roadmap (what is queued and executable today)
Everything below is broken into step-by-step plans a smaller model can execute
without exploring the code — see `plans/INDEX.md`. All 19 were dry-run end to end.
| Order | Plans | What it delivers | Source |
|-------|-------|------------------|--------|
| 1 | `001` | Timing marks (`ytp:boot`, `ytp:search`, `ytp:tap-to-play`) + yt-dlp duration logs | A "Measuring" |
| 2 | `002`–`003` | Compressed + ETagged shell (app.js 426 → 94 KB), self-hosted fonts | A1.1–A1.3 |
| 3 | `004`–`005` | One yt-dlp resolve per video at a time; resolve likely next plays before the tap | A3.1–A3.2 |
| 4 | `006`–`007` | Search via InnerTube (~0.8 s, yt-dlp fallback); long-lived yt-dlp workers | A2.1, A3.3 |
| 5 | `008`–`019` | **Peer-to-peer sharing** — design and rules in `docs/p2p-architecture.md` | Part B0 |
Not queued yet (write plans for them with `/plan-queue` when reached): A1.4 minify,
A1.5 deferred boot work, A1.6 lazy modules, A2.2 suggestions, A2.3/A2.4, A3.4/A3.5,
A4 (VPS media cache), A5 polish, and the Part B feature list.
## Where the time goes today (measured against prod)
@@ -99,6 +117,21 @@ before the user taps.**
## Part B — New features
### B0. Peer-to-peer video sharing (queued: plans 008–019)
The server keeps the user list, the video list and metadata; every validated copy
gets a content id = SHA-256 of its bytes; devices that save a video become persistent
**holders**; when YouTube and the server copy are gone, devices serve each other over
WebRTC and can restore the server's copy. Rules the owner set (2026-09-29):
**P2P is on by default**, the **server malware scan is off by default** (hashing +
media validation always run), and holder records are **persistent** — the UI shows
when each was last verified and marks old ones **stale** instead of dropping them.
Full design, data model, flows and security rules: `docs/p2p-architecture.md`.
The earlier phase 02–06 draft is superseded; its differences are listed at the end of
that document.
### Other features (not queued yet)
Ranked by fit with how the app is actually used (worship sets, sing-alongs,
offline playback). Each is sized; most reuse machinery that already exists.
@@ -144,16 +177,15 @@ offline playback). Each is sized; most reuse machinery that already exists.
## Suggested order
1. **Week 1 — quick speed wins:** add the timing marks (Measuring), A1.1
compression, A1.2 ETags, A1.3 fonts, A3.1 coalescing, A3.2 warm-on-intent,
A3.4 instant UI. These are all small, and together they fix most of what
feels slow today.
2. **Week 2:** A2.1 InnerTube search + A2.2 suggestions, A1.4 defer/minify.
3. **Week 3:** A3.3 persistent yt-dlp worker, A1.5 deferred boot work, then
measure the WireGuard link (A4) and decide on the VPS media cache.
4. **Then features**, starting with the small high-fit ones: section loops,
1. **Run the queue** (`/run-queue`): plans `001`–`007` (speed), then `008`–`019`
(peer-to-peer). Deploy after `007` and measure with the `ytp:*` marks and the
`[ytdlp]` log lines before starting P2P.
2. **Next speed plans to write:** A3.4 instant UI on tap, A2.2 suggestions,
A1.5 deferred boot work, then measure the WireGuard link (A4) and decide on the
VPS media cache.
3. **Then features**, starting with the small high-fit ones: section loops,
confidence monitor, count-in, lyrics search, stats wrap-up — then
service plans and chord charts.
Each row is sized to be one commit (CLAUDE.md commit rules) and can be queued
with the `plan-queue` skill.
Each item is sized to be one commit (CLAUDE.md commit rules); queue new ones with
the `plan-queue` skill.

155
docs/p2p-architecture.md Normal file
View File

@@ -0,0 +1,155 @@
# Peer-to-peer video sharing — architecture (2026-09-29)
Source of truth for plans `008`–`019` in `plans/queue/`. Every P2P plan links here
instead of repeating the rules. It replaces the earlier "phase 02–06" draft; the
differences are listed at the end.
## What the owner asked for
> Peer-to-peer saving of videos. The server stores the user list, the video list and
> metadata. While the original source is online, a video is available for streaming
> and download. When it is downloaded to a device, the server records that device in
> the list of holders, so the video stays reachable from devices after the source is
> gone. Top or recent files stay on the server under a total space limit; files that
> don't meet the criteria (e.g. number of views, also stored on the server) are
> deleted first. The video id is the file hash of the highest-quality copy. The
> database grows over time. Each device has its own database that can be synced or
> added to the server's, with verification that the file exists. A file must first be
> downloaded by the server and checked before its hash is added to the server DB.
Corrections to the earlier draft (owner, 2026-09-29):
1. **P2P is ON by default** (server and every client).
2. **The server's malware scan is OFF by default** (admin opt-in). Hashing and the
media validation gate are ALWAYS on and cannot be turned off.
3. **Holder records are persistent, not short-lived leases.** A device stays listed
as a holder until it says the file is gone, fails a check, or its reported list no
longer contains it. The UI shows when each holder was last verified and marks it
**stale** when that is older than `P2P_STALE_DAYS` (default 7). Online-right-now is
a separate, live signal.
## Vocabulary
| Term | Meaning |
|------|---------|
| **source** | Where bytes originally come from: YouTube (via yt-dlp) or a server upload (`upl_…`). |
| **video id** | Existing ids (`dQw4w9WgXcQ`, `upl_…`). Still used everywhere in the app and API. |
| **content id (`cid`)** | Lowercase hex SHA-256 of the exact file bytes. The P2P identity of a file. One video id can have several cids over time (a better master, the HEVC copy). |
| **master** | The best copy the server keeps for a video: today the validated ≤720p H.264+AAC faststart MP4 in `MEDIA_DIR` (`<id>.<gen>.mp4`). The compression lane's HEVC copy is a second, separately hashed file. Raising the master quality later just creates new cids linked by `video_id`. |
| **verified content** | A `p2p_content` row. Exists only after the server itself held the complete bytes, computed the SHA-256 itself and `validateMedia()` passed (plus the malware scan if enabled). |
| **holder** | A device that reported holding a cid. Row in `p2p_holders`, never deleted by time. |
| **online** | The device has an open `/ws/p2p` socket right now (in memory only). |
| **stale** | `now - last_verified_at > P2P_STALE_DAYS`. Shown in the UI, still listed. |
## Data model (server, libsql — grows forever)
Added by plan 008 in `server/p2p-db.js` (`initP2pSchema()` runs after `initDb()`):
```sql
p2p_content (cid PK, video_id, size, height, vcodec, acodec, duration, meta JSON,
origin 'server'|'intake', status 'verified'|'revoked',
scan 'skipped'|'clean', created_at ms, verified_at ms)
p2p_devices (device_id PK 'dev_<16hex>', secret_hash, fingerprint, profile,
share 0|1, created_at ms, last_seen_at ms)
p2p_holders (cid, device_id, status 'active'|'removed', trust 'reported'|'challenged',
first_reported_at ms, last_verified_at ms, removed_at ms NULL,
PRIMARY KEY (cid, device_id))
video_views (video_id, day 'YYYY-MM-DD', n, PRIMARY KEY (video_id, day))
media_cache.sha256 -- new column: cid of the current <id>.<gen>.mp4
```
"User list" = the existing `users` (fingerprints) and `profiles` tables plus
`p2p_devices`. Nothing is ever deleted from `p2p_content`; a bad file is `revoked`.
## Configuration (`server/p2p-config.js`, plan 008)
| Env | Default | Meaning |
|-----|---------|---------|
| `P2P_ENABLED` | `1` (on) | `0` turns off every P2P route, the hub and client features. |
| `P2P_MALWARE_SCAN` | `0` (off) | `1` runs `P2P_SCAN_CMD <file>` before admission; exit 0 = clean, 1 = infected (rejected), other = error (not admitted, retried later). |
| `P2P_SCAN_CMD` | `clamscan --no-summary --infected` | Needs an image built with `--build-arg INSTALL_CLAMAV=1`. |
| `P2P_STALE_DAYS` | `7` | Holder older than this is shown as stale. |
| `P2P_KEEP_MIN_VIEWS` | `3` | Retention: views in the last `P2P_KEEP_DAYS` that make a server copy "top". |
| `P2P_KEEP_DAYS` | `30` | Window for counting views. |
| `P2P_KEEP_RECENT_DAYS` | `14` | Retention: played this recently = "recent". |
| `P2P_INTAKE_DIR` | `<DB dir>/p2p-intake` | Quarantine for device uploads. Never served. |
| `P2P_INTAKE_MAX_BYTES` | `3 GiB` | Largest accepted intake upload. |
Client settings (`data.settings`, per profile): `p2pShare: true` (let other devices
download my saved videos, and report holdings), `p2pReceive: true` (fetch from other
devices when YouTube and the server can't serve).
## Flows
1. **Server fetch (existing media cache) → verified content** (plan 009).
`runFetch` / `runOptimize` hash the promoted file, store `media_cache.sha256`, run the
scan if enabled, then upsert `p2p_content` (`origin 'server'`). `/api/download` sends
`X-Content-SHA256`. A backfill hashes already-cached files at boot, one at a time.
2. **Device save** (plan 012). The OPFS worker hashes while it writes. If the server
sent `X-Content-SHA256` and the hash differs, the save fails (bonus integrity check).
The device DB (`IndexedDB ytp-device`, store `files`) records `{videoId, cid, size,
savedAt, lastCheckedAt, state}`; `state` is `verified` when the hashes matched,
`unverified` when the server sent no hash, `unhashed` for old saves and the
main-thread fallback path.
3. **Holdings sync** (plan 013). Device registers once (`POST /api/p2p/device` →
`deviceId` + `secret`, kept in `localStorage.ytpDevice`). It reports its holdings
(`POST /api/p2p/holdings`, full list at launch, deltas after save/delete). The server
accepts only cids in `p2p_content` with `status='verified'`; unknown cids come back
in `unknown` (candidates for intake). When the server still has the file it returns
up to 5 **range challenges**; a correct answer sets `trust='challenged'`, a wrong one
removes the holder. `last_verified_at` = time of the last report where the device
re-checked the file (exists, same size; full re-hash every 30 days). A full report
marks every active holder row of that device that is missing from the list as
`removed`. There is **no TTL**.
4. **Presence** (plan 014). `/ws/p2p` socket per device, authenticated with the device
secret. Online status lives only in memory. The hub also relays WebRTC signalling
between two online devices and carries server → device requests (plan 018).
5. **Availability** (plans 014/015). `GET /api/p2p/holders?v=<videoId>` lists cids and
their holders: opaque peer id (never the fingerprint/profile), `online`,
`lastVerifiedAt`, `stale`, `trust`, plus `serverHas`. The UI shows e.g.
"📡 On 3 devices · 1 online now · last checked 2 d ago", with stale holders greyed.
6. **Peer download** (plan 017). WebRTC data channel (STUN only, same ICE list as watch
party), 64 KiB frames with `bufferedAmount` back-pressure, receiver writes through a
worker into OPFS while hashing; only a matching SHA-256 is committed. The new copy
is a holder at the next report. Download-then-play; no progressive peer streaming.
Used when `/api/streams` fails and the device has no copy, and from a
"Get from a device" button.
7. **Intake** (plan 016). A device can hand a file to the server
(`POST /api/p2p/intake` → ticket, `PUT` the bytes). The server writes it to the
quarantine dir, hashes it, runs `validateMedia()`, runs the scan if enabled, and only
then inserts `p2p_content` (`origin 'intake'`). If the server has no copy of that
video it adopts the file into the media cache (budget permitting).
8. **Rehydrate** (plan 018). When a video is requested, its source fails, the server
evicted its copy, and a verified holder is online with `p2pShare` on, the hub asks
that device to upload it through intake (known cid → quick accept).
9. **Retention** (plan 010). Views are counted per video per day. When the media cache
needs room it evicts in this order: copies that are neither "top"
(`views in P2P_KEEP_DAYS ≥ P2P_KEEP_MIN_VIEWS`) nor "recent" (played within
`P2P_KEEP_RECENT_DAYS`), fewest views first, then oldest; only then the qualifying
ones by LRU. The 10-minute play protection and `MEDIA_CACHE_MAX_BYTES` stay.
Evicting a server copy never deletes `p2p_content` or holder rows.
## Security rules every plan must keep
- No cid enters `p2p_content` unless the SERVER computed it over bytes it holds and
`validateMedia` passed. Clients can never insert or edit content rows, views or trust.
- Intake files live in `P2P_INTAKE_DIR`, never under `./public` or `MEDIA_DIR`, and
are deleted on failure.
- Device secrets: 32 random bytes, only `sha256(secret)` stored, compared with
`timingSafeEqual`.
- Holder lists never expose fingerprints, profile names or IPs; a peer id is
`sha256('peer:' + device_id).slice(0, 12)`.
- The hub relays signalling only between two authenticated, online devices, with a
per-socket message budget.
- `P2P_ENABLED=0` must leave the rest of the app working exactly as before.
- Jobs stay server-owned; never pass a request `AbortSignal` into them (CLAUDE.md).
## Where this differs from the earlier phase 02–06 draft
| Earlier draft | Now |
|---------------|-----|
| "Default-off" P2P subsystem; "no inventory/upload from default settings" | P2P on by default; devices report holdings and seed by default (can be turned off). |
| Mandatory scanner, "scan skip is failure" | Scanner off by default (`P2P_MALWARE_SCAN=0`); hash + `validateMedia` mandatory. |
| Short-lived online leases; "cache availability only as an expiring hint" | Persistent holder rows with `last_verified_at` and a stale marker; online status is separate. |
| Migrate all localStorage (`_ytpdata`) to IndexedDB first | Not now: the device DB holds files + cids only. `_ytpdata` stays in localStorage (lower risk). |
| Collections, invitations, scoped principals, signed manifests, TURN, renditions lineage | Deferred. Scope is one shared catalog + device secrets; add later if needed. |
| Separate `server/p2p/*` directory with migrations ledger | Flat files `server/p2p-*.js` matching the repo's style (`party.js`, `remote.js`). |

29
plans/INDEX.md Normal file
View File

@@ -0,0 +1,29 @@
# Plan Queue Index
Speed first (001–007, independent of P2P), then peer-to-peer sharing (008–019,
see `docs/p2p-architecture.md`). Run in order; each plan's `depends_on` lists what
must be in `done/` first. Every plan was dry-run end to end on 2026-09-29 (all 19
applied in order to a clean checkout: 13 server test files and 61 frontend tests
green, app boots with no JS errors, P2P on by default, offline boot works).
| Seq | ID | Title | Status | Commit | Notes |
|-----|----|-------|--------|--------|-------|
| 001 | 001-perf-timing-marks-105acc | Add startup, search and play timing marks plus yt-dlp duration logs | queued | | baseline numbers for the rest |
| 002 | 002-compress-and-etag-shell-bd459c | Serve the app shell gzip/brotli-compressed with ETags | queued | | app.js 426 KB → 94 KB (br) |
| 003 | 003-self-host-fonts-89466b | Self-host the three web fonts and drop the render-blocking Google Fonts CSS | queued | | needs network once; CSP font-src fix |
| 004 | 004-coalesce-stream-resolves-a92d40 | Coalesce concurrent resolveStreams calls for the same video | queued | | |
| 005 | 005-warm-streams-on-intent-47b3d3 | Warm the stream cache for likely next plays | queued | | cold play 7.5 s → cached 1.2 s |
| 006 | 006-innertube-search-48066b | Answer searches from YouTube InnerTube directly with yt-dlp fallback | queued | | search 4–5 s → ~0.8 s |
| 007 | 007-ytdlp-worker-045800 | Keep one long-lived yt-dlp worker process instead of spawning per call | queued | | ~1 s per yt-dlp call |
| 008 | 008-p2p-schema-and-config-127966 | Add P2P tables, config flags and db helpers | queued | | P2P ON, malware scan OFF by default |
| 009 | 009-server-content-hash-186e7f | Hash every validated server copy and register it as verified content | queued | | uses plans/patches/009-* |
| 010 | 010-views-and-retention-d0c6ca | Count views and evict server copies by retention criteria before LRU | queued | | |
| 011 | 011-browser-sha256-e1793d | Add an incremental SHA-256 library for the browser and node tests | queued | | |
| 012 | 012-device-file-registry-288d55 | Add the on-device IndexedDB file registry and hash saves while downloading | queued | | browser harness |
| 013 | 013-device-identity-and-holdings-3ba493 | Register devices and report verified holdings to the server | queued | | persistent holders, no TTL |
| 014 | 014-p2p-presence-hub-ceced8 | Add the /ws/p2p presence and signalling hub and the holders endpoint | queued | | stale flag, never hidden |
| 015 | 015-availability-ui-and-settings-3b9397 | Show peer availability with stale markers and add Sharing settings | queued | | |
| 016 | 016-intake-and-server-verification-cfe031 | Let a device hand a file to the server for hashing and validation | queued | | needs ffmpeg for tests |
| 017 | 017-peer-transfer-1faaa7 | Download a verified file from another device over WebRTC | queued | | STUN only |
| 018 | 018-server-rehydrate-from-peer-4fb8bd | Restore an evicted server copy from an online holder | queued | | |
| 019 | 019-admin-p2p-panel-4dc623 | Add a P2P panel to the admin page | queued | | |

22
plans/README.md Normal file
View File

@@ -0,0 +1,22 @@
# plans/
Executable plan queue (skills `plan-queue` / `run-queue`). Each file in `queue/` is
self-contained: exact files, anchors, code and copy-paste verification.
- `INDEX.md` — the order and status of every plan.
- `queue/ active/ done/ failed/` — plan lifecycle.
- `patches/` — unified diffs referenced by plans (`git apply plans/patches/<name>.diff`).
They were produced from code that passed its tests, and are meant to be applied in plan
order (e.g. `016-media-cache-adopt.diff` expects `009` and `010` applied). If a patch
does not apply, stop and report — never hand-edit around it.
- `harness/` — small throwaway servers + Playwright checks that exercise the browser
side of the P2P plans for real (OPFS, IndexedDB, WebSocket, WebRTC). Run the server
from `server/` (`bun ../plans/harness/<x>-server.js`), then `node <x>-check.mjs` from
`plans/harness/` (needs `npm i --no-save playwright` and a Chromium; the cloud image has
one at `/opt/pw-browsers/chromium-1194/chrome-linux/chrome`, override with `CHROMIUM=`).
Environment notes: server tests need `cd server && bun install`; media / intake tests need
`ffmpeg` + `ffprobe`; plan 003 needs network access to Google Fonts once; plan 007's
pool test needs `bin/yt-dlp` (`npm run setup`) or skips.
Stop a background server by PID (`… & SRV=$!` … `kill $SRV`) — `pkill -f <name>` also
matches the shell running the command and kills it.

0
plans/active/.gitkeep Normal file
View File

0
plans/done/.gitkeep Normal file
View File

0
plans/failed/.gitkeep Normal file
View File

22
plans/harness/device-db-check.mjs Normal file
View File

@@ -0,0 +1,22 @@
// Plan 012 harness check. Needs playwright (npm i --no-save playwright) and a
// Chromium: CHROMIUM=/path/to/chrome (default: the cloud image's /opt/pw-browsers one).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
p.on('console', (m) => console.log('console:', m.text()));
await p.goto('http://localhost:8766/');
const out = await p.evaluate(async () => {
const sha = await (await fetch('/sha')).text();
const good = await OPFS.downloadVideo('good', '/api/download/good');
const bad = await OPFS.downloadVideo('bad', '/api/download/bad');
const nohash = await OPFS.downloadVideo('nohash', '/api/download/nohash');
const list = (await OPFS.listVideos()).map((x) => x.id).sort();
await DeviceDB.putFile({ videoId: 'good', cid: good.sha256, size: good.size, state: 'verified' });
const rec = await DeviceDB.getFile('good');
const byCid = await DeviceDB.getByCid(good.sha256);
await DeviceDB.deleteFile('good');
const after = await DeviceDB.listFiles();
return { shaOk: good.sha256 === sha, goodExpected: good.expectedSha === sha, bad, nohashSha: nohash.sha256 === sha, nohashExpected: nohash.expectedSha, list, rec: rec && rec.state, byCid: byCid && byCid.videoId, after: after.length };
});
console.log(JSON.stringify(out));
await b.close();

17
plans/harness/device-db-server.js Normal file
View File

@@ -0,0 +1,17 @@
// Plan 012 harness: serves ../../frontend plus fake /api/download/<id> responses
// (good hash / wrong hash / no hash). Run: bun device-db-server.js
import { createHash } from 'node:crypto';
const FE = new URL('../../frontend/', import.meta.url).pathname;
const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256);
const sha = createHash('sha256').update(body).digest('hex');
Bun.serve({ port: 8766, fetch(req) {
const u = new URL(req.url);
if (u.pathname === '/api/download/good') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } });
if (u.pathname === '/api/download/bad') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': 'f'.repeat(64) } });
if (u.pathname === '/api/download/nohash') return new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length) } });
if (u.pathname === '/sha') return new Response(sha);
if (u.pathname === '/') return new Response('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script>', { headers: { 'Content-Type': 'text/html' } });
const f = Bun.file(FE + u.pathname.slice(1));
return new Response(f, { headers: { 'Content-Type': 'text/javascript' } });
} });
console.log('ready');

19
plans/harness/intake-check.mjs Normal file
View File

@@ -0,0 +1,19 @@
// Plan 016 harness check: an unknown saved file → contribute → accepted.
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
await p.goto('http://localhost:8769/');
const out = await p.evaluate(async () => {
const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer());
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('upAAAAAAAA9.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
P2PClient.start({ getSettings: () => ({}) });
const first = await P2PClient.sync();
const unknown = await P2PClient.unknownVideos();
const c = await P2PClient.contribute(unknown[0]);
const second = await P2PClient.sync();
return { firstUnknown: first.unknown.length, unknown, contribute: { ok: c.ok, adopted: c.adopted, cidOk: /^[0-9a-f]{64}$/.test(c.cid) }, secondAccepted: second.accepted.length, secondUnknown: second.unknown.length };
});
console.log(JSON.stringify(out));
await b.close();

29
plans/harness/intake-server.js Normal file
View File

@@ -0,0 +1,29 @@
// Plan 016 harness: routes + intake with real validateMedia (needs ffmpeg).
// Run from server/: bun ../plans/harness/intake-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, readFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h16-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js');
const { admitFile } = await import(process.cwd() + '/p2p-admit.js');
const { validateMedia } = await import(process.cwd() + '/media-cache.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const fx = join(root, 'fx.mp4');
spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]);
const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 };
const app = new Hono();
const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile });
app.get('/fixture.mp4', () => new Response(readFileSync(fx)));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8769, fetch: app.fetch });
console.log('ready');

26
plans/harness/p2p-client-check.mjs Normal file
View File

@@ -0,0 +1,26 @@
// Plan 013 harness check (see device-db-check.mjs for requirements).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
p.on('console', (m) => { if (m.type() === 'error') console.log('console:', m.text()); });
await p.goto('http://localhost:8767/');
const out = await p.evaluate(async () => {
const good = await OPFS.downloadVideo('goodAAAAAAA', '/api/download/goodAAAAAAA');
// A legacy save: in OPFS but no DeviceDB record, unknown to the server.
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('legacyAAAAA.mp4', { create: true })).createWritable();
await w.write(new Uint8Array(1000).fill(9)); await w.close();
await DeviceDB.putFile({ videoId: 'goodAAAAAAA', cid: good.sha256, size: good.size, state: 'unverified' });
await DeviceDB.putFile({ videoId: 'ghostAAAAAA', cid: 'e'.repeat(64), size: 5, state: 'verified' }); // no file
P2PClient.start({ getSettings: () => ({}) });
const res = await P2PClient.sync();
const recs = (await DeviceDB.listFiles()).map((r) => [r.videoId, r.state, !!r.cid]).sort();
const holders = await (await fetch('/debug/holders')).json();
// Turn sharing off → full withdrawal.
P2PClient.start({ getSettings: () => ({ p2pShare: false }) });
await P2PClient.sync();
const after = await (await fetch('/debug/holders')).json();
return { accepted: res && res.accepted.length, unknown: res && res.unknown.length, challenges: res && res.challenges.length, recs, trust: holders.map((h) => h.trust), afterShareOff: after.length, device: !!P2PClient.device() };
});
console.log(JSON.stringify(out));
await b.close();

27
plans/harness/p2p-client-server.js Normal file
View File

@@ -0,0 +1,27 @@
// Plan 013 harness: real p2p-routes + temp DB + the frontend files.
// Run from server/: bun ../plans/harness/p2p-client-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
const root = mkdtempSync(join(tmpdir(), 'ytp-h13-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { sha256Range } = await import(process.cwd() + '/hash.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const body = new Uint8Array(3 * 1024 * 1024 + 123).map((_, i) => (i * 7) % 256);
const sha = createHash('sha256').update(body).digest('hex');
const file = join(root, 'good.mp4'); writeFileSync(file, body);
await p2pDb.upsertContent({ cid: sha, videoId: 'goodAAAAAAA', size: body.length, origin: 'server', now: Date.now() });
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async (c) => (c === sha ? { path: file, size: body.length } : null), sha256Range });
app.get('/api/download/:id', () => new Response(body, { headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(body.length), 'X-Content-SHA256': sha } }));
app.get('/debug/holders', async (c) => c.json(await p2pDb.listHolders(sha)));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8767, fetch: app.fetch });
console.log('ready');

21
plans/harness/presence-check.mjs Normal file
View File

@@ -0,0 +1,21 @@
// Plan 014 harness check: two browsers, one signal (see device-db-check.mjs for requirements).
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const pages = [];
for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); await p.goto('http://localhost:8768/'); pages.push(p); }
const peers = [];
for (const p of pages) {
peers.push(await p.evaluate(async () => {
P2PClient.start({ getSettings: () => ({}) });
await P2PClient.sync();
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
window.__got = [];
P2PClient.onMessage('signal', (m) => window.__got.push(m));
return P2PClient.peer();
}));
}
const sent = await pages[0].evaluate((to) => P2PClient.signal(to, { hi: 1 }), peers[1]);
await new Promise((r) => setTimeout(r, 500));
const got = await pages[1].evaluate(() => window.__got);
console.log(JSON.stringify({ peers: peers.map((x) => typeof x === 'string' && x.length === 12), distinct: peers[0] !== peers[1], sent, got: got.map((m) => ({ from: m.from === peers[0], data: m.data })) }));
await b.close();

24
plans/harness/presence-server.js Normal file
View File

@@ -0,0 +1,24 @@
// Plan 014 harness: p2p-routes + p2p-hub on a real Bun server (websocket).
// Run from server/: bun ../plans/harness/presence-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h14-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8768, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

21
plans/harness/rehydrate-check.mjs Normal file
View File

@@ -0,0 +1,21 @@
// Plan 018 harness check: a holder is asked to restore an evicted copy.
import { chromium } from 'playwright';
const b = await chromium.launch({ executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome' });
const p = await b.newPage();
await p.goto('http://localhost:8771/');
const out = await p.evaluate(async () => {
const bytes = new Uint8Array(await (await fetch('/fixture.mp4')).arrayBuffer());
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('goneAAAAAAA.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
P2PClient.start({ getSettings: () => ({}) });
const rep = await P2PClient.sync(); // hashes the file, reports it → accepted (cid is verified content)
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
const s1 = await (await fetch('/api/streams?v=goneAAAAAAA')).json();
let dbg;
for (let i = 0; i < 50; i++) { dbg = await (await fetch('/debug')).json(); if (dbg.adopted.length) break; await new Promise((r) => setTimeout(r, 200)); }
const s2 = await (await fetch('/api/streams?v=goneAAAAAAA')).json();
return { accepted: rep.accepted.length, first: s1, adopted: dbg.adopted.map((a) => [a.videoId, a.sha256 === dbg.cid]), second: s2 };
});
console.log(JSON.stringify(out));
await b.close();

49
plans/harness/rehydrate-server.js Normal file
View File

@@ -0,0 +1,49 @@
// Plan 018 harness: hub + intake + rehydrator + a stand-in /api/streams whose
// source always fails and whose server copy is gone. Port 8771.
// Run from server/: bun ../plans/harness/rehydrate-server.js (needs ffmpeg)
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync, readFileSync, unlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
const root = mkdtempSync(join(tmpdir(), 'ytp-h18-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { registerIntakeRoutes } = await import(process.cwd() + '/p2p-intake.js');
const { createP2pHub, createRehydrator } = await import(process.cwd() + '/p2p-hub.js');
const { admitFile } = await import(process.cwd() + '/p2p-admit.js');
const { validateMedia } = await import(process.cwd() + '/media-cache.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const fx = join(root, 'fx.mp4');
spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', fx]);
const bytes = readFileSync(fx);
const cid = createHash('sha256').update(bytes).digest('hex');
await p2pDb.upsertContent({ cid, videoId: 'goneAAAAAAA', size: bytes.length, origin: 'server', now: Date.now() }); // verified earlier, file since evicted
const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir: join(root, 'intake'), intakeMaxBytes: 50e6 };
const adopted = [];
const app = new Hono();
const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
registerIntakeRoutes(app, { cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile,
serverHasCid: async () => adopted.length > 0,
adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256 }); unlinkSync(path); return { adopted: true }; } });
const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => adopted.length > 0 });
app.get('/api/streams', async (c) => {
const v = c.req.query('v');
if (await rehydrate(v)) return c.json({ ok: false, restoring: true, error: 'restoring from a device' }, 503);
return c.json({ ok: false, error: 'source unavailable' }, 500);
});
app.get('/debug', (c) => c.json({ adopted, cid }));
app.get('/fixture.mp4', () => new Response(bytes));
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8771, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

52
plans/harness/transfer-check.mjs Normal file
View File

@@ -0,0 +1,52 @@
// Plan 017 harness check: device A holds a file; device B downloads it over WebRTC.
import { chromium } from 'playwright';
const b = await chromium.launch({
executablePath: process.env.CHROMIUM || '/opt/pw-browsers/chromium-1194/chrome-linux/chrome',
args: ['--disable-features=WebRtcHideLocalIpsWithMdns'],
});
const pages = [];
for (let i = 0; i < 2; i++) { const ctx = await b.newContext(); const p = await ctx.newPage(); p.on('console', (m) => { if (m.type() === 'error') console.log('console', i, m.text()); }); await p.goto('http://localhost:8770/'); pages.push(p); }
const setup = (p) => p.evaluate(async () => {
P2PClient.start({ getSettings: () => ({}) });
await P2PClient.sync();
for (let i = 0; i < 50 && !P2PClient.isConnected(); i++) await new Promise((r) => setTimeout(r, 100));
P2PTransfer.start({ iceServers: [] });
return P2PClient.peer();
});
// Holder: write a 5 MB file + registry record.
const held = await pages[0].evaluate(async () => {
const bytes = new Uint8Array(5 * 1024 * 1024 + 777).map((_, i) => (i * 31 + 7) % 256);
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos', { create: true });
const w = await (await dir.getFileHandle('heldAAAAAAA.mp4', { create: true })).createWritable();
await w.write(bytes); await w.close();
const cid = Sha256.hex(bytes);
await DeviceDB.putFile({ videoId: 'heldAAAAAAA', cid, size: bytes.length, state: 'verified' });
return { cid, size: bytes.length };
});
const peerA = await setup(pages[0]);
await setup(pages[1]);
const t0 = Date.now();
const res = await pages[1].evaluate(async ({ peerA, held }) => {
const good = await P2PTransfer.download({ videoId: 'gotAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] });
const list = (await OPFS.listVideos()).map((x) => [x.id, x.size]);
const bad = await P2PTransfer.download({ videoId: 'badAAAAAAAA', cid: 'f'.repeat(64), size: held.size, peers: [{ peer: peerA }] });
const list2 = (await OPFS.listVideos()).map((x) => x.id);
return { good, list, bad, list2 };
}, { peerA, held });
// Holder's file gets corrupted (same size, one byte flipped) but its record still claims the cid.
await pages[0].evaluate(async () => {
const dir = await (await navigator.storage.getDirectory()).getDirectoryHandle('videos');
const fh = await dir.getFileHandle('heldAAAAAAA.mp4');
const bytes = new Uint8Array(await (await fh.getFile()).arrayBuffer());
bytes[12345] ^= 0xff;
const w = await fh.createWritable(); await w.write(bytes); await w.close();
});
res.corrupt = await pages[1].evaluate(async ({ peerA, held }) => {
const r = await P2PTransfer.download({ videoId: 'corAAAAAAAA', cid: held.cid, size: held.size, peers: [{ peer: peerA }] });
await new Promise((x) => setTimeout(x, 300));
const names = [];
for await (const [n] of (await (await navigator.storage.getDirectory()).getDirectoryHandle('videos')).entries()) names.push(n);
return { r, names: names.sort() };
}, { peerA, held });
console.log(JSON.stringify({ ...res, ms: Date.now() - t0 }));
await b.close();

24
plans/harness/transfer-server.js Normal file
View File

@@ -0,0 +1,24 @@
// Plan 017 harness: p2p-routes + p2p-hub + the transfer scripts (port 8770).
// Run from server/: bun ../plans/harness/transfer-server.js
const { Hono } = await import(process.cwd() + '/node_modules/hono/dist/index.js');
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-h14-'));
process.env.DB_PATH = join(root, 'h.db');
const dbmod = await import(process.cwd() + '/db.js');
const p2pDb = await import(process.cwd() + '/p2p-db.js');
const { registerP2pRoutes } = await import(process.cwd() + '/p2p-routes.js');
const { createP2pHub } = await import(process.cwd() + '/p2p-hub.js');
const FE = process.env.FE_DIR || new URL('../../frontend', import.meta.url).pathname;
await dbmod.initDb(); await p2pDb.initP2pSchema();
const app = new Hono();
registerP2pRoutes(app, { cfg: { enabled: true, staleDays: 7 }, p2pDb, fileForCid: async () => null, sha256Range: async () => '' });
const hub = createP2pHub({ getDevice: p2pDb.getDevice });
app.get('/', (c) => c.html('<!doctype html><script src="/opfs.js"></script><script src="/sha256.js"></script><script src="/device-db.js"></script><script src="/p2p-client.js"></script><script src="/p2p-transfer.js"></script>'));
app.get('/:f', (c) => new Response(Bun.file(FE + '/' + c.req.param('f')), { headers: { 'Content-Type': 'text/javascript' } }));
Bun.serve({ port: 8770, fetch(req, server) {
if (new URL(req.url).pathname === '/ws/p2p') return hub.upgrade(req, server);
return app.fetch(req, server);
}, websocket: hub.websocket });
console.log('ready');

63
plans/patches/009-media-cache-test.diff Normal file
View File

@@ -0,0 +1,63 @@
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -10,6 +10,8 @@
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const { createMediaCache, validateMedia, MediaSkip, HIGH, LOW, OPT_REV } = await import('./media-cache.js');
+const { initP2pSchema } = await import('./p2p-db.js');
+const { sha256File } = await import('./hash.js');
const fx = (name) => join(root, name);
const ff = (...args) => {
@@ -26,6 +28,7 @@
beforeAll(async () => {
await dbmod.initDb();
+ await initP2pSchema(); // adds media_cache.sha256
// 10 s 640x360 H.264 + AAC — the shape of a good cached copy.
ff('-f', 'lavfi', '-i', 'testsrc=size=640x360:rate=30:duration=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=10',
'-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', fx('good.mp4'));
@@ -102,6 +105,7 @@
download: async (id, out) => { calls.push(id); await sleep(30); copyFileSync(fx(state.fixture), out); return out; },
transcode: { enabled: false },
freeBytes: () => 100 * 1024 ** 3,
+ backfillDelayMs: -1,
...opts,
});
return { cache, dir, calls, state };
@@ -110,6 +114,35 @@
for (const r of await dbmod.listMedia()) await dbmod.deleteMedia(r.video_id);
}
+describe('content hashes (P2P cids)', () => {
+ test('a promoted copy stores the sha256 of its mp4 and reports it once', async () => {
+ await clearDb();
+ const seen = [];
+ const { cache, dir } = makeCache({ onReady: (i) => { seen.push(i); } });
+ await cache.init();
+ const row = await cache.ensureCached('hashAAAAAA1', { priority: HIGH });
+ const want = await sha256File(join(dir, `hashAAAAAA1.${row.gen}.mp4`));
+ expect((await dbmod.getMedia('hashAAAAAA1')).sha256).toBe(want);
+ expect(await until(() => seen.length === 1)).toBe(true);
+ expect(seen[0]).toMatchObject({ id: 'hashAAAAAA1', gen: row.gen, sha256: want, vcodec: 'h264' });
+ expect(seen[0].meta.title).toBe('T hashAAAAAA1');
+ });
+
+ test('backfill hashes ready copies that have no sha256 yet', async () => {
+ await clearDb();
+ const seen = [];
+ const { cache } = makeCache({ onReady: (i) => { seen.push(i); } });
+ await cache.init();
+ await cache.ensureCached('hashAAAAAA2', { priority: HIGH });
+ await dbmod.upsertMedia('hashAAAAAA2', { sha256: null });
+ seen.length = 0;
+ expect(await cache.backfillHashes()).toBe(1);
+ expect((await dbmod.getMedia('hashAAAAAA2')).sha256).toMatch(/^[0-9a-f]{64}$/);
+ expect(await until(() => seen.length === 1)).toBe(true);
+ expect(await cache.backfillHashes()).toBe(0); // nothing left
+ });
+});
+
describe('media cache jobs', () => {
test('fetches once, dedupes concurrent requests, then serves from disk', async () => {
await clearDb();

120
plans/patches/009-media-cache.diff Normal file
View File

@@ -0,0 +1,120 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -24,6 +24,7 @@
statSync, statfsSync, unlinkSync,
} from 'node:fs';
import { join } from 'node:path';
+import { sha256File } from './hash.js';
export const HIGH = 0; // explicit save / Broken re-download
export const LOW = 1; // auto-cache after a play
@@ -167,6 +168,9 @@
freeBytes = () => { const s = statfsSync(dir); return s.bavail * s.bsize; },
now = () => Date.now(),
log = console,
+ hashFile = sha256File, // server-computed SHA-256 = the file's P2P content id
+ onReady = null, // ({ id, gen, path, sha256, size, … }) after a validated copy lands
+ backfillDelayMs = 30_000, // hash pre-existing copies this long after init()
} = {}) {
const tmpDir = join(dir, '.tmp');
const fileFor = (id, gen, kind = 'mp4') => join(dir, `${id}.${gen}.${kind}`);
@@ -236,6 +240,20 @@
}
}
+ // Tell the P2P layer a validated file with a server-computed hash exists.
+ // Never throws and never delays the caller.
+ function notifyReady(id, gen, sha256, probe, metaJson) {
+ if (!onReady || !sha256) return;
+ let meta = {};
+ try { meta = JSON.parse(metaJson || '{}'); } catch { /* keep {} */ }
+ Promise.resolve()
+ .then(() => onReady({
+ id, gen, path: fileFor(id, gen), sha256, size: probe.size, height: probe.height,
+ vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta,
+ }))
+ .catch((e) => log.warn?.(`[media] onReady ${id} failed: ${e.message}`));
+ }
+
async function setStatus(job, status) {
job.status = status;
await db.upsertMedia(job.id, { status, updated_at: now() });
@@ -308,6 +326,7 @@
'-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]);
if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr));
+ const sha256 = await hashFile(norm);
const prev = await db.getMedia(id);
const gen = ((prev && prev.gen) || 0) + 1;
renameSync(norm, fileFor(id, gen));
@@ -318,13 +337,14 @@
size: probe.size + sizeOf(fileFor(id, gen, 'm4a')),
height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration,
optimized: 0, meta: JSON.stringify(metaFromInfo(info, probe.duration)),
- attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t,
+ attempts: 0, error: null, retry_at: 0, updated_at: t, last_access: t, sha256,
};
await db.upsertMedia(id, fields);
removeFiles(id, gen);
job.status = 'ready';
log.info?.(`[media] cached ${id} ${probe.height}p ${fmtMB(fields.size)}`);
enqueueOptimize(id);
+ notifyReady(id, gen, sha256, probe, fields.meta);
return { ...(prev || {}), video_id: id, ...fields };
} catch (err) {
job.status = 'failed';
@@ -427,6 +447,7 @@
log.info?.(`[media] kept original ${id}: ${probe.vcodec} ${fmtMB(probe.size)} vs ${fmtMB(oldSize)}`);
return;
}
+ const sha256 = await hashFile(out);
const gen = row.gen + 1;
renameSync(out, fileFor(id, gen));
const oldAudio = fileFor(id, row.gen, 'm4a');
@@ -434,8 +455,9 @@
try { linkSync(oldAudio, newAudio); } catch { copyFileSync(oldAudio, newAudio); }
await db.upsertMedia(id, {
gen, size: probe.size + sizeOf(newAudio), height: probe.height, vcodec: probe.vcodec,
- duration: probe.duration, optimized: OPT_REV, updated_at: now(),
+ duration: probe.duration, optimized: OPT_REV, updated_at: now(), sha256,
});
+ notifyReady(id, gen, sha256, probe, cur.meta);
removeFiles(id, gen, OLD_GEN_GRACE_MS);
log.info?.(`[media] optimized ${id}: ${fmtMB(oldSize)} → ${fmtMB(probe.size)} (${probe.vcodec})`);
} catch (err) {
@@ -622,7 +644,34 @@
}
const s = await db.mediaStats();
log.info?.(`[media] ${s.count} cached (${fmtMB(s.bytes)}), ${resumed} job(s) resumed, dir ${dir}`);
+ if (backfillDelayMs >= 0) {
+ const t = setTimeout(() => backfillHashes().catch(() => {}), backfillDelayMs);
+ t.unref?.();
+ }
+ }
+
+ // Copies cached before hashing existed get their sha256 in the background,
+ // one at a time, so their content ids can be admitted too.
+ async function backfillHashes() {
+ let n = 0;
+ for (const r of await db.listMedia()) {
+ if (r.status !== 'ready' || r.sha256) continue;
+ const p = fileFor(r.video_id, r.gen);
+ if (!existsSync(p)) continue;
+ try {
+ const sha256 = await hashFile(p);
+ const cur = await db.getMedia(r.video_id);
+ if (!cur || cur.status !== 'ready' || cur.gen !== r.gen) continue;
+ await db.upsertMedia(r.video_id, { sha256 });
+ notifyReady(r.video_id, r.gen, sha256,
+ { size: sizeOf(p), height: r.height, vcodec: r.vcodec, acodec: r.acodec, duration: r.duration }, r.meta);
+ n++;
+ } catch (e) {
+ log.warn?.(`[media] hash backfill ${r.video_id}: ${e.message}`);
+ }
+ }
+ return n;
}
- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId };
+ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes };
}

44
plans/patches/010-retention-eviction.diff Normal file
View File

@@ -0,0 +1,44 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -226,7 +226,10 @@
let { bytes } = await db.mediaStats();
if (bytes + est > maxBytes) {
const cutoff = now() - EVICT_PROTECT_MS;
- for (const r of await db.listMediaLru()) {
+ // Retention order (plan 010) when the server provides it: copies that
+ // are neither top nor recent go first. Otherwise plain LRU.
+ const order = db.listMediaEvictionOrder ? await db.listMediaEvictionOrder() : await db.listMediaLru();
+ for (const r of order) {
if (bytes + est <= maxBytes) break;
if (r.video_id === id || r.last_access > cutoff || jobs.has(r.video_id) || optActive === r.video_id) continue;
await evict(r.video_id, 'budget');
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -240,6 +240,27 @@
expect(await cache.getReady('lruAAAAAAA3')).not.toBeNull();
});
+ test('eviction follows listMediaEvictionOrder when the db provides it', async () => {
+ await clearDb();
+ const size = readFileSync(fx('good.mp4')).length;
+ const db = {
+ getMedia: dbmod.getMedia, upsertMedia: dbmod.upsertMedia, deleteMedia: dbmod.deleteMedia,
+ listMedia: dbmod.listMedia, listMediaLru: dbmod.listMediaLru, touchMedia: dbmod.touchMedia,
+ mediaStats: dbmod.mediaStats,
+ // Retention says the MORE recently played #2 is the one to drop (e.g. no views).
+ listMediaEvictionOrder: async () => (await dbmod.listMediaLru()).reverse(),
+ };
+ const { cache } = makeCache({ db, maxBytes: size * 2 + 60 * 250 * 1024 + 50_000, duration: 10 });
+ await cache.init();
+ await cache.ensureCached('retAAAAAAA1', { priority: HIGH });
+ await cache.ensureCached('retAAAAAAA2', { priority: HIGH });
+ await dbmod.upsertMedia('retAAAAAAA1', { last_access: 1000 });
+ await dbmod.upsertMedia('retAAAAAAA2', { last_access: 2000 });
+ await cache.ensureCached('retAAAAAAA3', { priority: HIGH });
+ expect(await dbmod.getMedia('retAAAAAAA2')).toBeNull();
+ expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull();
+ });
+
test('free-disk guard skips caching', async () => {
await clearDb();
const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 });

75
plans/patches/012-opfs-hash.diff Normal file
View File

@@ -0,0 +1,75 @@
--- a/frontend/opfs-worker.js
+++ b/frontend/opfs-worker.js
@@ -13,12 +13,20 @@
* Out messages:
* { type: 'unsupported' } → caller falls back to main thread
* { type: 'progress', received } → bytes written so far
- * { type: 'done', ext } → file stored as <videoId>.<ext>
+ * { type: 'done', ext, sha256, expectedSha, size }
+ * → file stored as <videoId>.<ext>;
+ * sha256 = hash of the stored bytes
+ * (P2P content id), expectedSha = the
+ * server's X-Content-SHA256 or null
* { type: 'error', error } → failed; .part cleaned up
* ========================================================================== */
'use strict';
+// Incremental SHA-256 (frontend/sha256.js): the file is hashed while it is
+// written, so the device knows its content id without re-reading the file.
+try { importScripts('/sha256.js'); } catch { /* hashing unavailable — save still works */ }
+
async function getVideosDir() {
const root = await navigator.storage.getDirectory();
return root.getDirectoryHandle('videos', { create: true });
@@ -60,6 +68,7 @@
dir = await getVideosDir();
const partHandle = await dir.getFileHandle(partName, { create: true });
const access = await partHandle.createSyncAccessHandle();
+ const hasher = self.Sha256 ? self.Sha256.create() : null;
let offset = 0;
try {
const reader = res.body.getReader();
@@ -67,6 +76,7 @@
const { done, value } = await reader.read();
if (done) break;
access.write(value, { at: offset });
+ if (hasher) hasher.update(value);
offset += value.byteLength;
self.postMessage({ type: 'progress', received: offset });
}
@@ -81,6 +91,14 @@
if (expected > 0 && offset !== expected) {
throw new Error(`download cut short (${offset} of ${expected} bytes)`);
}
+ // The server names the hash of what it sent (media cache copies). A
+ // mismatch means the bytes were damaged on the way — never keep them.
+ const sha256 = hasher ? hasher.hex() : null;
+ const sent = (res.headers.get('x-content-sha256') || '').trim().toLowerCase();
+ const expectedSha = /^[0-9a-f]{64}$/.test(sent) ? sent : null;
+ if (sha256 && expectedSha && sha256 !== expectedSha) {
+ throw new Error('integrity check failed (content hash mismatch)');
+ }
// Finalize: .part → permanent name. Prefer the native rename, but treat
// ANY move() failure as "unavailable" and fall back to a chunked copy —
@@ -111,7 +129,7 @@
await dir.removeEntry(partName);
}
- self.postMessage({ type: 'done', ext });
+ self.postMessage({ type: 'done', ext, sha256, expectedSha, size: offset });
} catch (err) {
// Never leave a corrupt partial behind
try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ }
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -136,7 +136,7 @@
};
worker.onmessage = (e) => {
const m = e.data || {};
- if (m.type === 'done') finish({ ok: true });
+ if (m.type === 'done') finish({ ok: true, sha256: m.sha256 || null, expectedSha: m.expectedSha || null, size: m.size || 0 });
else if (m.type === 'unsupported') finish({ ok: false, fallback: true });
else if (m.type === 'error') finish({ ok: false, error: m.error });
// 'progress' messages are informational; ignored here

22
plans/patches/013-opfs-readrange.diff Normal file
View File

@@ -0,0 +1,22 @@
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -108,6 +108,19 @@
}
},
+ // Bytes [offset, offset+length) of a saved video — answers the server's
+ // P2P range challenges (p2p-client.js). null when the file is missing.
+ async readRange(videoId, offset, length) {
+ try {
+ const found = await findHandle(videoId);
+ if (!found) return null;
+ const file = await found[0].getFile();
+ return new Uint8Array(await file.slice(offset, offset + length).arrayBuffer());
+ } catch {
+ return null;
+ }
+ },
+
revokeUrl(url) {
if (url && _blobUrls.has(url)) {
URL.revokeObjectURL(url);

105
plans/patches/014-p2p-client-presence.diff Normal file
View File

@@ -0,0 +1,105 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -6,6 +6,9 @@
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
+ * onMessage(type, fn) / signal(to, data) / peer()
+ * live /ws/p2p socket (plan 014): the
+ * device is "online" while it is open
*
* What it does, in order, each sync:
* 1. registers the device once (localStorage ytpDevice)
@@ -148,6 +151,74 @@
return j;
}
+ // ---- presence socket (/ws/p2p) — plan 014 ------------------------------------
+ // Open while sharing OR receiving is on. Credentials go in the first message,
+ // never the URL. Reconnects with backoff (5 s … 5 min).
+ let ws = null;
+ let myPeer = null;
+ let wsRetry = 0;
+ let wsTimer = null;
+ const listeners = new Map(); // type -> Set<fn>
+
+ const wantSocket = () => {
+ const s = hooks.getSettings();
+ return s.p2pShare !== false || s.p2pReceive !== false;
+ };
+
+ function onMessage(type, fn) {
+ if (!listeners.has(type)) listeners.set(type, new Set());
+ listeners.get(type).add(fn);
+ return () => listeners.get(type).delete(fn);
+ }
+ function emit(m) {
+ for (const fn of listeners.get(m.type) || []) { try { fn(m); } catch { /* a listener's bug is not ours */ } }
+ }
+
+ function connect() {
+ if (ws || !wantSocket()) return;
+ const d = device();
+ if (!d) return;
+ const proto = location.protocol === 'https:' ? 'wss:' : 'ws:';
+ let sock;
+ try { sock = new WebSocket(`${proto}//${location.host}/ws/p2p`); } catch { return; }
+ ws = sock;
+ let ping = null;
+ sock.onopen = () => {
+ sock.send(JSON.stringify({ type: 'auth', device: d.deviceId, secret: d.secret }));
+ // The server drops sockets idle for 120 s (server.js websocketHandler).
+ ping = setInterval(() => { try { sock.send('{"type":"ping"}'); } catch { /* closing */ } }, 50_000);
+ };
+ sock.onmessage = (e) => {
+ let m;
+ try { m = JSON.parse(e.data); } catch { return; }
+ if (m.type === 'hello') { myPeer = m.peer; wsRetry = 0; }
+ emit(m);
+ };
+ sock.onclose = () => {
+ clearInterval(ping);
+ if (ws !== sock) return;
+ ws = null;
+ myPeer = null;
+ if (!wantSocket()) return;
+ clearTimeout(wsTimer);
+ wsTimer = setTimeout(connect, Math.min(300_000, 5000 * 2 ** wsRetry++));
+ };
+ }
+
+ function disconnect() {
+ clearTimeout(wsTimer);
+ const sock = ws;
+ ws = null;
+ myPeer = null;
+ if (sock) { try { sock.close(); } catch { /* gone */ } }
+ }
+
+ function signal(to, data) {
+ if (!ws || ws.readyState !== 1) return false;
+ ws.send(JSON.stringify({ type: 'signal', to, data }));
+ return true;
+ }
+
async function syncOnce() {
if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null;
if (navigator.onLine === false) return null;
@@ -158,6 +229,7 @@
await hashPending(recs);
const res = await report(recs, dev);
lastSync = Date.now();
+ if (wantSocket()) connect(); else disconnect();
return res;
}
@@ -185,5 +257,8 @@
});
}
- window.P2PClient = { start, changed, sync, device, authHeaders, config };
+ window.P2PClient = {
+ start, changed, sync, device, authHeaders, config,
+ onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer),
+ };
}());

213
plans/patches/014-p2p-hub-new.diff Normal file
View File

@@ -0,0 +1,213 @@
--- /dev/null
+++ b/server/p2p-hub.js
@@ -0,0 +1,120 @@
+/* ============================================================================
+ * p2p-hub.js — live presence + WebRTC signalling for P2P devices, and the
+ * availability payload (docs/p2p-architecture.md flows 4–5).
+ *
+ * /ws/p2p (credentials never go in the URL — proxies log URLs)
+ * you → server {type:'auth', device, secret} first message, within 10 s
+ * server → you {type:'hello', peer} after auth
+ * you → peer {type:'signal', to:<peer>, data} relayed as
+ * server → peer {type:'signal', from:<your peer>, data}
+ * server → you {type:'error', error}
+ * Server-initiated messages (plan 018) go through hub.send(deviceId, msg).
+ *
+ * "Online" lives only in memory: a server restart forgets it, a closed
+ * socket drops it. Holder ROWS are persistent (p2p_holders) — the payload
+ * reports both: `online` (now) and `lastVerifiedAt` / `stale` (history).
+ * Peers are addressed by an opaque id (peerIdOf), never by device id.
+ * ========================================================================== */
+import { createHash, timingSafeEqual } from 'node:crypto';
+
+const sha = (s) => createHash('sha256').update(String(s)).digest('hex');
+const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); };
+export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12);
+
+const MAX_MSG = 64 * 1024;
+const BUDGET_PER_MIN = 300;
+
+export function createP2pHub({ getDevice, enabled = () => true, now = () => Date.now(), log = console } = {}) {
+ const online = new Map(); // deviceId -> ws
+ const byPeer = new Map(); // peer -> deviceId (online only)
+
+ const send = (ws, m) => { try { ws.send(JSON.stringify(m)); } catch { /* gone */ } };
+
+ function upgrade(req, server) {
+ if (!enabled()) return new Response('p2p disabled', { status: 404 });
+ const ok = server.upgrade(req, { data: { hub: 'p2p', authed: false, budget: [] } });
+ return ok ? undefined : new Response('upgrade failed', { status: 400 });
+ }
+
+ function open(ws) {
+ const t = setTimeout(() => { if (!ws.data.authed) { try { ws.close(4401, 'auth timeout'); } catch { /* gone */ } } }, 10_000);
+ t.unref?.();
+ }
+
+ async function auth(ws, m) {
+ const deviceId = String(m.device || '');
+ const secret = String(m.secret || '');
+ const d = /^dev_[0-9a-f]{16}$/.test(deviceId) ? await getDevice(deviceId).catch(() => null) : null;
+ if (!d || !same(d.secret_hash, sha(secret))) { try { ws.close(4401, 'unknown device'); } catch { /* gone */ } return; }
+ ws.data.authed = true;
+ ws.data.deviceId = deviceId;
+ ws.data.peer = peerIdOf(deviceId);
+ const prev = online.get(deviceId);
+ if (prev && prev !== ws) { prev.data.deviceId = null; try { prev.close(4409, 'replaced'); } catch { /* gone */ } }
+ online.set(deviceId, ws);
+ byPeer.set(ws.data.peer, deviceId);
+ send(ws, { type: 'hello', peer: ws.data.peer });
+ }
+
+ async function message(ws, raw) {
+ const t = now();
+ ws.data.budget = ws.data.budget.filter((x) => t - x < 60_000);
+ if (ws.data.budget.length >= BUDGET_PER_MIN) { send(ws, { type: 'error', error: 'slow down' }); return; }
+ ws.data.budget.push(t);
+ const s = typeof raw === 'string' ? raw : Buffer.from(raw).toString('utf8');
+ if (s.length > MAX_MSG) return;
+ let m;
+ try { m = JSON.parse(s); } catch { return; }
+ if (!ws.data.authed) { if (m && m.type === 'auth' && !ws.data.authing) { ws.data.authing = true; await auth(ws, m); } return; }
+ if (m && m.type === 'signal' && typeof m.to === 'string') {
+ const target = online.get(byPeer.get(m.to));
+ if (!target || target === ws) { send(ws, { type: 'error', error: 'peer offline', to: m.to }); return; }
+ send(target, { type: 'signal', from: ws.data.peer, data: m.data });
+ }
+ }
+
+ function close(ws) {
+ const { deviceId, peer } = ws.data || {};
+ if (deviceId && online.get(deviceId) === ws) {
+ online.delete(deviceId);
+ byPeer.delete(peer);
+ }
+ }
+
+ return {
+ upgrade,
+ websocket: { open, message, close },
+ isOnline: (deviceId) => online.has(deviceId),
+ send: (deviceId, msg) => { const ws = online.get(deviceId); if (!ws) return false; send(ws, msg); return true; },
+ onlineCount: () => online.size,
+ };
+}
+
+// GET /api/p2p/holders payload. Only devices that share are listed; holders
+// are never dropped for age — `stale` just flags an old last check.
+export async function holdersPayload({ videoId, cid, p2pDb, isOnline, staleDays, serverHas, now = Date.now() }) {
+ const contents = cid
+ ? [await p2pDb.getContent(cid)].filter((c) => c && c.status === 'verified')
+ : await p2pDb.listContentForVideo(videoId);
+ const staleMs = staleDays * 86400_000;
+ const out = [];
+ for (const c of contents) {
+ const holders = (await p2pDb.listHolders(c.cid, 50))
+ .filter((h) => Number(h.share) === 1)
+ .map((h) => ({
+ peer: peerIdOf(h.device_id),
+ online: isOnline(h.device_id),
+ lastVerifiedAt: Number(h.last_verified_at),
+ stale: now - Number(h.last_verified_at) > staleMs,
+ trust: h.trust,
+ }))
+ .sort((a, b) => (b.online - a.online) || (b.lastVerifiedAt - a.lastVerifiedAt));
+ out.push({
+ cid: c.cid, videoId: c.video_id, size: Number(c.size), height: Number(c.height), vcodec: c.vcodec,
+ serverHas: !!(await serverHas(c.cid)),
+ counts: { holders: holders.length, online: holders.filter((h) => h.online).length, fresh: holders.filter((h) => !h.stale).length },
+ holders,
+ });
+ }
+ return { ok: true, staleDays, cids: out };
+}
--- /dev/null
+++ b/server/p2p-hub.test.js
@@ -0,0 +1,87 @@
+// Presence/signalling hub and the holders payload (plan 014).
+import { test, expect, beforeAll } from 'bun:test';
+import { mkdtempSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createHash } from 'node:crypto';
+
+const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-hub-'));
+process.env.DB_PATH = join(root, 'test.db');
+const dbmod = await import('./db.js');
+const p2pDb = await import('./p2p-db.js');
+const { createP2pHub, holdersPayload, peerIdOf } = await import('./p2p-hub.js');
+
+const sha = (s) => createHash('sha256').update(s).digest('hex');
+const SECRET_A = '1'.repeat(64);
+const SECRET_B = '2'.repeat(64);
+const CID = 'a'.repeat(64);
+const DAY = 86400_000;
+const NOW = Date.UTC(2026, 8, 29, 12);
+
+function fakeWs(data) {
+ return { data, sent: [], closed: null, send(s) { this.sent.push(JSON.parse(s)); }, close(code) { this.closed = code; } };
+}
+
+beforeAll(async () => {
+ await dbmod.initDb();
+ await p2pDb.initP2pSchema();
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000a', secretHash: sha(SECRET_A), now: NOW });
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000b', secretHash: sha(SECRET_B), now: NOW });
+ await p2pDb.createDevice({ deviceId: 'dev_000000000000000c', secretHash: sha('x'), now: NOW });
+ await p2pDb.touchDevice('dev_000000000000000c', { now: NOW, share: false });
+ await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1234, height: 720, vcodec: 'h264', origin: 'server', now: NOW });
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000a', now: NOW - 1 * DAY });
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000b', now: NOW - 30 * DAY }); // stale, still listed
+ await p2pDb.upsertHolder({ cid: CID, deviceId: 'dev_000000000000000c', now: NOW }); // share off → hidden
+});
+
+test('auth on open, hello with opaque peer id, bad secret is closed', async () => {
+ const hub = createP2pHub({ getDevice: p2pDb.getDevice });
+ const a = fakeWs({ budget: [] });
+ hub.websocket.open(a);
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: 'x', data: {} })); // ignored before auth
+ expect(a.sent.length).toBe(0);
+ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A }));
+ expect(a.sent[0]).toEqual({ type: 'hello', peer: peerIdOf('dev_000000000000000a') });
+ expect(hub.isOnline('dev_000000000000000a')).toBe(true);
+ const bad = fakeWs({ budget: [] });
+ hub.websocket.open(bad);
+ await hub.websocket.message(bad, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_A }));
+ expect(bad.closed).toBe(4401);
+ expect(hub.isOnline('dev_000000000000000b')).toBe(false);
+});
+
+test('signals are relayed between online peers only; close drops presence', async () => {
+ const hub = createP2pHub({ getDevice: p2pDb.getDevice });
+ const a = fakeWs({ budget: [] });
+ const b = fakeWs({ budget: [] });
+ hub.websocket.open(a);
+ hub.websocket.open(b);
+ await hub.websocket.message(a, JSON.stringify({ type: 'auth', device: 'dev_000000000000000a', secret: SECRET_A }));
+ await hub.websocket.message(b, JSON.stringify({ type: 'auth', device: 'dev_000000000000000b', secret: SECRET_B }));
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: { sdp: 'x' } }));
+ expect(b.sent.at(-1)).toEqual({ type: 'signal', from: a.data.peer, data: { sdp: 'x' } });
+ hub.websocket.close(b);
+ expect(hub.isOnline('dev_000000000000000b')).toBe(false);
+ await hub.websocket.message(a, JSON.stringify({ type: 'signal', to: b.data.peer, data: {} }));
+ expect(a.sent.at(-1)).toMatchObject({ type: 'error', error: 'peer offline' });
+ expect(hub.send('dev_000000000000000a', { type: 'ping' })).toBe(true);
+ expect(hub.send('dev_000000000000000b', { type: 'ping' })).toBe(false);
+});
+
+test('holders payload: persistent rows, online flag, stale marker, share-off hidden', async () => {
+ const online = new Set(['dev_000000000000000b']);
+ const p = await holdersPayload({
+ videoId: 'dQw4w9WgXcQ', p2pDb, isOnline: (d) => online.has(d), staleDays: 7,
+ serverHas: async () => false, now: NOW,
+ });
+ expect(p.staleDays).toBe(7);
+ expect(p.cids.length).toBe(1);
+ const c = p.cids[0];
+ expect(c).toMatchObject({ cid: CID, size: 1234, height: 720, serverHas: false, counts: { holders: 2, online: 1, fresh: 1 } });
+ expect(c.holders.map((h) => [h.peer, h.online, h.stale])).toEqual([
+ [peerIdOf('dev_000000000000000b'), true, true], // online first even though stale
+ [peerIdOf('dev_000000000000000a'), false, false],
+ ]);
+ expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids
+});

107
plans/patches/015-p2p-core-new.diff Normal file
View File

@@ -0,0 +1,107 @@
--- /dev/null
+++ b/frontend/p2p-core.js
@@ -0,0 +1,55 @@
+/* ============================================================================
+ * p2p-core.js — pure helpers for peer-to-peer UI (window.P2PCore / node)
+ *
+ * formatAvailability(payload, now) turns GET /api/p2p/holders into the line
+ * under the now-playing title, e.g.
+ * "📡 On 3 devices · 1 online now · last checked 2d ago"
+ * Holders are persistent (docs/p2p-architecture.md): they are never hidden
+ * for age; when every holder is older than the server's staleDays the line
+ * says so and is marked stale.
+ * ========================================================================== */
+(function (root) {
+ 'use strict';
+
+ function ago(ms, now) {
+ const s = Math.max(0, Math.round((now - ms) / 1000));
+ if (s < 90) return 'just now';
+ const m = Math.round(s / 60);
+ if (m < 90) return m + 'm ago';
+ const h = Math.round(m / 60);
+ if (h < 48) return h + 'h ago';
+ return Math.round(h / 24) + 'd ago';
+ }
+
+ function formatAvailability(payload, now) {
+ if (!payload || !payload.ok || !Array.isArray(payload.cids)) return null;
+ const holders = [];
+ for (const c of payload.cids) for (const h of c.holders || []) holders.push(h);
+ if (!holders.length) return null;
+ const online = holders.filter((h) => h.online).length;
+ const fresh = holders.filter((h) => !h.stale).length;
+ const newest = Math.max(...holders.map((h) => Number(h.lastVerifiedAt) || 0));
+ const n = holders.length;
+ const allStale = fresh === 0;
+ const text = `📡 On ${n} device${n === 1 ? '' : 's'} · ${online ? online + ' online now' : 'none online'}`
+ + ` · last checked ${ago(newest, now)}` + (allStale ? ' (not checked recently)' : '');
+ const title = holders.map((h) =>
+ `${h.peer} · ${h.online ? 'online' : 'offline'} · checked ${ago(Number(h.lastVerifiedAt) || 0, now)}`
+ + `${h.stale ? ' (stale)' : ''}${h.trust === 'challenged' ? ' · spot-checked' : ''}`).join('\n');
+ return { text, title, stale: allStale, online, holders: n };
+ }
+
+ // Holders worth trying for a download: online first, then freshest check.
+ function pickPeers(payload, cid) {
+ if (!payload || !Array.isArray(payload.cids)) return [];
+ const c = payload.cids.find((x) => x.cid === cid) || payload.cids[0];
+ if (!c) return [];
+ return (c.holders || []).filter((h) => h.online)
+ .sort((a, b) => (b.trust === 'challenged') - (a.trust === 'challenged') || b.lastVerifiedAt - a.lastVerifiedAt)
+ .map((h) => ({ peer: h.peer, cid: c.cid, size: c.size }));
+ }
+
+ const P2PCore = { ago, formatAvailability, pickPeers };
+ if (typeof module !== 'undefined' && module.exports) module.exports = P2PCore;
+ else root.P2PCore = P2PCore;
+})(typeof globalThis !== 'undefined' ? globalThis : this);
--- /dev/null
+++ b/frontend/p2p-core.test.js
@@ -0,0 +1,46 @@
+'use strict';
+const { test } = require('node:test');
+const assert = require('node:assert');
+const C = require('./p2p-core');
+
+const NOW = Date.UTC(2026, 8, 29, 12);
+const H = 3600_000;
+const payload = (holders, extra = {}) => ({ ok: true, staleDays: 7, cids: [{ cid: 'a'.repeat(64), size: 10, holders, ...extra }] });
+
+test('ago buckets', () => {
+ assert.strictEqual(C.ago(NOW - 30_000, NOW), 'just now');
+ assert.strictEqual(C.ago(NOW - 20 * 60_000, NOW), '20m ago');
+ assert.strictEqual(C.ago(NOW - 5 * H, NOW), '5h ago');
+ assert.strictEqual(C.ago(NOW - 72 * H, NOW), '3d ago');
+});
+
+test('no holders → nothing to show', () => {
+ assert.strictEqual(C.formatAvailability(payload([]), NOW), null);
+ assert.strictEqual(C.formatAvailability(null, NOW), null);
+});
+
+test('mixed holders: counts, newest check, not stale', () => {
+ const f = C.formatAvailability(payload([
+ { peer: 'p1', online: true, lastVerifiedAt: NOW - 30 * 24 * H, stale: true, trust: 'reported' },
+ { peer: 'p2', online: false, lastVerifiedAt: NOW - 48 * H, stale: false, trust: 'challenged' },
+ ]), NOW);
+ assert.strictEqual(f.text, '📡 On 2 devices · 1 online now · last checked 2d ago');
+ assert.strictEqual(f.stale, false);
+ assert.match(f.title, /p1 · online · checked 30d ago \(stale\)/);
+ assert.match(f.title, /p2 · offline · checked 2d ago · spot-checked/);
+});
+
+test('every holder stale → still listed, flagged', () => {
+ const f = C.formatAvailability(payload([{ peer: 'p1', online: false, lastVerifiedAt: NOW - 20 * 24 * H, stale: true }]), NOW);
+ assert.strictEqual(f.text, '📡 On 1 device · none online · last checked 20d ago (not checked recently)');
+ assert.strictEqual(f.stale, true);
+});
+
+test('pickPeers: online only, spot-checked first', () => {
+ const p = payload([
+ { peer: 'off', online: false, lastVerifiedAt: NOW, trust: 'challenged' },
+ { peer: 'on1', online: true, lastVerifiedAt: NOW, trust: 'reported' },
+ { peer: 'on2', online: true, lastVerifiedAt: NOW - H, trust: 'challenged' },
+ ]);
+ assert.deepStrictEqual(C.pickPeers(p).map((x) => x.peer), ['on2', 'on1']);
+});

91
plans/patches/016-client-intake.diff Normal file
View File

@@ -0,0 +1,91 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -6,6 +6,9 @@
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
+ * contribute(videoId) / unknownVideos()
+ * hand a saved file the server doesn't
+ * know yet to /api/p2p/intake (plan 016)
* onMessage(type, fn) / signal(to, data) / peer()
* live /ws/p2p socket (plan 014): the
* device is "online" while it is open
@@ -33,6 +36,7 @@
let again = false;
let timer = null;
let lastSync = 0;
+ let lastUnknown = []; // cids the server did not recognise at the last report
function device() {
try {
@@ -130,6 +134,7 @@
if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; }
const j = await r.json();
if (!j || !j.ok) return null;
+ lastUnknown = Array.isArray(j.unknown) ? j.unknown : [];
const accepted = new Set(j.accepted || []);
for (const rec of recs) {
if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); }
@@ -151,6 +156,33 @@
return j;
}
+ // ---- intake (plan 016) ---------------------------------------------------------
+ // Send one saved video to the server so it can hash + validate it and add its
+ // cid to the catalog. Uploads the whole file: only on the user's request
+ // ("Verify & share") or when the server asks for a copy (plan 018).
+ async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) {
+ const dev = await ensureDevice();
+ const rec = await window.DeviceDB.getFile(videoId);
+ const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null;
+ if (!file) return { ok: false, error: 'not saved on this device' };
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
+ const t = await (await fetch('/api/p2p/intake', {
+ method: 'POST', headers: { ...h, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }),
+ })).json().catch(() => ({ ok: false, error: 'server unreachable' }));
+ if (!t.ok || t.known) { if (t.known) changed(); return t; }
+ const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file }))
+ .json().catch(() => ({ ok: false, error: 'upload failed' }));
+ if (r.ok) changed();
+ return r;
+ }
+
+ async function unknownVideos() {
+ if (!lastUnknown.length || !window.DeviceDB) return [];
+ const recs = await window.DeviceDB.listFiles();
+ return recs.filter((r) => r.cid && lastUnknown.includes(r.cid)).map((r) => r.videoId);
+ }
+
// ---- presence socket (/ws/p2p) — plan 014 ------------------------------------
// Open while sharing OR receiving is on. Credentials go in the first message,
// never the URL. Reconnects with backoff (5 s … 5 min).
@@ -258,7 +290,7 @@
}
window.P2PClient = {
- start, changed, sync, device, authHeaders, config,
+ start, changed, sync, device, authHeaders, config, contribute, unknownVideos,
onMessage, signal, peer: () => myPeer, isConnected: () => !!(ws && ws.readyState === 1 && myPeer),
};
}());
--- a/frontend/opfs.js
+++ b/frontend/opfs.js
@@ -108,6 +108,17 @@
}
},
+ // The saved File itself (disk-backed, not read into memory) — used as an
+ // upload body by P2P intake. null when not saved.
+ async getFileObject(videoId) {
+ try {
+ const found = await findHandle(videoId);
+ return found ? await found[0].getFile() : null;
+ } catch {
+ return null;
+ }
+ },
+
// Bytes [offset, offset+length) of a saved video — answers the server's
// P2P range challenges (p2p-client.js). null when the file is missing.
async readRange(videoId, offset, length) {

81
plans/patches/016-media-cache-adopt.diff Normal file
View File

@@ -0,0 +1,81 @@
--- a/server/media-cache.js
+++ b/server/media-cache.js
@@ -522,6 +522,42 @@
}
// The "Broken" button: drop the copy and fetch it again at high priority.
+ // Promote a file that came from somewhere else (P2P intake / rehydrate from
+ // a device) as this video's server copy. The caller has already hashed it
+ // and run validateMedia(). The bytes are kept EXACTLY (no remux — the cid
+ // must stay true); only the audio sidecar is derived. Never replaces an
+ // existing ready copy or races a running fetch.
+ async function adoptFile(id, src, { sha256, probe, meta = {} }) {
+ if (!isMediaId(id)) throw new Error('adopt: bad video id');
+ const cur = await db.getMedia(id);
+ if (cur && cur.status === 'ready' && readyFilesExist(cur)) return { adopted: false, reason: 'already cached' };
+ if (jobs.has(id)) return { adopted: false, reason: 'fetch in progress' };
+ await makeRoom(id, probe.size);
+ const prefix = `${id}-${now()}-adopt`;
+ const m4a = join(tmpDir, `${prefix}.m4a`);
+ try {
+ const ex = await run(ffmpeg, ['-v', 'error', '-nostdin', '-y', '-i', src,
+ '-map', '0:a:0', '-c', 'copy', '-movflags', '+faststart', '-f', 'mp4', m4a]);
+ if (ex.code !== 0 || sizeOf(m4a) < 1024) throw new Error('audio sidecar failed: ' + tail(ex.stderr));
+ const gen = ((cur && cur.gen) || 0) + 1;
+ try { renameSync(src, fileFor(id, gen)); } catch { copyFileSync(src, fileFor(id, gen)); unlinkSync(src); }
+ renameSync(m4a, fileFor(id, gen, 'm4a'));
+ const t = now();
+ await db.upsertMedia(id, {
+ status: 'ready', gen, size: probe.size + sizeOf(fileFor(id, gen, 'm4a')),
+ height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration,
+ optimized: OPT_REV, // re-encoding would change the bytes other devices hold
+ meta: JSON.stringify(meta), priority: HIGH, auto: 0, attempts: 0, error: null, retry_at: 0,
+ created_at: (cur && cur.created_at) || t, updated_at: t, last_access: t, sha256,
+ });
+ removeFiles(id, gen);
+ log.info?.(`[media] adopted ${id} ${probe.height}p ${fmtMB(probe.size)} from P2P`);
+ return { adopted: true, gen };
+ } finally {
+ sweepTmp(prefix);
+ }
+ }
+
async function redownload(id) {
if (!isMediaId(id)) throw new MediaSkip('invalid video id');
const existing = jobs.get(id);
@@ -676,5 +712,5 @@
return n;
}
- return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes };
+ return { init, ensureCached, redownload, verify, getReady, filePath, touch, status, stats, isMediaId, backfillHashes, adoptFile };
}
--- a/server/media-cache.test.js
+++ b/server/media-cache.test.js
@@ -261,6 +261,26 @@
expect(await dbmod.getMedia('retAAAAAAA1')).not.toBeNull();
});
+ test('adoptFile promotes a validated outside file byte-for-byte, once', async () => {
+ await clearDb();
+ const { cache, dir, calls } = makeCache();
+ await cache.init();
+ const src = join(root, 'adopt-src.mp4');
+ copyFileSync(fx('good-fs.mp4'), src);
+ const bytes = readFileSync(src);
+ const probe = await validateMedia(src, 0);
+ const r = await cache.adoptFile('adoptAAAAA1', src, { sha256: 'e'.repeat(64), probe, meta: { title: 'from a device' } });
+ expect(r.adopted).toBe(true);
+ const row = await cache.getReady('adoptAAAAA1');
+ expect(row.sha256).toBe('e'.repeat(64));
+ expect(row.optimized).toBe(OPT_REV);
+ expect(readFileSync(join(dir, `adoptAAAAA1.${row.gen}.mp4`)).equals(bytes)).toBe(true);
+ expect(existsSync(join(dir, `adoptAAAAA1.${row.gen}.m4a`))).toBe(true);
+ expect(calls.length).toBe(0); // nothing fetched from the source
+ copyFileSync(fx('good-fs.mp4'), src);
+ expect((await cache.adoptFile('adoptAAAAA1', src, { sha256: 'f'.repeat(64), probe })).adopted).toBe(false);
+ });
+
test('free-disk guard skips caching', async () => {
await clearDb();
const { cache, calls } = makeCache({ freeBytes: () => 1024 ** 3, minFreeBytes: 5 * 1024 ** 3 });

230
plans/patches/016-p2p-intake-new.diff Normal file
View File

@@ -0,0 +1,230 @@
--- /dev/null
+++ b/server/p2p-intake.js
@@ -0,0 +1,121 @@
+/* ============================================================================
+ * p2p-intake.js — a device hands a file to the server for verification
+ * (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did
+ * not come from the server's own fetch can become verified content.
+ *
+ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? }
+ * → { ok, known:true } cid already verified — nothing to send
+ * → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min
+ * PUT /api/p2p/intake/:ticket (device) raw file body
+ * → { ok, cid, adopted } admitted (+ adopted into the media cache)
+ * → 4xx { ok:false, error } rejected; nothing kept
+ *
+ * Order is fixed: bytes land in P2P_INTAKE_DIR (never served) → the SERVER
+ * hashes them while writing → a claimed cid must match → validateMedia() →
+ * admitFile() (malware scan only when P2P_MALWARE_SCAN=1) → holder row for
+ * the uploader → optional adoption into the media cache. Any failure deletes
+ * the file.
+ * ========================================================================== */
+import { createHash, randomBytes } from 'node:crypto';
+import { createWriteStream, mkdirSync, unlinkSync } from 'node:fs';
+import { join } from 'node:path';
+
+const CID_RE = /^[0-9a-f]{64}$/;
+const ID_RE = /^[A-Za-z0-9_-]{6,64}$/;
+const TICKET_TTL = 30 * 60_000;
+const MAX_ACTIVE = 2;
+// Display text from the device — untrusted: control chars/brackets stripped, capped.
+const clean = (v, max) => String(v || '').replace(/[\u0000-\u001f\u007f<>]+/g, ' ').replace(/\s+/g, ' ').trim().slice(0, max);
+
+export function registerIntakeRoutes(app, deps) {
+ const {
+ cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null,
+ now = () => Date.now(), log = console,
+ } = deps;
+ const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy }
+ let active = 0;
+ mkdirSync(cfg.intakeDir, { recursive: true });
+
+ const sweep = () => { const t = now(); for (const [k, v] of tickets) if (!v.busy && v.expiresAt < t) tickets.delete(k); };
+
+ app.post('/api/p2p/intake', gate, requireDevice, async (c) => {
+ const d = c.get('device');
+ const body = await c.req.json().catch(() => ({}));
+ const videoId = String(body.videoId || '');
+ const cid = body.cid ? String(body.cid).toLowerCase() : null;
+ const size = Number(body.size);
+ if (!ID_RE.test(videoId) || (cid && !CID_RE.test(cid))) return c.json({ ok: false, error: 'bad videoId or cid' }, 400);
+ if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413);
+ if (cid) {
+ const known = await p2pDb.getContent(cid);
+ if (known && known.status === 'verified') return c.json({ ok: true, known: true });
+ if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410);
+ }
+ sweep();
+ for (const v of tickets.values()) if (v.deviceId === d.device_id) return c.json({ ok: false, error: 'an upload from this device is already open' }, 429);
+ const ticket = randomBytes(16).toString('hex');
+ const expiresAt = now() + TICKET_TTL;
+ const meta = { title: clean(body.title, 300), channel: clean(body.channel, 200) };
+ tickets.set(ticket, { deviceId: d.device_id, videoId, cid, size, meta, expiresAt, busy: false });
+ return c.json({ ok: true, ticket, url: `/api/p2p/intake/${ticket}`, expiresAt });
+ });
+
+ app.put('/api/p2p/intake/:ticket', gate, requireDevice, async (c) => {
+ const d = c.get('device');
+ const ticket = c.req.param('ticket');
+ const tk = tickets.get(ticket);
+ if (!tk || tk.deviceId !== d.device_id || tk.expiresAt < now()) return c.json({ ok: false, error: 'unknown or expired ticket' }, 404);
+ if (tk.busy) return c.json({ ok: false, error: 'upload already running' }, 409);
+ if (active >= MAX_ACTIVE) return c.json({ ok: false, error: 'server busy, retry later' }, 503);
+ tk.busy = true;
+ active++;
+ const path = join(cfg.intakeDir, ticket + '.mp4');
+ let keep = false;
+ try {
+ const h = createHash('sha256');
+ let got = 0;
+ const out = createWriteStream(path);
+ const reader = c.req.raw.body ? c.req.raw.body.getReader() : null;
+ if (!reader) throw Object.assign(new Error('empty body'), { status: 400 });
+ try {
+ for (;;) {
+ const { done, value } = await reader.read();
+ if (done) break;
+ got += value.byteLength;
+ if (got > tk.size) throw Object.assign(new Error('more bytes than announced'), { status: 413 });
+ h.update(value);
+ if (!out.write(value)) await new Promise((r) => out.once('drain', r));
+ }
+ } finally {
+ await new Promise((r) => out.end(r));
+ }
+ if (got !== tk.size) throw Object.assign(new Error(`size mismatch (${got} of ${tk.size})`), { status: 400 });
+ const cid = h.digest('hex');
+ if (tk.cid && cid !== tk.cid) throw Object.assign(new Error('content hash mismatch'), { status: 400 });
+ let probe;
+ try { probe = await validateMedia(path, 0, { codecs: ['h264', 'hevc'] }); }
+ catch (e) { throw Object.assign(new Error(e.message), { status: 422 }); }
+ const adm = await admitFile(
+ { path, cid, videoId: tk.videoId, size: got, height: probe.height, vcodec: probe.vcodec, acodec: probe.acodec, duration: probe.duration, meta: tk.meta, origin: 'intake' },
+ { cfg, upsertContent: p2pDb.upsertContent, log },
+ );
+ if (!adm.ok) throw Object.assign(new Error('not admitted: ' + adm.reason), { status: 422 });
+ await p2pDb.upsertHolder({ cid, deviceId: d.device_id, trust: 'challenged', now: now() });
+ let adopted = false;
+ if (adopt) {
+ try { adopted = !!(await adopt(tk.videoId, path, { sha256: cid, probe, meta: tk.meta })).adopted; keep = adopted; }
+ catch (e) { log.warn?.(`[p2p] adopt ${tk.videoId} failed: ${e.message}`); }
+ }
+ log.info?.(`[p2p] intake ${tk.videoId} ${cid.slice(0, 12)} admitted${adopted ? ' + adopted' : ''}`);
+ return c.json({ ok: true, cid, adopted });
+ } catch (e) {
+ return c.json({ ok: false, error: e.message }, e.status || 500);
+ } finally {
+ tickets.delete(ticket);
+ active--;
+ if (!keep) { try { unlinkSync(path); } catch { /* never written */ } }
+ }
+ });
+
+ return { openTickets: () => tickets.size, active: () => active };
+}
--- /dev/null
+++ b/server/p2p-intake.test.js
@@ -0,0 +1,103 @@
+// Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg.
+import { test, expect, beforeAll } from 'bun:test';
+import { mkdtempSync, readFileSync, readdirSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createHash } from 'node:crypto';
+import { Hono } from 'hono';
+
+const root = mkdtempSync(join(tmpdir(), 'ytp-intake-'));
+process.env.DB_PATH = join(root, 'test.db');
+const dbmod = await import('./db.js');
+const p2pDb = await import('./p2p-db.js');
+const { registerP2pRoutes } = await import('./p2p-routes.js');
+const { registerIntakeRoutes } = await import('./p2p-intake.js');
+const { admitFile } = await import('./p2p-admit.js');
+const { validateMedia } = await import('./media-cache.js');
+
+const quiet = { warn() {}, info() {} };
+const intakeDir = join(root, 'intake');
+const cfg = { enabled: true, malwareScan: false, scanCmd: 'true', staleDays: 7, intakeDir, intakeMaxBytes: 50 * 1024 * 1024 };
+let app;
+let dev;
+let good;
+let goodCid;
+const adopted = [];
+
+const req = (path, { method = 'GET', body, raw } = {}) => app.request(path, {
+ method,
+ headers: { ...(raw ? {} : { 'Content-Type': 'application/json' }), 'X-Device': dev.deviceId + '.' + dev.secret },
+ body: raw || (body ? JSON.stringify(body) : undefined),
+});
+
+beforeAll(async () => {
+ await dbmod.initDb();
+ await p2pDb.initP2pSchema();
+ const f = join(root, 'good.mp4');
+ const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=size=320x180:rate=25:duration=4', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=4',
+ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', f]);
+ if (r.status !== 0) throw new Error('ffmpeg fixture failed: ' + r.stderr);
+ good = readFileSync(f);
+ goodCid = createHash('sha256').update(good).digest('hex');
+ app = new Hono();
+ const p2p = registerP2pRoutes(app, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ registerIntakeRoutes(app, {
+ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet,
+ adopt: async (videoId, path, info) => { adopted.push({ videoId, sha256: info.sha256, meta: info.meta }); return { adopted: false }; },
+ });
+ dev = await (await app.request('/api/p2p/device', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' })).json();
+});
+
+test('a valid file is hashed by the server, validated, admitted and its uploader becomes a holder', async () => {
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, title: 'Song <b>x</b>', channel: 'Choir' } })).json();
+ expect(t.ok).toBe(true);
+ const r = await req(t.url, { method: 'PUT', raw: good });
+ expect(r.status).toBe(200);
+ expect(await r.json()).toEqual({ ok: true, cid: goodCid, adopted: false });
+ expect((await p2pDb.getContent(goodCid))).toMatchObject({ origin: 'intake', status: 'verified', scan: 'skipped', video_id: 'upAAAAAAAA1' });
+ expect((await p2pDb.listHolders(goodCid))[0]).toMatchObject({ device_id: dev.deviceId, trust: 'challenged' });
+ expect(adopted).toEqual([{ videoId: 'upAAAAAAAA1', sha256: goodCid, meta: { title: 'Song b x /b', channel: 'Choir' } }]);
+ expect(JSON.parse((await p2pDb.getContent(goodCid)).meta).title).toBe('Song b x /b');
+ expect(readdirSync(intakeDir)).toEqual([]); // not adopted → deleted
+ // Already verified → nothing to upload next time.
+ expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true });
+});
+
+test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => {
+ const cases = [
+ [{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/],
+ [{ bytes: good.subarray(0, Math.floor(good.length * 0.6)) }, 422, /validation/],
+ [{ bytes: Buffer.alloc(200 * 1024, 7) }, 422, /validation/],
+ ];
+ for (const [c, status, msg] of cases) {
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA2', cid: c.cid, size: c.bytes.length } })).json();
+ const r = await req(t.url, { method: 'PUT', raw: c.bytes });
+ expect(r.status).toBe(status);
+ expect((await r.json()).error).toMatch(msg);
+ }
+ expect(readdirSync(intakeDir)).toEqual([]);
+ expect((await p2pDb.listContentForVideo('upAAAAAAAA2')).length).toBe(0);
+});
+
+test('size limits and tickets are enforced', async () => {
+ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: cfg.intakeMaxBytes + 1 } })).status).toBe(413);
+ const t = await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA3', size: 10 } })).json();
+ expect((await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA4', size: 10 } })).status).toBe(429); // one open per device
+ const r = await req(t.url, { method: 'PUT', raw: Buffer.alloc(11) });
+ expect(r.status).toBe(413);
+ expect((await req(t.url, { method: 'PUT', raw: Buffer.alloc(10) })).status).toBe(404); // ticket consumed
+});
+
+test('malware scan on: an infected verdict is never admitted', async () => {
+ const cfg2 = { ...cfg, malwareScan: true, scanCmd: 'false' }; // exit 1 = infected
+ const app2 = new Hono();
+ const p2p = registerP2pRoutes(app2, { cfg: cfg2, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ registerIntakeRoutes(app2, { cfg: cfg2, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet });
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
+ const t = await (await app2.request('/api/p2p/intake', { method: 'POST', headers: { ...h, 'Content-Type': 'application/json' }, body: JSON.stringify({ videoId: 'upAAAAAAAA5', size: good.length }) })).json();
+ await p2pDb.revokeContent(goodCid); // make sure it is not simply "already known"
+ const r = await app2.request(t.url, { method: 'PUT', headers: h, body: good });
+ expect(r.status).toBe(422);
+ expect((await r.json()).error).toMatch(/scan infected/);
+});

280
plans/patches/017-p2p-transfer-new.diff Normal file
View File

@@ -0,0 +1,280 @@
--- /dev/null
+++ b/frontend/p2p-transfer.js
@@ -0,0 +1,184 @@
+/* ============================================================================
+ * p2p-transfer.js — download a verified file from another device over a
+ * WebRTC data channel (docs/p2p-architecture.md flow 6). window.P2PTransfer.
+ *
+ * start({ canShare }) serve requests from other devices (called once)
+ * download({ videoId, cid, size, peers, onProgress }) → { ok, error? }
+ *
+ * Signalling rides P2PClient.signal()/onMessage('signal') over /ws/p2p:
+ * {k:'offer', xid, cid, sdp} · {k:'answer', xid, sdp} · {k:'ice', xid, cand}
+ * {k:'deny', xid, reason}
+ * Data channel "file" (ordered): requester sends {t:'get'}; sender answers
+ * {t:'meta', size}, 64 KiB binary frames (back-pressured on bufferedAmount),
+ * then {t:'end'}. The receiver writes through p2p-recv-worker.js, which only
+ * commits the file when its SHA-256 equals the cid. STUN only — devices
+ * behind strict NATs won't connect (same as watch-party voice); the next
+ * holder is tried.
+ * ========================================================================== */
+(function () {
+ 'use strict';
+
+ const ICE = [{ urls: 'stun:stun.l.google.com:19302' }, { urls: 'stun:stun1.l.google.com:19302' }];
+ const FRAME = 64 * 1024;
+ const HIGH_WATER = 4 * 1024 * 1024;
+ const OPEN_TIMEOUT = 20_000;
+ const IDLE_TIMEOUT = 30_000;
+ const MAX_SERVING = 1;
+
+ let iceServers = ICE;
+ let canShare = () => true;
+ const sessions = new Map(); // xid -> { pc, onSignal }
+ let serving = 0;
+
+ const newXid = () => Math.random().toString(36).slice(2) + Date.now().toString(36);
+
+ function onSignal(m) {
+ const d = m && m.data;
+ if (!d || !d.xid) return;
+ const s = sessions.get(d.xid);
+ if (s) { s.onSignal(d, m.from); return; }
+ if (d.k === 'offer') serve(d, m.from).catch(() => {});
+ }
+
+ // ---- sender -------------------------------------------------------------------
+ async function serve(offer, from) {
+ const deny = (reason) => window.P2PClient.signal(from, { k: 'deny', xid: offer.xid, reason });
+ if (!canShare()) return deny('sharing off');
+ if (serving >= MAX_SERVING) return deny('busy');
+ const rec = await window.DeviceDB.getByCid(offer.cid);
+ const file = rec && window.OPFS.getFileObject ? await window.OPFS.getFileObject(rec.videoId) : null;
+ if (!rec || !file || file.size !== rec.size) return deny('not here');
+ serving++;
+ const pc = new RTCPeerConnection({ iceServers });
+ let done = false;
+ const finish = () => {
+ if (done) return;
+ done = true;
+ serving--;
+ sessions.delete(offer.xid);
+ try { pc.close(); } catch { /* closed */ }
+ };
+ sessions.set(offer.xid, {
+ pc,
+ onSignal: (d) => { if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {}); },
+ });
+ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(from, { k: 'ice', xid: offer.xid, cand: e.candidate.toJSON() }); };
+ pc.onconnectionstatechange = () => { if (['failed', 'closed', 'disconnected'].includes(pc.connectionState)) finish(); };
+ setTimeout(() => { if (pc.connectionState !== 'connected') finish(); }, OPEN_TIMEOUT);
+ pc.ondatachannel = (e) => {
+ const dc = e.channel;
+ dc.bufferedAmountLowThreshold = 1024 * 1024;
+ dc.onmessage = async (ev) => {
+ let msg;
+ try { msg = JSON.parse(ev.data); } catch { return; }
+ if (msg.t !== 'get') return;
+ try {
+ dc.send(JSON.stringify({ t: 'meta', size: file.size }));
+ for (let pos = 0; pos < file.size && !done; pos += 4 * FRAME) {
+ const buf = await file.slice(pos, pos + 4 * FRAME).arrayBuffer();
+ for (let i = 0; i < buf.byteLength; i += FRAME) {
+ if (dc.bufferedAmount > HIGH_WATER) {
+ await new Promise((r) => { dc.onbufferedamountlow = () => { dc.onbufferedamountlow = null; r(); }; });
+ }
+ dc.send(buf.slice(i, i + FRAME));
+ }
+ }
+ dc.send(JSON.stringify({ t: 'end' }));
+ } catch { finish(); }
+ };
+ dc.onclose = finish;
+ };
+ await pc.setRemoteDescription({ type: 'offer', sdp: offer.sdp });
+ const answer = await pc.createAnswer();
+ await pc.setLocalDescription(answer);
+ window.P2PClient.signal(from, { k: 'answer', xid: offer.xid, sdp: answer.sdp });
+ }
+
+ // ---- receiver -----------------------------------------------------------------
+ function tryPeer({ videoId, cid, size, peer, onProgress }) {
+ return new Promise((resolve) => {
+ const xid = newXid();
+ const pc = new RTCPeerConnection({ iceServers });
+ const worker = new Worker('/p2p-recv-worker.js');
+ let settled = false;
+ let idle = null;
+ let expected = size;
+ const end = (res) => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(idle);
+ clearTimeout(openTimer);
+ sessions.delete(xid);
+ try { pc.close(); } catch { /* closed */ }
+ if (!res.ok) worker.postMessage({ op: 'abort' });
+ setTimeout(() => worker.terminate(), res.ok ? 0 : 2000);
+ resolve(res);
+ };
+ const bump = () => { clearTimeout(idle); idle = setTimeout(() => end({ ok: false, error: 'peer went quiet' }), IDLE_TIMEOUT); };
+ const openTimer = setTimeout(() => end({ ok: false, error: 'could not connect' }), OPEN_TIMEOUT);
+
+ worker.onmessage = (e) => {
+ const m = e.data || {};
+ if (m.op === 'progress' && onProgress) onProgress(m.received, expected);
+ if (m.op === 'done') end(m.ok ? { ok: true, sha256: m.sha256, size: m.size } : { ok: false, error: m.error });
+ };
+ worker.postMessage({ op: 'open', videoId });
+
+ sessions.set(xid, {
+ pc,
+ onSignal: (d) => {
+ if (d.k === 'answer') pc.setRemoteDescription({ type: 'answer', sdp: d.sdp }).catch(() => end({ ok: false, error: 'bad answer' }));
+ else if (d.k === 'ice' && d.cand) pc.addIceCandidate(d.cand).catch(() => {});
+ else if (d.k === 'deny') end({ ok: false, error: 'peer declined: ' + d.reason });
+ },
+ });
+ pc.onicecandidate = (e) => { if (e.candidate) window.P2PClient.signal(peer, { k: 'ice', xid, cand: e.candidate.toJSON() }); };
+ const dc = pc.createDataChannel('file', { ordered: true });
+ dc.binaryType = 'arraybuffer';
+ dc.onopen = () => { clearTimeout(openTimer); bump(); dc.send(JSON.stringify({ t: 'get' })); };
+ dc.onmessage = (e) => {
+ bump();
+ if (typeof e.data === 'string') {
+ let m;
+ try { m = JSON.parse(e.data); } catch { return; }
+ if (m.t === 'meta') {
+ if (m.size !== size) end({ ok: false, error: 'peer has a different file size' });
+ expected = m.size;
+ } else if (m.t === 'end') {
+ clearTimeout(idle);
+ worker.postMessage({ op: 'finish', cid, size });
+ }
+ return;
+ }
+ worker.postMessage({ op: 'chunk', buf: e.data }, [e.data]);
+ };
+ dc.onclose = () => { if (!settled) setTimeout(() => end({ ok: false, error: 'channel closed' }), 5000); };
+ (async () => {
+ try {
+ const offer = await pc.createOffer();
+ await pc.setLocalDescription(offer);
+ if (!window.P2PClient.signal(peer, { k: 'offer', xid, cid, sdp: offer.sdp })) end({ ok: false, error: 'not connected to the P2P hub' });
+ } catch (err) { end({ ok: false, error: err.message }); }
+ })();
+ });
+ }
+
+ // Try each online holder in turn until one delivers a verified file.
+ async function download({ videoId, cid, size, peers, onProgress }) {
+ if (!window.P2PClient || !window.P2PClient.isConnected()) return { ok: false, error: 'not connected to the P2P hub' };
+ let last = { ok: false, error: 'no online device has this video' };
+ for (const p of peers || []) {
+ last = await tryPeer({ videoId, cid, size, peer: p.peer, onProgress });
+ if (last.ok) return last;
+ }
+ return last;
+ }
+
+ function start(opts = {}) {
+ if (opts.canShare) canShare = opts.canShare;
+ if (opts.iceServers) iceServers = opts.iceServers;
+ window.P2PClient.onMessage('signal', onSignal);
+ }
+
+ window.P2PTransfer = { start, download };
+}());
--- /dev/null
+++ b/frontend/p2p-recv-worker.js
@@ -0,0 +1,90 @@
+/* ============================================================================
+ * p2p-recv-worker.js — writes a file arriving from another device into OPFS
+ * while hashing it; commits it ONLY when the SHA-256 equals the expected
+ * content id (docs/p2p-architecture.md flow 6).
+ *
+ * In: { op:'open', videoId } start videos/<videoId>.p2p.part
+ * { op:'chunk', buf } ArrayBuffer (transferred)
+ * { op:'finish', cid, size } verify + rename to <videoId>.mp4
+ * { op:'abort' } drop the partial file
+ * Out: { op:'opened' } | { op:'progress', received } |
+ * { op:'done', ok:true, sha256, size } | { op:'done', ok:false, error }
+ * The ".part" suffix keeps OPFS.listVideos() from ever listing a partial file.
+ * ========================================================================== */
+'use strict';
+importScripts('/sha256.js');
+
+let dir = null;
+let handle = null;
+let access = null;
+let partName = '';
+let finalName = '';
+let hasher = null;
+let offset = 0;
+let lastProgress = 0;
+
+async function cleanup() {
+ try { if (access) access.close(); } catch { /* closed */ }
+ access = null;
+ try { if (dir && partName) await dir.removeEntry(partName); } catch { /* gone */ }
+}
+
+// Messages are handled strictly one after another: an async handler would
+// otherwise let 'chunk' or 'abort' run while 'open' is still awaiting OPFS.
+let chain = Promise.resolve();
+self.onmessage = (e) => { chain = chain.then(() => onOp(e.data || {})); };
+
+async function onOp(m) {
+ try {
+ if (m.op === 'open') {
+ const root = await navigator.storage.getDirectory();
+ dir = await root.getDirectoryHandle('videos', { create: true });
+ partName = `${m.videoId}.p2p.part`;
+ finalName = `${m.videoId}.mp4`;
+ handle = await dir.getFileHandle(partName, { create: true });
+ access = await handle.createSyncAccessHandle();
+ access.truncate(0);
+ hasher = self.Sha256.create();
+ offset = 0;
+ self.postMessage({ op: 'opened' });
+ } else if (m.op === 'chunk') {
+ const u8 = new Uint8Array(m.buf);
+ access.write(u8, { at: offset });
+ hasher.update(u8);
+ offset += u8.byteLength;
+ if (offset - lastProgress > 1024 * 1024) { lastProgress = offset; self.postMessage({ op: 'progress', received: offset }); }
+ } else if (m.op === 'finish') {
+ access.truncate(offset);
+ access.flush();
+ access.close();
+ access = null;
+ const sha256 = hasher.hex();
+ if (offset !== m.size) throw new Error(`size mismatch (${offset} of ${m.size})`);
+ if (sha256 !== m.cid) throw new Error('content hash mismatch');
+ try { await dir.removeEntry(finalName); } catch { /* no previous copy */ }
+ let renamed = false;
+ if (typeof handle.move === 'function') { try { await handle.move(finalName); renamed = true; } catch { /* copy below */ } }
+ if (!renamed) {
+ const out = await (await dir.getFileHandle(finalName, { create: true })).createSyncAccessHandle();
+ try {
+ const file = await handle.getFile();
+ for (let pos = 0; pos < file.size; pos += 8 * 1024 * 1024) {
+ const buf = new Uint8Array(await file.slice(pos, pos + 8 * 1024 * 1024).arrayBuffer());
+ out.write(buf, { at: pos });
+ }
+ out.truncate(file.size);
+ out.flush();
+ } finally { out.close(); }
+ await dir.removeEntry(partName);
+ }
+ partName = '';
+ self.postMessage({ op: 'done', ok: true, sha256, size: offset });
+ } else if (m.op === 'abort') {
+ await cleanup();
+ self.postMessage({ op: 'done', ok: false, error: 'aborted' });
+ }
+ } catch (err) {
+ await cleanup();
+ self.postMessage({ op: 'done', ok: false, error: err && err.message ? err.message : String(err) });
+ }
+}

40
plans/patches/018-client-restore.diff Normal file
View File

@@ -0,0 +1,40 @@
--- a/frontend/p2p-client.js
+++ b/frontend/p2p-client.js
@@ -160,7 +160,7 @@
// Send one saved video to the server so it can hash + validate it and add its
// cid to the catalog. Uploads the whole file: only on the user's request
// ("Verify & share") or when the server asks for a copy (plan 018).
- async function contribute(videoId, { cid = null, title = '', channel = '' } = {}) {
+ async function contribute(videoId, { cid = null, title = '', channel = '', restore = false } = {}) {
const dev = await ensureDevice();
const rec = await window.DeviceDB.getFile(videoId);
const file = window.OPFS.getFileObject ? await window.OPFS.getFileObject(videoId) : null;
@@ -168,7 +168,7 @@
const h = { 'X-Device': dev.deviceId + '.' + dev.secret };
const t = await (await fetch('/api/p2p/intake', {
method: 'POST', headers: { ...h, 'Content-Type': 'application/json' },
- body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel }),
+ body: JSON.stringify({ videoId, cid: cid || (rec && rec.cid) || undefined, size: file.size, title, channel, restore }),
})).json().catch(() => ({ ok: false, error: 'server unreachable' }));
if (!t.ok || t.known) { if (t.known) changed(); return t; }
const r = await (await fetch(t.url, { method: 'PUT', headers: h, body: file }))
@@ -280,6 +280,19 @@
timer = setTimeout(sync, 5000);
}
+ // Plan 018: the server lost its copy and the source is gone — it asks one
+ // holder to send the file back. Only while sharing is on, one at a time,
+ // and only for the exact cid this device holds.
+ let restoring = false;
+ onMessage('upload-request', async (m) => {
+ if (restoring || hooks.getSettings().p2pShare === false) return;
+ const rec = window.DeviceDB ? await window.DeviceDB.getFile(m.videoId) : null;
+ if (!rec || rec.cid !== m.cid) return;
+ restoring = true;
+ try { await contribute(m.videoId, { cid: m.cid, restore: true }); } catch { /* next request retries */ }
+ finally { restoring = false; }
+ });
+
function start(h) {
hooks = { ...hooks, ...(h || {}) };
const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1));

125
plans/patches/018-server-rehydrate.diff Normal file
View File

@@ -0,0 +1,125 @@
--- a/server/p2p-hub.js
+++ b/server/p2p-hub.js
@@ -118,3 +118,28 @@
}
return { ok: true, staleDays, cids: out };
}
+
+// Flow 8 (plan 018): the source is gone and the server evicted its copy, so
+// ask ONE online holder that shares to upload it through intake
+// ({type:'upload-request', videoId, cid}). At most one request per video per
+// 10 minutes; returns true when a device was asked (or recently was).
+export function createRehydrator({ p2pDb, hub, hasServerCopy, enabled = () => true, now = () => Date.now() }) {
+ const asked = new Map(); // videoId -> ms
+ return async function rehydrate(videoId) {
+ if (!enabled()) return false;
+ if (await hasServerCopy(videoId)) return false;
+ const last = asked.get(videoId);
+ if (last && now() - last < 10 * 60_000) return true;
+ for (const c of await p2pDb.listContentForVideo(videoId)) {
+ for (const h of await p2pDb.listHolders(c.cid, 50)) {
+ if (Number(h.share) !== 1 || !hub.isOnline(h.device_id)) continue;
+ if (hub.send(h.device_id, { type: 'upload-request', videoId, cid: c.cid })) {
+ asked.set(videoId, now());
+ if (asked.size > 5000) asked.delete(asked.keys().next().value);
+ return true;
+ }
+ }
+ }
+ return false;
+ };
+}
--- a/server/p2p-hub.test.js
+++ b/server/p2p-hub.test.js
@@ -85,3 +85,21 @@
]);
expect(JSON.stringify(p)).not.toContain('dev_'); // never leak device ids
});
+
+test('rehydrator asks one online sharing holder, once per 10 min, only when the server has no copy', async () => {
+ const { createRehydrator } = await import('./p2p-hub.js');
+ const sent = [];
+ let t = NOW;
+ let serverHas = false;
+ const hub = { isOnline: (d) => d === 'dev_000000000000000a' || d === 'dev_000000000000000c', send: (d, m) => { sent.push([d, m]); return true; } };
+ const rehydrate = createRehydrator({ p2pDb, hub, hasServerCopy: async () => serverHas, now: () => t });
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true);
+ // c is online but has sharing off; a is online and shares.
+ expect(sent).toEqual([['dev_000000000000000a', { type: 'upload-request', videoId: 'dQw4w9WgXcQ', cid: CID }]]);
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(true); // recently asked — no second message
+ expect(sent.length).toBe(1);
+ t += 11 * 60_000;
+ serverHas = true;
+ expect(await rehydrate('dQw4w9WgXcQ')).toBe(false); // server copy is back
+ expect(await rehydrate('unknownVid1')).toBe(false); // nobody holds it
+});
--- a/server/p2p-intake.js
+++ b/server/p2p-intake.js
@@ -3,8 +3,10 @@
* (docs/p2p-architecture.md flow 7). The ONLY path by which bytes that did
* not come from the server's own fetch can become verified content.
*
- * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel? }
+ * POST /api/p2p/intake (device) { videoId, cid?, size, title?, channel?, restore? }
* → { ok, known:true } cid already verified — nothing to send
+ * (unless restore:true and the server
+ * lost its copy — plan 018)
* → { ok, ticket, url, expiresAt } PUT the bytes to url within 30 min
* PUT /api/p2p/intake/:ticket (device) raw file body
* → { ok, cid, adopted } admitted (+ adopted into the media cache)
@@ -30,6 +32,7 @@
export function registerIntakeRoutes(app, deps) {
const {
cfg, p2pDb, gate, requireDevice, validateMedia, admitFile, adopt = null,
+ serverHasCid = async () => true, // (cid) → does the server still hold these bytes?
now = () => Date.now(), log = console,
} = deps;
const tickets = new Map(); // ticket -> { deviceId, videoId, cid, size, expiresAt, busy }
@@ -48,7 +51,8 @@
if (!(size > 0) || size > cfg.intakeMaxBytes) return c.json({ ok: false, error: 'bad size' }, 413);
if (cid) {
const known = await p2pDb.getContent(cid);
- if (known && known.status === 'verified') return c.json({ ok: true, known: true });
+ const restore = body.restore === true && known && known.status === 'verified' && !(await serverHasCid(cid));
+ if (known && known.status === 'verified' && !restore) return c.json({ ok: true, known: true });
if (known && known.status === 'revoked') return c.json({ ok: false, error: 'revoked' }, 410);
}
sweep();
--- a/server/p2p-intake.test.js
+++ b/server/p2p-intake.test.js
@@ -1,6 +1,6 @@
// Device → server intake: hash, validate, (scan), admit (plan 016). Needs ffmpeg.
import { test, expect, beforeAll } from 'bun:test';
-import { mkdtempSync, readFileSync, readdirSync } from 'node:fs';
+import { mkdtempSync, readFileSync, readdirSync, unlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
@@ -64,6 +64,27 @@
expect(await (await req('/api/p2p/intake', { method: 'POST', body: { videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length } })).json()).toEqual({ ok: true, known: true });
});
+test('restore: a known cid is uploaded again only when the server lost its copy', async () => {
+ let serverHas = true;
+ const app3 = new Hono();
+ const p2p = registerP2pRoutes(app3, { cfg, p2pDb, fileForCid: async () => null, sha256Range: async () => '', log: quiet });
+ const got = [];
+ registerIntakeRoutes(app3, {
+ cfg, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, validateMedia, admitFile, log: quiet,
+ serverHasCid: async () => serverHas,
+ adopt: async (videoId, path, info) => { got.push(info.sha256); unlinkSync(path); return { adopted: true }; }, // a real adopt moves the file
+ });
+ const h = { 'X-Device': dev.deviceId + '.' + dev.secret, 'Content-Type': 'application/json' };
+ const open = async () => (await app3.request('/api/p2p/intake', { method: 'POST', headers: h, body: JSON.stringify({ videoId: 'upAAAAAAAA1', cid: goodCid, size: good.length, restore: true }) })).json();
+ expect(await open()).toEqual({ ok: true, known: true }); // server still has it
+ serverHas = false;
+ const t = await open();
+ expect(t.ticket).toMatch(/^[0-9a-f]{32}$/);
+ const r = await (await app3.request(t.url, { method: 'PUT', headers: { 'X-Device': h['X-Device'] }, body: good })).json();
+ expect(r).toEqual({ ok: true, cid: goodCid, adopted: true });
+ expect(got).toEqual([goodCid]);
+});
+
test('claimed cid mismatch, truncated media, and non-media are rejected and deleted', async () => {
const cases = [
[{ cid: 'f'.repeat(64), bytes: good }, 400, /hash mismatch/],

0
plans/queue/.gitkeep Normal file
View File

123
plans/queue/001-perf-timing-marks-105acc.md Normal file
View File

@@ -0,0 +1,123 @@
---
id: 001-perf-timing-marks-105acc
title: Add startup, search and play timing marks plus yt-dlp duration logs
created: 2026-09-29
depends_on: []
est_files: 2
---
# 001 — Add startup, search and play timing marks plus yt-dlp duration logs
## Objective
Every later speed plan needs a before/after number. After this plan:
- The browser records `performance.measure` entries `ytp:boot`, `ytp:search`,
`ytp:tap-to-play`, and `window.__ytpPerf()` returns their latest values in ms.
- The server logs one line per yt-dlp call: `[ytdlp] <kind> <ms>ms ok|fail`.
Measured baseline (prod, 2026-09-29): search 4.1–5.0 s, first play `/api/streams` 7.5 s,
second play 1.2 s, `/api/version` 1.25 s.
## Context the executor must NOT rediscover
`server/server.js:143-165` — the only place yt-dlp is spawned:
```js
function runYtdlp(args, { signal } = {}) {
return new Promise((resolve, reject) => {
const child = spawn(YTDLP, args, { stdio: ['ignore', 'pipe', 'pipe'] });
...
child.on('error', (e) => reject(new Error('yt-dlp not found: ' + e.message)));
child.on('close', (code) => {
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
});
}
```
`frontend/app.js`:
- `async function boot()` starts at ~line 9749 (`wirePlayerEvents();` is its first line).
The first `render()` call inside boot happens after the `hasShareParam` block.
- `async function runSearchQuery(q, { instant = null } = {})` at ~line 8583; the
success path ends with `RecentSearches.cacheResults(q, results);`.
- `Player.loadVideo(videoObj, …)` at ~line 1650; first statement is
`if (!this._handoff) Transition.cancel();`.
- The master element `playing` listener at ~line 2287:
```js
el.addEventListener('playing', () => {
if (!masterIs(el)) return;
showSpinner(false);
```
## Steps
1. `server/server.js` — in `runYtdlp`, directly after the `const child = spawn(...)` line add:
```js
const t0 = Date.now();
const kind = String(args.find((a) => /^ytsearch|^https?:/.test(String(a))) || args[0] || '')
.replace(/^ytsearch\d*:.*/, 'search').replace(/^https?:\/\/[^/]+\/watch.*/, 'video').slice(0, 40);
```
and replace the `child.on('close', …)` handler body with:
```js
child.on('close', (code) => {
console.log(`[ytdlp] ${kind} ${Date.now() - t0}ms ${code === 0 ? 'ok' : 'fail'}`);
if (code !== 0) reject(new Error(err.trim() || 'yt-dlp exited with code ' + code));
else resolve(out);
});
```
2. `frontend/app.js` — near the top of the file, directly after the line
`const APP_VERSION = '1.0.0';` (~line 24), add:
```js
// Timing marks for the speed work (plans/). performance.measure entries are
// visible in DevTools → Performance; __ytpPerf() prints the latest ones.
function perfMark(name) { try { performance.mark(name); } catch { /* old browser */ } }
function perfMeasure(name, start) {
try { performance.measure(name, start); } catch { /* start mark missing */ }
}
window.__ytpPerf = () => {
const out = {};
try { for (const m of performance.getEntriesByType('measure')) if (m.name.startsWith('ytp:')) out[m.name] = Math.round(m.duration); } catch { /* none */ }
return out;
};
```
3. `frontend/app.js` `boot()` — first line of the function body: `perfMark('ytp:boot-start');`.
Immediately after the FIRST `render();` call inside `boot()` add
`perfMeasure('ytp:boot', 'ytp:boot-start');`.
4. `frontend/app.js` `runSearchQuery` — after `const mySeq = ++searchSeq;` add
`perfMark('ytp:search-start');`. After `RecentSearches.cacheResults(q, results);` add
`perfMeasure('ytp:search', 'ytp:search-start');`.
5. `frontend/app.js` `Player.loadVideo` — first line of the body: `perfMark('ytp:tap');`.
6. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` add:
```js
if (performance.getEntriesByName('ytp:tap').length) {
perfMeasure('ytp:tap-to-play', 'ytp:tap');
try { performance.clearMarks('ytp:tap'); } catch { /* ignore */ }
}
```
## Out of scope / do NOT touch
- No reporting endpoint, no UI. Don't change any behaviour, only add marks/logs.
- Do not touch `runYtdlpResilient` or the fallback-client logic.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
grep -c "perfMark\|perfMeasure" frontend/app.js
```
Expected: no syntax errors, `SERVER_OK`, tests `fail 0`, grep count ≥ 8.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

156
plans/queue/002-compress-and-etag-shell-bd459c.md Normal file
View File

@@ -0,0 +1,156 @@
---
id: 002-compress-and-etag-shell-bd459c
title: Serve the app shell gzip/brotli-compressed with ETags
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 2
---
# 002 — Serve the app shell gzip/brotli-compressed with ETags
## Objective
Measured on prod: `app.js` is sent as 426 244 bytes with no `content-encoding`
and no `ETag` (2.8 s download at ~150 KB/s). gzip -9 makes it 117 527 bytes;
the whole shell drops from 618 KB to ~158 KB. After this plan:
- Every static text file under `./public` (`.js .css .html .json .webmanifest .svg`)
is served compressed (`br` preferred, else `gzip`) when the client accepts it,
with `Vary: Accept-Encoding`.
- Every static file carries a strong `ETag` and `If-None-Match` returns `304`.
- `index.html` (build-stamped at request time) and `/sw.js` (BUILD_TAG-injected) are
also compressed + ETagged using the text they actually send.
- `Cache-Control` values stay EXACTLY as they are today (`no-cache` for the shell,
`no-store` for sw.js) — CLAUDE.md "Update-flow architecture" depends on that.
## Context the executor must NOT rediscover
`server/server.js:1895-1910` today:
```js
function indexHtml(c) {
if (_indexSource === null) {
try { _indexSource = readFileSync('./public/index.html', 'utf8'); }
catch { return c.text('index.html not found', 404); }
}
return c.html(_indexSource.replace('__BUILD_TAG__', BUILD_TAG), 200, { 'Cache-Control': 'no-cache' });
}
app.get('/', indexHtml);
app.get('/index.html', indexHtml);
...
app.use('/*', serveStatic({ root: './public', onFound: (_path, c) => { c.header('Cache-Control', 'no-cache'); } }));
// SPA fallback — return index.html for any unmatched path
app.get('/*', indexHtml);
```
`/sw.js` handler at ~line 1860 ends with `return c.text(src, 200, { ...no-store headers })`.
Bun provides `Bun.gzipSync(buf, { level: 9 })`. Brotli: `import { brotliCompressSync, constants } from 'node:zlib'`
(works in Bun). `createHash` is already imported from `node:crypto` at the top.
**Never compress** `/api/play`, `/api/media/*`, `/api/download/*` (Range/binary) —
they are not served by `serveStatic`, so a static-only middleware cannot touch them.
## Steps
1. `server/server.js` — add near the top imports:
`import { brotliCompressSync, constants as zlibConstants } from 'node:zlib';`
2. `server/server.js` — directly ABOVE `function indexHtml(c) {` add this helper block:
```js
// Compressed + ETagged static text. The shell is ~620 KB raw / ~160 KB gzip
// and every byte crosses the slow VPS→homelab link, so compress once per
// file content and keep it in memory. ETag = sha256 of the RAW bytes, so a
// `no-cache` revalidation costs a 304 instead of the whole file.
const COMPRESSIBLE = /\.(js|css|html|json|webmanifest|svg|txt)$/i;
const compressedCache = new Map(); // key -> { etag, raw, gz, br, type }
function compressedEntry(key, raw, type) {
let e = compressedCache.get(key);
const etag = '"' + createHash('sha256').update(raw).digest('hex').slice(0, 32) + '"';
if (e && e.etag === etag) return e;
e = {
etag, raw, type,
gz: Bun.gzipSync(raw, { level: 9 }),
br: brotliCompressSync(raw, { params: { [zlibConstants.BROTLI_PARAM_QUALITY]: 11 } }),
};
compressedCache.set(key, e);
return e;
}
function sendCompressed(c, e, cacheControl) {
const headers = { 'Content-Type': e.type, 'Cache-Control': cacheControl, ETag: e.etag, Vary: 'Accept-Encoding' };
const inm = c.req.header('if-none-match') || '';
if (inm.split(',').map((s) => s.trim()).includes(e.etag)) return new Response(null, { status: 304, headers });
const ae = c.req.header('accept-encoding') || '';
if (/\bbr\b/.test(ae)) return new Response(e.br, { headers: { ...headers, 'Content-Encoding': 'br' } });
if (/\bgzip\b/.test(ae)) return new Response(e.gz, { headers: { ...headers, 'Content-Encoding': 'gzip' } });
return new Response(e.raw, { headers });
}
const MIME = { js: 'text/javascript; charset=utf-8', css: 'text/css; charset=utf-8', html: 'text/html; charset=utf-8',
json: 'application/json', webmanifest: 'application/manifest+json', svg: 'image/svg+xml', txt: 'text/plain; charset=utf-8' };
```
3. `server/server.js` — change `indexHtml` so its return line becomes:
```js
const html = _indexSource.replace('__BUILD_TAG__', BUILD_TAG);
return sendCompressed(c, compressedEntry('index.html', Buffer.from(html), MIME.html), 'no-cache');
```
4. `server/server.js` — in the `/sw.js` handler, replace its final return, which is exactly:
```js
return c.text(src, 200, {
'Content-Type': 'application/javascript; charset=utf-8',
'Cache-Control': 'no-store, no-cache, must-revalidate',
});
```
with:
```js
return sendCompressed(c, compressedEntry('sw.js', Buffer.from(src), MIME.js), 'no-store, no-cache, must-revalidate');
```
(`text/javascript` is a valid service-worker MIME type.) Dry-run result: app.js 426 244 → 93 717 bytes (br).
5. `server/server.js` — directly BEFORE the `app.use('/*', serveStatic(...))` line add:
```js
app.get('/*', async (c, next) => {
const p = decodeURIComponent(new URL(c.req.url).pathname);
if (!COMPRESSIBLE.test(p) || p.includes('..') || p.startsWith('/api/')) return next();
const file = Bun.file('./public' + p);
if (!(await file.exists())) return next();
const raw = Buffer.from(await file.arrayBuffer());
const ext = p.slice(p.lastIndexOf('.') + 1).toLowerCase();
return sendCompressed(c, compressedEntry(p, raw, MIME[ext] || 'application/octet-stream'), 'no-cache');
});
```
(`/sw.js`, `/` and `/index.html` are registered earlier and win; this only covers the rest.)
## Out of scope / do NOT touch
- `frontend/sw.js`, `frontend/sw-update.js`: no changes. The SW fetches shell files with
`cache: 'reload'` and `?__ytpfresh=` — query strings don't affect the pathname match, fine.
- Do not change any `Cache-Control` value. Do not add `hono/compress` globally (it would
hit Range media responses).
- Do not touch `computeBuildTag`.
## Verification
```bash
cd /home/user/ytplayer/server && [ -e public ] || ln -s ../frontend public
PORT=3999 bun server.js > /tmp/ytp002.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip, br' http://localhost:3999/app.js | grep -iE 'content-encoding|etag|cache-control|vary'
ET=$(curl -s -D - -o /dev/null http://localhost:3999/app.js | grep -i '^etag' | cut -d' ' -f2 | tr -d '\r')
curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" http://localhost:3999/app.js
curl -s -H 'Accept-Encoding: gzip' --compressed http://localhost:3999/ | grep -c 'ytp-build'
curl -s -o /dev/null -D - -H 'Accept-Encoding: gzip' http://localhost:3999/sw.js | grep -iE 'content-encoding|cache-control'
curl -s --compressed http://localhost:3999/sw.js | grep -c "__BUILD_TAG__ !== 'undefined'" ; true
kill $SRV; true
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `content-encoding: br`, an `etag`, `cache-control: no-cache`, `vary: Accept-Encoding`;
the If-None-Match request prints `304`; index grep prints `1`; sw.js shows `content-encoding: gzip`
and its original no-store cache-control; the last grep prints `0` (tag was injected); tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines: surprises, deviations from the steps, anything
skipped and why.
Do not commit. Do not push. Do not touch files outside the Steps.

131
plans/queue/003-self-host-fonts-89466b.md Normal file
View File

@@ -0,0 +1,131 @@
---
id: 003-self-host-fonts-89466b
title: Self-host the three web fonts and drop the render-blocking Google Fonts CSS
created: 2026-09-29
depends_on: [002-compress-and-etag-shell-bd459c]
est_files: 5
---
# 003 — Self-host the three web fonts
## Objective
`frontend/index.html` loads a render-blocking stylesheet from `fonts.googleapis.com`
(3 families, 10 weights), which costs two extra origins (DNS+TLS each) before first
paint and only works offline through the SW's `ytplayer-fonts` runtime cache. After
this plan the fonts are files under `frontend/fonts/`, declared in
`frontend/fonts/fonts.css`, precached with the shell, and the display face is
preloaded. Visual result must be identical.
## Context the executor must NOT rediscover
`frontend/index.html:20-26` today:
```html
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600;12..96,700;12..96,800&family=Hanken+Grotesk:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="styles.css" />
```
CSS tokens (`frontend/styles.css:27-29`) reference the family names
`"Bricolage Grotesque"`, `"Hanken Grotesk"`, `"JetBrains Mono"` — keep those names.
`frontend/sw.js:55-71` — the `SHELL` array (precache list). Every new shell file must be
listed there or it will be missing offline. `frontend/sw.js:165` keeps a runtime rule for
the Google hosts — leave it (harmless, and old clients may still request them).
Google serves one variable `woff2` per family per subset when asked with a modern UA.
`fitLyricLines` already re-fits on `document.fonts` load (CLAUDE.md), so swap is safe.
## Steps
1. Create `scripts/fetch-fonts.js` (Node ≥18, no deps):
```js
// Downloads the app's Google fonts once (latin + latin-ext subsets) into
// frontend/fonts/ and writes frontend/fonts/fonts.css pointing at them.
// Re-run only when the font list changes.
const fs = require('node:fs');
const path = require('node:path');
const CSS_URL = 'https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,600..800&family=Hanken+Grotesk:wght@400..700&family=JetBrains+Mono:wght@400..700&display=swap';
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36';
const OUT = path.join(__dirname, '..', 'frontend', 'fonts');
(async () => {
fs.mkdirSync(OUT, { recursive: true });
const css = await (await fetch(CSS_URL, { headers: { 'User-Agent': UA } })).text();
// parts[k] ends with the "/* <subset> */" comment of the block in parts[k+1].
const parts = css.split('@font-face');
let out = '/* Generated by scripts/fetch-fonts.js — do not edit by hand. */\n';
let n = 0;
for (let k = 1; k < parts.length; k++) {
const subset = (parts[k - 1].match(/\/\*\s*([\w-]+)\s*\*\/\s*$/) || [])[1];
const b = parts[k].slice(0, parts[k].indexOf('}') + 1); // just this block
if (subset !== 'latin' && subset !== 'latin-ext') continue;
const url = (b.match(/url\((https:[^)]+\.woff2)\)/) || [])[1];
const fam = (b.match(/font-family:\s*'([^']+)'/) || [])[1];
if (!url || !fam) continue;
const file = `${fam.replace(/\s+/g, '')}-${subset}.woff2`;
const buf = Buffer.from(await (await fetch(url)).arrayBuffer());
fs.writeFileSync(path.join(OUT, file), buf);
out += '@font-face' + b.replace(url, file).trimEnd() + '\n';
n++;
}
fs.writeFileSync(path.join(OUT, 'fonts.css'), out);
console.log(`wrote ${n} font files + fonts.css`);
})().catch((e) => { console.error(e); process.exit(1); });
```
Run it: `node scripts/fetch-fonts.js`. Expect `wrote 6 font files + fonts.css`
(3 families × 2 subsets). If the network is blocked, STOP and report — do not hand-write fonts.
2. Open `frontend/fonts/fonts.css` and confirm every block has `font-display: swap;`; if a
block lacks it, add it.
3. `frontend/index.html` — replace the 5 lines from `<link rel="preconnect" href="https://fonts.googleapis.com" />`
through the closing `/>` of the Google stylesheet link with:
```html
<link rel="preload" href="fonts/BricolageGrotesque-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="preload" href="fonts/HankenGrotesk-latin.woff2" as="font" type="font/woff2" crossorigin />
<link rel="stylesheet" href="fonts/fonts.css" />
```
(Use the exact filenames the script produced — check with `ls frontend/fonts`.)
4. `frontend/sw.js` `SHELL` array — after `'/styles.css',` add `'/fonts/fonts.css',` and one
`'/fonts/<file>.woff2',` line per generated woff2 file.
5. `frontend/index.html` line 16 — the Content-Security-Policy `<meta>` has
`font-src https://fonts.gstatic.com data:` which would BLOCK self-hosted fonts. Change that part to
`font-src 'self' https://fonts.gstatic.com data:` (leave the rest of the policy unchanged;
`style-src` already allows `'self'`). `frontend/admin.html` has no Google Fonts link — leave it alone.
6. `package.json` scripts — add `"fetch-fonts": "node scripts/fetch-fonts.js"`.
## Out of scope / do NOT touch
- Do not rename the font families or edit `styles.css`.
- Do not remove the fonts rule in `sw.js` fetch handler or `UTILITY_CACHES`.
- Do not subset further (e.g. glyph-level subsetting) — out of scope.
## Verification
```bash
cd /home/user/ytplayer && ls -la frontend/fonts
grep -c 'href="https://fonts.googleapis' frontend/index.html
grep -c "font-src 'self'" frontend/index.html
for f in $(ls frontend/fonts/*.woff2); do grep -c "/fonts/$(basename $f)" frontend/sw.js; done
node --test frontend/*.test.js 2>&1 | tail -3
cd server && [ -e public ] || ln -s ../frontend public; PORT=3998 bun server.js >/tmp/ytp003.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://localhost:3998/fonts/fonts.css
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3998/fonts/$(ls ../frontend/fonts | grep woff2 | head -1)
kill $SRV; true
```
Expected: 6 `.woff2` + `fonts.css`; google link grep `0`; CSP grep `1`; each sw.js grep `1`; tests `fail 0`;
`200 text/css…` and `200`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` plus unified diff of text files (not the woff2 binaries).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

85
plans/queue/004-coalesce-stream-resolves-a92d40.md Normal file
View File

@@ -0,0 +1,85 @@
---
id: 004-coalesce-stream-resolves-a92d40
title: Coalesce concurrent resolveStreams calls for the same video
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 1
---
# 004 — Coalesce concurrent resolveStreams calls
## Objective
`resolveStreams(videoId)` checks `streamCache` but has no in-flight map, so two
requests for the same id that arrive before the first finishes (a warm-up + the
real play, two devices, the media cache's `getInfo` + `/api/streams`) each spawn a
~6 s `yt-dlp -J`. After this plan, concurrent callers share ONE promise; a failure
is not cached (the next call retries).
## Context the executor must NOT rediscover
`server/server.js:457-494`:
```js
async function resolveStreams(videoId) {
const now = Date.now();
const cached = streamCache.get(videoId);
if (cached && now < cached.expiresAt) return cached;
const out = await runYtdlpResilient(['-J', '--no-warnings', `https://www.youtube.com/watch?v=${videoId}`]);
const info = JSON.parse(out);
...
streamCache.set(videoId, entry);
return entry;
}
```
`streamCache` and `STREAM_CACHE_MAX` are declared just above it.
## Steps
1. `server/server.js` — rename the existing function `resolveStreams` to
`resolveStreamsUncached` (definition only; body unchanged).
2. Directly after that function add:
```js
// One yt-dlp -J per video at a time: concurrent callers (warm-up + play,
// two devices, the media cache's getInfo) share the in-flight promise.
const inflightStreams = new Map(); // videoId -> Promise<entry>
function resolveStreams(videoId) {
const cached = streamCache.get(videoId);
if (cached && Date.now() < cached.expiresAt) return Promise.resolve(cached);
let p = inflightStreams.get(videoId);
if (!p) {
p = resolveStreamsUncached(videoId).finally(() => inflightStreams.delete(videoId));
inflightStreams.set(videoId, p);
}
return p;
}
```
All existing callers keep calling `resolveStreams` (it still returns a Promise).
3. Create `server/streams-inflight.test.js`? — NO. Keep it in-file; instead verify with the
script below.
## Out of scope / do NOT touch
- Cache TTL logic, `runYtdlpResilient`, format filtering.
## Verification
```bash
cd /home/user/ytplayer/server && grep -n "function resolveStreams\|function resolveStreamsUncached\|inflightStreams" server.js
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
```
Expected: both functions + map present, `SERVER_OK`, all server test files `0 fail`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

111
plans/queue/005-warm-streams-on-intent-47b3d3.md Normal file
View File

@@ -0,0 +1,111 @@
---
id: 005-warm-streams-on-intent-47b3d3
title: Warm the stream cache for likely next plays
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40]
est_files: 2
---
# 005 — Warm the stream cache for likely next plays
## Objective
A cold `/api/streams` costs ~6 s of yt-dlp; a cached one ~0 s (1.2 s total on prod,
all network). Start that work before the user taps:
- new `GET /api/streams/warm?v=<id>` → `204` immediately, resolves in the background
(shares the in-flight promise from plan 004), at most 2 warm resolves at a time,
skipped when the server already holds a ready media copy.
- the client warms: the top 3 search results after a search renders; a card on
`pointerdown` (touch/mouse down fires ~100–300 ms before `click`); the next 2 queue
items when a song starts playing.
## Context the executor must NOT rediscover
- `server/server.js` `resolveStreams(videoId)` returns a Promise and dedupes (plan 004).
- `media.getReady(videoId)` (server/media-cache.js) resolves the ready row or null.
- `server/server.js:532` — `app.get('/api/streams', async (c) => {` — register the new
route directly ABOVE it (Hono matches `/api/streams/warm` separately anyway).
- Client `frontend/app.js`:
- `const YT_ID_RE = /^[A-Za-z0-9_-]{11}$/;` at ~line 1202 (declared later in the file
than the helper you add — fine, it is only read at call time).
- `runSearchQuery` success path (~line 8603): `searchResults = results; … renderList(); RecentSearches.cacheResults(q, results);`
- `renderCard(v, index, list)` at ~line 7725; it has `card.addEventListener('click', (e) => {` at ~line 7781.
- globals `queue` (array of video objects) and `queueIndex` (~line 353).
- master `playing` listener at ~line 2287 (`el.addEventListener('playing', () => { if (!masterIs(el)) return; …`).
- `WEB` constant is true for the PWA; `cachedIds` is a Set of ids saved on this device.
## Steps
1. `server/server.js` — above `app.get('/api/streams', …)` add:
```js
// GET /api/streams/warm?v=<id> — fire-and-forget: resolve streams into
// streamCache so the real /api/streams a moment later is instant. Bounded
// so a scrolling user can't queue dozens of yt-dlp processes.
const WARM_MAX = 2;
let warmActive = 0;
app.get('/api/streams/warm', async (c) => {
const id = (c.req.query('v') || '').trim();
if (!/^[A-Za-z0-9_-]{11}$/.test(id)) return c.body(null, 204);
if (warmActive >= WARM_MAX) return c.body(null, 204);
try { if (await media.getReady(id)) return c.body(null, 204); } catch { /* fall through */ }
warmActive++;
resolveStreams(id).catch(() => {}).finally(() => { warmActive--; });
return c.body(null, 204);
});
```
2. `frontend/app.js` — directly after the `const API = { … };` object (~line 320) add:
```js
// Pre-resolve streams for videos the user is likely to play next (server
// /api/streams/warm). Each id is warmed at most once per 20 min per tab.
const warmedAt = new Map();
function warmStreams(ids) {
if (!WEB || navigator.onLine === false) return;
const now = Date.now();
for (const id of ids) {
if (!id || !/^[A-Za-z0-9_-]{11}$/.test(id) || cachedIds.has(id)) continue;
if (now - (warmedAt.get(id) || 0) < 20 * 60_000) continue;
warmedAt.set(id, now);
fetch(`/api/streams/warm?v=${encodeURIComponent(id)}`, { priority: 'low' }).catch(() => {});
}
}
```
3. `frontend/app.js` `runSearchQuery` — after `RecentSearches.cacheResults(q, results);` add
`warmStreams(results.slice(0, 3).map((r) => r.id));`
4. `frontend/app.js` `renderCard` — directly BEFORE `card.addEventListener('click', (e) => {` add:
```js
card.addEventListener('pointerdown', () => warmStreams([v.id]), { passive: true });
```
5. `frontend/app.js` master `playing` listener — after `if (!masterIs(el)) return;` (and after
the plan-001 perf lines if present) add:
```js
if (Array.isArray(queue) && queueIndex >= 0) warmStreams(queue.slice(queueIndex + 1, queueIndex + 3).map((x) => x && x.id));
```
## Out of scope / do NOT touch
- Do not warm on hover/scroll, do not warm uploads (`upl_…`) — they need no yt-dlp.
- Do not change `/api/streams` itself.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js && echo APP_OK
cd server && bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
[ -e public ] || ln -s ../frontend public; PORT=3997 bun server.js >/tmp/ytp005.log 2>&1 & SRV=$!; sleep 4
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=bad'
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3997/api/streams/warm?v=dQw4w9WgXcQ'
kill $SRV; true
cd .. && node --test frontend/*.test.js 2>&1 | tail -3
```
Expected: `APP_OK`, `SERVER_OK`, `204`, `204`, tests `fail 0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

212
plans/queue/006-innertube-search-48066b.md Normal file
View File

@@ -0,0 +1,212 @@
---
id: 006-innertube-search-48066b
title: Answer searches from YouTube InnerTube directly with yt-dlp fallback
created: 2026-09-29
depends_on: [001-perf-timing-marks-105acc]
est_files: 4
---
# 006 — InnerTube search with yt-dlp fallback
## Objective
`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process
(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search
API returns the same data in a few hundred ms. After this plan `/api/search` tries
InnerTube first (6 s timeout), maps results to the existing card shape, and falls
back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos.
Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri
bridge contract are untouched.
## Context the executor must NOT rediscover
A trimmed real response is committed at `server/fixtures/innertube-search.json`
(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`,
and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29):
```
contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
.itemSectionRenderer.contents[].videoRenderer:
videoId -> id
title.runs[0].text -> title
ownerText.runs[0].text -> channel
ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId -> channelId (UC…)
ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl -> '/@handle' or '/channel/UC…'
lengthText.simpleText "5:43" | "1:59:47" | absent (live) -> duration seconds (0 if absent)
```
Request that works (no key needed):
```
POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
Content-Type: application/json
{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}
```
The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable.
Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first
cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the
fallback is for. Do not "fix" the 403 by adding cookies/keys.
Existing card shape — `server/server.js:293-305` `slimEntry`:
```js
return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };
```
`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`.
Current route `server/server.js:365-395` (inside it):
```js
let mine = [];
try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
try {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
const results = [...mine, ...parseCards(out)];
```
Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script
runs each file separately joined by `&&`.
## Steps
1. Create `server/innertube.js`:
```js
/* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
* parseSearch() is pure (tested against fixtures/innertube-search.json);
* search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
export function lengthToSeconds(s) {
if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
}
export function parseSearch(json) {
const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
?.sectionListRenderer?.contents;
if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
const out = [];
for (const s of sections) {
for (const it of s?.itemSectionRenderer?.contents || []) {
const v = it && it.videoRenderer;
if (!v || typeof v.videoId !== 'string') continue;
const owner = v.ownerText?.runs?.[0] || {};
const be = owner.navigationEndpoint?.browseEndpoint || {};
const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
out.push({
id: v.videoId,
title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
channel: owner.text || '',
channelId: be.browseId || '',
channelUrl: path ? `https://www.youtube.com${path}` : '',
duration: lengthToSeconds(v.lengthText?.simpleText),
thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
});
}
}
return out;
}
export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ context: { client: CLIENT }, query: q }),
signal: AbortSignal.timeout(timeoutMs),
});
if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
return parseSearch(await res.json());
}
```
2. Create `server/innertube.test.js`:
```js
import { test, expect } from 'bun:test';
import { readFileSync } from 'node:fs';
import { parseSearch, lengthToSeconds, search } from './innertube.js';
const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
test('parses video renderers into slim cards', () => {
const r = parseSearch(fixture);
expect(r.length).toBe(4);
expect(r[0]).toEqual({
id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
});
expect(r[1].duration).toBe(7187);
expect(r[3].duration).toBe(0); // live, no lengthText
});
test('lengthToSeconds', () => {
expect(lengthToSeconds('5:43')).toBe(343);
expect(lengthToSeconds('1:59:47')).toBe(7187);
expect(lengthToSeconds('LIVE')).toBe(0);
expect(lengthToSeconds(undefined)).toBe(0);
});
test('unexpected shape throws (caller falls back to yt-dlp)', () => {
expect(() => parseSearch({})).toThrow();
});
test('search() throws on HTTP error', async () => {
const fetchImpl = async () => new Response('no', { status: 429 });
await expect(search('x', { fetchImpl })).rejects.toThrow('429');
});
```
If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's
`lengthText` and use that value (the fixture is the source of truth).
3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`.
4. `server/server.js` — add import next to the other local imports:
`import * as innertube from './innertube.js';`
5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];`
head with:
```js
try {
let yt = [];
try {
yt = await innertube.search(q);
} catch (e) {
console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
}
if (!yt.length) {
const out = await runYtdlpResilient([
`ytsearch${SEARCH_LIMIT}:${q}`,
'--dump-json', '--flat-playlist',
'--no-warnings', '--ignore-errors',
]);
yt = parseCards(out);
}
const results = [...mine, ...yt];
```
Everything after (`searchCache.set`, return, catch) stays as is.
6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')`
skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented
line `# SEARCH_INNERTUBE: "0" # force yt-dlp search` next to the other commented env vars.
## Out of scope / do NOT touch
- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging.
- Card shape, `searchCache`, `app.js`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
kill $SRV; true
grep -c "innertube failed" /tmp/ytp006.log
```
Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns
`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container
has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

338
plans/queue/007-ytdlp-worker-045800.md Normal file
View File

@@ -0,0 +1,338 @@
---
id: 007-ytdlp-worker-045800
title: Keep one long-lived yt-dlp worker process instead of spawning per call
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b]
est_files: 5
---
# 007 — Long-lived yt-dlp worker pool
## Objective
Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with
yt-dlp 2026.08.19 (same host, same network):
| call | per-call spawn | warm worker |
|------|----------------|-------------|
| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s |
| `-J <video>` | 3.32 s | 2.29 s |
After this plan, the "read-only" yt-dlp calls (search fallback, channel, playlist
expand, `-J` stream resolve) go to a pool of 2 long-lived Python workers that import
`yt_dlp` once. Downloads (which write files and use an AbortSignal) keep spawning.
Any pool infrastructure problem falls back to spawning transparently. A bot-check answer
recycles the workers and retries that call with a fresh spawn; each worker is replaced after
100 requests. `YTDLP_WORKER=0` disables the pool.
Dry-run notes (2026-09-29): the worker calls yt-dlp's own CLI entry (`yt_dlp._real_main`) — a
first version using `parse_options()` + `YoutubeDL` directly behaved differently from the CLI.
During testing the container's IP got bot-checked by YouTube for plain CLI calls too, so the
pooled `-J` path could only be measured before that (2.29 s vs 3.32 s). Measure on prod with
plan 001's `[ytdlp] pooled …` log lines and keep `YTDLP_WORKER=0` as the escape hatch.
## Context the executor must NOT rediscover
- In Docker, yt-dlp is the release **zipapp** at `/usr/local/bin/yt-dlp`; Python can import
`yt_dlp` from it by putting that path on `sys.path`. `python3` is installed in the image
(Dockerfile apt line). `/etc/yt-dlp.conf` (the `--js-runtimes bun:…` line) is honoured
because `yt_dlp.parse_options()` reads config files like the CLI does.
- The Dockerfile copies `server/` to `/app`, so a file at `server/ytdlp-worker.py` ships
automatically.
- `server/server.js:143` — `function runYtdlp(args, { signal } = {})` spawns yt-dlp
(after plan 001 it also logs `[ytdlp] <kind> <ms>ms ok|fail`).
- `server/server.js:189` — `async function runYtdlpResilient(args, opts = {})` calls
`runYtdlp(withCookies(args), opts)` and, on a bot check, retries with
`runYtdlp(withCookies(['--extractor-args', …, ...args]), opts)`. `opts` is passed through.
- Read-only call sites to mark `{ pooled: true }` (line numbers from before plans 004/006;
search by the text):
1. `/api/search` fallback: `runYtdlpResilient([\`ytsearch${SEARCH_LIMIT}:${q}\`, …])`
2. `/api/channel`: `runYtdlpResilient([` … `'--playlist-end', String(CHANNEL_LIMIT),`
3. `resolveStreamsUncached`: `runYtdlpResilient(['-J', '--no-warnings', \`https://www.youtube.com/watch?v=${videoId}\`])`
4. `/api/playlist/expand` (~line 1712): `const out = await runYtdlpResilient([`
Do NOT mark the calls at ~861, ~932, ~988 (downloads) or `server/notes.js:362`.
- `const YTDLP = process.env.YTDLP_PATH || 'yt-dlp';` near the top of server.js.
- Expected noise on a machine WITHOUT yt-dlp (local dev before `npm run setup`): three
`ModuleNotFoundError: No module named 'yt_dlp'` lines, then
`[ytdlp-pool] disabled after 3 failed starts …` — the server keeps working via spawn. Not a bug.
## Steps
1. Create `server/ytdlp-worker.py` with exactly:
```python
#!/usr/bin/env python3
"""ytdlp-worker.py - one long-lived yt-dlp process answering many requests.
Spawning yt-dlp per call pays Python start-up + extractor import every time.
This worker imports yt_dlp ONCE and runs each request's argv in-process
through the CLI's own entry point (yt_dlp._real_main), so config files such
as /etc/yt-dlp.conf and everything else the command line sets up apply.
Protocol (newline-delimited JSON):
stdin : {"id": <int>, "args": [<yt-dlp argv>...]}
stdout: {"id": <int>, "code": <int>, "out": "<captured stdout>", "err": "<stderr tail>"}
Only for calls whose result is printed to stdout (-J, --dump-json, --version).
Usage: python3 ytdlp-worker.py <path-to-yt-dlp-zipapp-or-empty>
"""
import io, json, sys, contextlib
if len(sys.argv) > 1 and sys.argv[1]:
sys.path.insert(0, sys.argv[1]) # the yt-dlp release binary is a zipapp
import yt_dlp # noqa: E402
proto_out = sys.stdout
sys.stdout = io.StringIO() # nothing may leak onto the protocol pipe
def run(args):
out, err = io.StringIO(), io.StringIO()
code = 0
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
try:
# The CLI's own entry point (not parse_options + YoutubeDL): it also
# sets up what the command line does (JS challenge solving, plugins,
# post-processing defaults). A bare YoutubeDL got "Sign in to
# confirm you're not a bot" where the CLI succeeded.
ret = yt_dlp._real_main(args)
code = ret[0] if isinstance(ret, tuple) else (ret or 0)
except SystemExit as e:
code = e.code if isinstance(e.code, int) else 1
except Exception as e: # report, never die
err.write(f"ERROR: {e}\n")
code = 1
return code, out.getvalue(), err.getvalue()[-8000:]
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
code, out, err = run([str(a) for a in req.get("args", [])])
resp = {"id": req.get("id"), "code": code, "out": out, "err": err}
except Exception as e:
resp = {"id": None, "code": 1, "out": "", "err": f"worker: {e}"}
proto_out.write(json.dumps(resp) + "\n")
proto_out.flush()
```
2. Create `server/ytdlp-pool.js` with exactly:
```js
/* ytdlp-pool.js — a small pool of long-lived yt-dlp workers (ytdlp-worker.py).
*
* run(args) resolves stdout exactly like a spawned `yt-dlp <args>` would, or
* rejects with the captured stderr. Each worker handles one request at a time;
* extra requests queue. A worker that exits or exceeds the timeout is killed
* and replaced. Errors carrying `poolInfra: true` mean "the pool could not run
* this" — the caller falls back to a normal per-call spawn. Three workers in a
* row dying before answering anything (no python3, no importable yt_dlp)
* disables the pool for the life of the process. Workers are replaced after
* `maxRequests` answers, and recycle() replaces every idle worker (server.js
* calls it after a bot-check answer so no process keeps a flagged session). */
import { spawn } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const WORKER = join(dirname(fileURLToPath(import.meta.url)), 'ytdlp-worker.py');
const infra = (msg) => Object.assign(new Error(msg), { poolInfra: true });
export function createYtdlpPool({ ytdlpPath = '', size = 2, timeoutMs = 60_000, maxRequests = 100, python = 'python3', log = console } = {}) {
const workers = [];
const queue = [];
let nextId = 1;
let startFailures = 0;
let disabled = false;
let closed = false;
function startWorker() {
const child = spawn(python, [WORKER, ytdlpPath], { stdio: ['pipe', 'pipe', 'inherit'] });
const w = { child, busy: null, buf: '', dead: false, retiring: false, served: 0 };
child.stdout.setEncoding('utf8');
child.stdout.on('data', (d) => {
w.buf += d;
let nl;
while ((nl = w.buf.indexOf('\n')) >= 0) {
const line = w.buf.slice(0, nl);
w.buf = w.buf.slice(nl + 1);
let msg;
try { msg = JSON.parse(line); } catch { continue; }
const job = w.busy;
if (!job || msg.id !== job.id) continue;
clearTimeout(job.timer);
w.busy = null;
w.served++;
startFailures = 0;
if (msg.code === 0) job.resolve(msg.out);
else job.reject(new Error((msg.err || '').trim() || 'yt-dlp exited with code ' + msg.code));
if (w.served >= maxRequests) retire(w);
pump();
}
});
const onDead = (why) => {
if (w.dead) return;
w.dead = true;
const i = workers.indexOf(w);
if (i >= 0) workers.splice(i, 1);
if (w.busy) { clearTimeout(w.busy.timer); w.busy.reject(infra('yt-dlp worker ' + why)); w.busy = null; }
if (closed) return;
if (!w.served && ++startFailures >= 3) {
disabled = true;
log.warn?.(`[ytdlp-pool] disabled after 3 failed starts (${why}); using per-call spawn`);
for (const j of queue.splice(0)) j.reject(infra('pool disabled'));
return;
}
if (!w.retiring) log.warn?.(`[ytdlp-pool] worker ${why}; restarting`);
const t = setTimeout(() => { if (!closed && !disabled) { workers.push(startWorker()); pump(); } }, 1000);
t.unref?.();
};
child.on('exit', (code) => onDead('exited (' + code + ')'));
child.on('error', (e) => onDead('failed to start: ' + e.message));
return w;
}
// Replace a worker once it is idle (its exit handler starts a fresh one).
function retire(w) {
if (w.retiring || w.dead) return;
w.retiring = true;
try { w.child.kill(); } catch { /* gone */ }
}
for (let i = 0; i < size; i++) workers.push(startWorker());
function pump() {
for (const w of workers) {
if (w.dead || w.retiring || w.busy || !queue.length) continue;
const job = queue.shift();
w.busy = job;
job.timer = setTimeout(() => { try { w.child.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
try { w.child.stdin.write(JSON.stringify({ id: job.id, args: job.args }) + '\n'); }
catch (e) { clearTimeout(job.timer); w.busy = null; job.reject(infra(e.message)); }
}
}
function run(args) {
if (closed || disabled) return Promise.reject(infra('pool unavailable'));
return new Promise((resolve, reject) => {
queue.push({ id: nextId++, args: args.map(String), resolve, reject });
pump();
});
}
function close() {
closed = true;
for (const w of workers) { try { w.child.kill(); } catch { /* gone */ } }
for (const j of queue.splice(0)) j.reject(infra('pool closed'));
}
function recycle() {
for (const w of workers) if (!w.busy) retire(w);
}
return { run, close, recycle, get disabled() { return disabled; } };
}
```
3. Create `server/ytdlp-pool.test.js` with exactly:
```js
import { test, expect } from 'bun:test';
import { existsSync } from 'node:fs';
import { createYtdlpPool } from './ytdlp-pool.js';
// The worker imports yt_dlp from the release zipapp. Skip when none is around.
const YTDLP = [process.env.YTDLP_PATH, '../bin/yt-dlp', Bun.which('yt-dlp')].find((p) => p && existsSync(p)) || '';
const quiet = { warn() {}, info() {} };
test.skipIf(!YTDLP)('runs requests and reports yt-dlp errors without dying', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, log: quiet });
try {
const v = await pool.run(['--version']);
expect(v.trim()).toMatch(/^\d{4}\.\d{2}\.\d{2}/);
const err = await pool.run(['--definitely-not-a-flag']).catch((e) => e);
expect(err).toBeInstanceOf(Error);
expect(err.poolInfra).toBeFalsy();
expect((await pool.run(['--version'])).trim()).toBe(v.trim());
} finally { pool.close(); }
}, 30000);
test('a broken python disables the pool and rejects as infra', async () => {
const pool = createYtdlpPool({ python: '/nonexistent/python3', size: 1, log: quiet });
const err = await pool.run(['--version']).catch((e) => e);
expect(err.poolInfra).toBe(true);
await new Promise((r) => setTimeout(r, 3500));
expect(pool.disabled).toBe(true);
expect((await pool.run(['--version']).catch((e) => e)).poolInfra).toBe(true);
pool.close();
}, 15000);
test.skipIf(!YTDLP)('workers are replaced after maxRequests and by recycle(), without failing requests', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, maxRequests: 1, log: quiet });
try {
const a = await pool.run(['--version']);
const b = await pool.run(['--version']); // served by a fresh worker
expect(b).toBe(a);
pool.recycle();
expect(await pool.run(['--version'])).toBe(a);
expect(pool.disabled).toBe(false);
} finally { pool.close(); }
}, 30000);
```
4. `server/package.json` "test" script — append ` && bun test ./ytdlp-pool.test.js`.
5. `server/server.js`:
a. Add import: `import { createYtdlpPool } from './ytdlp-pool.js';`
b. Rename the existing `function runYtdlp(args, { signal } = {})` to
`function runYtdlpSpawn(args, { signal } = {})` (body unchanged).
c. Directly after it add:
```js
// Read-only calls (-J, --dump-json) go to long-lived workers that import
// yt_dlp once (~1 s saved per call). Downloads keep spawning. Pool trouble
// (not a yt-dlp error) falls back to a spawn. YTDLP_WORKER=0 disables it.
const ytdlpPool = process.env.YTDLP_WORKER === '0' ? null : createYtdlpPool({
ytdlpPath: Bun.which(YTDLP) || YTDLP,
size: Math.max(1, Number(process.env.YTDLP_WORKERS) || 2),
});
function runYtdlp(args, opts = {}) {
if (!opts.pooled || !ytdlpPool || opts.signal) return runYtdlpSpawn(args, opts);
const t0 = Date.now();
return ytdlpPool.run(args).then(
(out) => { console.log(`[ytdlp] pooled ${Date.now() - t0}ms ok`); return out; },
(err) => {
if (err.poolInfra) return runYtdlpSpawn(args, opts);
// A bot check can stick to a long-lived process: replace the workers and
// answer this call the old way, from a fresh process.
if (BOT_CHECK_RE.test(err.message)) { ytdlpPool.recycle(); return runYtdlpSpawn(args, opts); }
console.log(`[ytdlp] pooled ${Date.now() - t0}ms fail`);
throw err;
},
);
}
```
d. At the 4 read-only call sites listed in Context, add a second argument `{ pooled: true }`
to `runYtdlpResilient(...)`. Example: `runYtdlpResilient(['-J', '--no-warnings', url], { pooled: true })`.
6. `docker-compose.yml` — next to the commented yt-dlp env lines add:
`# YTDLP_WORKER: "0" # disable the long-lived yt-dlp worker pool`
`# YTDLP_WORKERS: "2" # pool size`
## Out of scope / do NOT touch
- Download paths (~861, ~932, ~988), `server/notes.js`, `withSaveSlot`, fallback-client logic.
- Dockerfile (python3 is already installed; the worker file ships with `server/`).
## Verification
```bash
cd /home/user/ytplayer && npm run setup >/dev/null 2>&1; ls -la bin/yt-dlp
cd server && bun test ./ytdlp-pool.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail|skip)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "pooled: true" server.js
```
Expected: pool tests `3 pass` (or `1 pass 2 skip` if `bin/yt-dlp` could not be downloaded —
say so in Findings); every file `0 fail`; `SERVER_OK`; grep prints `4`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

427
plans/queue/008-p2p-schema-and-config-127966.md Normal file
View File

@@ -0,0 +1,427 @@
---
id: 008-p2p-schema-and-config-127966
title: Add P2P tables, config flags and db helpers
created: 2026-09-29
depends_on: []
est_files: 5
---
# 008 — P2P tables, config flags and db helpers
## Objective
Lay the storage foundation for peer-to-peer sharing described in
`docs/p2p-architecture.md` (READ IT FIRST — it is short). After this plan:
- `server/p2p-config.js` exports `P2P` (config) and `loadP2pConfig(env)`.
Defaults: **P2P enabled**, **malware scan disabled**, stale after 7 days.
- `server/p2p-db.js` creates `p2p_content`, `p2p_devices`, `p2p_holders`,
`video_views` and adds `media_cache.sha256`, with query helpers.
- The server calls `initP2pSchema()` at boot. Nothing else changes behaviour yet.
## Context the executor must NOT rediscover
- `server/db.js` exports `db` (libsql client) and `initDb()`. Its `MEDIA_COLS` set
(~line 308) whitelists columns `upsertMedia()` may write:
```js
const MEDIA_COLS = new Set([
'status', 'gen', 'size', 'height', 'vcodec', 'acodec', 'duration', 'optimized',
'meta', 'priority', 'auto', 'attempts', 'error', 'retry_at', 'created_at',
'updated_at', 'last_access', 'hits',
]);
```
- `server/server.js` `main()` (~line 1916): `await initDb();` then `await media.init();`.
- Server test conventions: `bun:test`, a temp `DB_PATH` set BEFORE importing `./db.js`
(see `server/notes.test.js:1-12`), and each file runs in its own `bun test` process via the
`server/package.json` "test" script (db.js is a singleton).
## Steps
1. Create `server/p2p-config.js` with exactly:
```js
/* p2p-config.js — peer-to-peer settings (see docs/p2p-architecture.md).
* P2P is ON unless P2P_ENABLED=0. The malware scan is OFF unless
* P2P_MALWARE_SCAN=1. Hashing + validateMedia are never optional. */
import { dirname, join } from 'node:path';
const num = (v, d) => (Number.isFinite(Number(v)) && String(v).trim() !== '' ? Number(v) : d);
export function loadP2pConfig(env = process.env) {
const dbDir = dirname(env.DB_PATH || './data/ytplayer.db');
return {
enabled: env.P2P_ENABLED !== '0',
malwareScan: env.P2P_MALWARE_SCAN === '1',
scanCmd: (env.P2P_SCAN_CMD || 'clamscan --no-summary --infected').trim(),
staleDays: num(env.P2P_STALE_DAYS, 7),
keepMinViews: num(env.P2P_KEEP_MIN_VIEWS, 3),
keepDays: num(env.P2P_KEEP_DAYS, 30),
keepRecentDays: num(env.P2P_KEEP_RECENT_DAYS, 14),
intakeDir: env.P2P_INTAKE_DIR || join(dbDir, 'p2p-intake'),
intakeMaxBytes: num(env.P2P_INTAKE_MAX_BYTES, 3 * 1024 ** 3),
};
}
export const P2P = loadP2pConfig();
```
2. Create `server/p2p-db.js` — copy VERBATIM from the "p2p-db.js" appendix at the end of this plan.
3. Create `server/p2p-db.test.js` — copy VERBATIM from the "p2p-db.test.js" appendix.
4. `server/db.js` — add `'sha256'` to the end of `MEDIA_COLS` (after `'hits'`).
5. `server/package.json` "test" script — append ` && bun test ./p2p-db.test.js`.
6. `server/server.js` — add `import { initP2pSchema } from './p2p-db.js';` next to the other
local imports, and in `main()` directly after `await initDb();` add `await initP2pSchema();`.
7. `docker-compose.yml` — in the `ytplayer` service `environment:` block, after the
`LYRICS_WORKER_TOKEN` line, add:
```yaml
# Peer-to-peer sharing (docs/p2p-architecture.md). ON by default.
P2P_ENABLED: "${P2P_ENABLED:-1}"
# Malware scan before a file's hash is admitted. OFF by default; needs an
# image built with INSTALL_CLAMAV=1. Hashing + media validation always run.
P2P_MALWARE_SCAN: "${P2P_MALWARE_SCAN:-0}"
# P2P_STALE_DAYS: "7" # holder shown as stale after this many days unchecked
# P2P_KEEP_MIN_VIEWS: "3" # server keeps copies with ≥ this many views…
# P2P_KEEP_DAYS: "30" # …in this many days
# P2P_KEEP_RECENT_DAYS: "14" # …or played this recently
```
## Out of scope / do NOT touch
- No routes, no media-cache changes, no frontend changes (later plans).
- Do not edit `initDb()`'s SQL; the new column is added by `initP2pSchema()`.
## Verification
```bash
cd /home/user/ytplayer/server && bun test ./p2p-db.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
```
Expected: `4 pass 0 fail`; every file `0 fail`; `SERVER_OK`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files in full).
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — p2p-db.js
```js
/* ============================================================================
* p2p-db.js — tables + queries for peer-to-peer sharing
* (docs/p2p-architecture.md). Shares the libsql client from db.js.
*
* p2p_content one row per verified file (cid = sha256 of the bytes); never
* deleted, only revoked — the catalog grows over time
* p2p_devices registered devices (secret stored as sha256)
* p2p_holders which device holds which cid; PERSISTENT (no TTL) with
* last_verified_at — the UI decides what is "stale"
* video_views per-video per-day view counts (retention criteria)
* All timestamps are ms epochs.
* ========================================================================== */
import { db } from './db.js';
export async function initP2pSchema() {
await db.executeMultiple(`
CREATE TABLE IF NOT EXISTS p2p_content (
cid TEXT PRIMARY KEY,
video_id TEXT NOT NULL,
size INTEGER NOT NULL,
height INTEGER NOT NULL DEFAULT 0,
vcodec TEXT,
acodec TEXT,
duration REAL NOT NULL DEFAULT 0,
meta TEXT NOT NULL DEFAULT '{}',
origin TEXT NOT NULL, -- server | intake
status TEXT NOT NULL DEFAULT 'verified', -- verified | revoked
scan TEXT NOT NULL DEFAULT 'skipped', -- skipped | clean
created_at INTEGER NOT NULL,
verified_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_p2p_content_video ON p2p_content (video_id, created_at DESC);
CREATE TABLE IF NOT EXISTS p2p_devices (
device_id TEXT PRIMARY KEY,
secret_hash TEXT NOT NULL,
fingerprint TEXT,
profile TEXT,
share INTEGER NOT NULL DEFAULT 1,
created_at INTEGER NOT NULL,
last_seen_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS p2p_holders (
cid TEXT NOT NULL,
device_id TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active', -- active | removed
trust TEXT NOT NULL DEFAULT 'reported', -- reported | challenged
first_reported_at INTEGER NOT NULL,
last_verified_at INTEGER NOT NULL,
removed_at INTEGER,
PRIMARY KEY (cid, device_id)
);
CREATE INDEX IF NOT EXISTS idx_p2p_holders_device ON p2p_holders (device_id, status);
CREATE TABLE IF NOT EXISTS video_views (
video_id TEXT NOT NULL,
day TEXT NOT NULL,
n INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (video_id, day)
);
`);
// media_cache.sha256 — the cid of the current <id>.<gen>.mp4 (plan 009).
try { await db.execute('ALTER TABLE media_cache ADD COLUMN sha256 TEXT'); }
catch (e) { if (!/duplicate column/i.test(String(e.message))) throw e; }
}
const rowsOf = (r) => r.rows.map((row) => {
const o = {};
r.columns.forEach((c, i) => { o[c] = row[i]; });
return o;
});
// ---- content ----------------------------------------------------------------
export async function upsertContent(c) {
await db.execute({
sql: `INSERT INTO p2p_content (cid, video_id, size, height, vcodec, acodec, duration, meta, origin, status, scan, created_at, verified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'verified', ?, ?, ?)
ON CONFLICT(cid) DO UPDATE SET verified_at = excluded.verified_at, scan = excluded.scan`,
args: [c.cid, c.videoId, c.size, c.height || 0, c.vcodec || null, c.acodec || null, c.duration || 0,
JSON.stringify(c.meta || {}), c.origin, c.scan || 'skipped', c.now, c.now],
});
}
export async function getContent(cid) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_content WHERE cid = ?', args: [cid] });
return rowsOf(r)[0] || null;
}
export async function listContentForVideo(videoId) {
const r = await db.execute({
sql: "SELECT * FROM p2p_content WHERE video_id = ? AND status = 'verified' ORDER BY created_at DESC LIMIT 10",
args: [videoId],
});
return rowsOf(r);
}
export async function knownCids(cids) {
if (!cids.length) return new Set();
const out = new Set();
for (let i = 0; i < cids.length; i += 200) {
const part = cids.slice(i, i + 200);
const r = await db.execute({
sql: `SELECT cid FROM p2p_content WHERE status = 'verified' AND cid IN (${part.map(() => '?').join(',')})`,
args: part,
});
for (const row of r.rows) out.add(row[0]);
}
return out;
}
export async function revokeContent(cid) {
await db.execute({ sql: "UPDATE p2p_content SET status = 'revoked' WHERE cid = ?", args: [cid] });
}
// ---- devices ------------------------------------------------------------------
export async function createDevice({ deviceId, secretHash, fingerprint, profile, now }) {
await db.execute({
sql: `INSERT INTO p2p_devices (device_id, secret_hash, fingerprint, profile, share, created_at, last_seen_at)
VALUES (?, ?, ?, ?, 1, ?, ?)`,
args: [deviceId, secretHash, fingerprint || null, profile || null, now, now],
});
}
export async function getDevice(deviceId) {
const r = await db.execute({ sql: 'SELECT * FROM p2p_devices WHERE device_id = ?', args: [deviceId] });
return rowsOf(r)[0] || null;
}
export async function touchDevice(deviceId, { now, share, profile } = {}) {
await db.execute({
sql: `UPDATE p2p_devices SET last_seen_at = ?,
share = COALESCE(?, share), profile = COALESCE(?, profile)
WHERE device_id = ?`,
args: [now, share === undefined ? null : (share ? 1 : 0), profile || null, deviceId],
});
}
// ---- holders (persistent; never expired by time) ------------------------------
export async function upsertHolder({ cid, deviceId, trust = 'reported', now }) {
await db.execute({
sql: `INSERT INTO p2p_holders (cid, device_id, status, trust, first_reported_at, last_verified_at)
VALUES (?, ?, 'active', ?, ?, ?)
ON CONFLICT(cid, device_id) DO UPDATE SET
status = 'active', removed_at = NULL, last_verified_at = excluded.last_verified_at,
trust = CASE WHEN p2p_holders.trust = 'challenged' OR excluded.trust = 'challenged'
THEN 'challenged' ELSE 'reported' END`,
args: [cid, deviceId, trust, now, now],
});
}
export async function setHolderTrust({ cid, deviceId, trust, now }) {
await db.execute({
sql: 'UPDATE p2p_holders SET trust = ?, last_verified_at = ? WHERE cid = ? AND device_id = ?',
args: [trust, now, cid, deviceId],
});
}
export async function removeHolder({ cid, deviceId, now }) {
await db.execute({
sql: "UPDATE p2p_holders SET status = 'removed', removed_at = ? WHERE cid = ? AND device_id = ? AND status = 'active'",
args: [now, cid, deviceId],
});
}
// A full report: every active holding of this device NOT in `keep` is removed.
export async function removeHoldersExcept({ deviceId, keep, now }) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
const keepSet = new Set(keep);
let removed = 0;
for (const row of r.rows) {
if (keepSet.has(row[0])) continue;
await removeHolder({ cid: row[0], deviceId, now });
removed++;
}
return removed;
}
export async function activeHoldingsOf(deviceId) {
const r = await db.execute({
sql: "SELECT cid FROM p2p_holders WHERE device_id = ? AND status = 'active'",
args: [deviceId],
});
return r.rows.map((row) => row[0]);
}
// Holders of one cid, joined with the device's share flag. Newest check first.
export async function listHolders(cid, limit = 50) {
const r = await db.execute({
sql: `SELECT h.device_id, h.trust, h.first_reported_at, h.last_verified_at, d.share
FROM p2p_holders h JOIN p2p_devices d ON d.device_id = h.device_id
WHERE h.cid = ? AND h.status = 'active'
ORDER BY h.last_verified_at DESC LIMIT ?`,
args: [cid, limit],
});
return rowsOf(r);
}
// ---- views + stats --------------------------------------------------------------
export function dayKey(ms) {
return new Date(ms).toISOString().slice(0, 10);
}
export async function addView(videoId, now) {
await db.execute({
sql: `INSERT INTO video_views (video_id, day, n) VALUES (?, ?, 1)
ON CONFLICT(video_id, day) DO UPDATE SET n = n + 1`,
args: [videoId, dayKey(now)],
});
}
export async function viewsSince(videoId, sinceMs) {
const r = await db.execute({
sql: 'SELECT COALESCE(SUM(n), 0) FROM video_views WHERE video_id = ? AND day >= ?',
args: [videoId, dayKey(sinceMs)],
});
return Number(r.rows[0][0]) || 0;
}
export async function p2pStats() {
const one = async (sql) => Number((await db.execute(sql)).rows[0][0]) || 0;
return {
content: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'verified'"),
revoked: await one("SELECT COUNT(*) FROM p2p_content WHERE status = 'revoked'"),
devices: await one('SELECT COUNT(*) FROM p2p_devices'),
holders: await one("SELECT COUNT(*) FROM p2p_holders WHERE status = 'active'"),
heldCids: await one("SELECT COUNT(DISTINCT cid) FROM p2p_holders WHERE status = 'active'"),
};
}
```
## Appendix — p2p-db.test.js
```js
// P2P tables against a real temp libsql DB (docs/p2p-architecture.md).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const { loadP2pConfig } = await import('./p2p-config.js');
const CID = 'a'.repeat(64);
const CID2 = 'b'.repeat(64);
const T0 = Date.UTC(2026, 8, 29, 12);
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
await P.initP2pSchema(); // idempotent (ALTER TABLE duplicate column is ignored)
});
test('config defaults: P2P on, malware scan off', () => {
const c = loadP2pConfig({});
expect(c.enabled).toBe(true);
expect(c.malwareScan).toBe(false);
expect(c.staleDays).toBe(7);
expect(loadP2pConfig({ P2P_ENABLED: '0', P2P_MALWARE_SCAN: '1' })).toMatchObject({ enabled: false, malwareScan: true });
});
test('content upsert/get/known/revoke', async () => {
await P.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: 1000, height: 720, vcodec: 'h264', acodec: 'aac', duration: 212, meta: { title: 'x' }, origin: 'server', now: T0 });
const c = await P.getContent(CID);
expect(c.video_id).toBe('dQw4w9WgXcQ');
expect(c.status).toBe('verified');
expect([...(await P.knownCids([CID, CID2]))]).toEqual([CID]);
expect((await P.listContentForVideo('dQw4w9WgXcQ')).length).toBe(1);
await P.upsertContent({ cid: CID2, videoId: 'dQw4w9WgXcQ', size: 5, origin: 'intake', now: T0 });
await P.revokeContent(CID2);
expect([...(await P.knownCids([CID2]))]).toEqual([]);
});
test('holders persist, never expire, and a full report removes missing ones', async () => {
await P.createDevice({ deviceId: 'dev_1', secretHash: 'h', fingerprint: 'fp', now: T0 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 });
// 90 days later with no new report: still listed (UI marks it stale).
let hs = await P.listHolders(CID);
expect(hs.length).toBe(1);
expect(hs[0].last_verified_at).toBe(T0);
await P.setHolderTrust({ cid: CID, deviceId: 'dev_1', trust: 'challenged', now: T0 + 1000 });
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', trust: 'reported', now: T0 + 2000 });
hs = await P.listHolders(CID);
expect(hs[0].trust).toBe('challenged'); // a later plain report never downgrades trust
expect(hs[0].last_verified_at).toBe(T0 + 2000);
expect(await P.removeHoldersExcept({ deviceId: 'dev_1', keep: [], now: T0 + 3000 })).toBe(1);
expect((await P.listHolders(CID)).length).toBe(0);
expect(await P.activeHoldingsOf('dev_1')).toEqual([]);
await P.upsertHolder({ cid: CID, deviceId: 'dev_1', now: T0 + 4000 }); // re-added
expect(await P.activeHoldingsOf('dev_1')).toEqual([CID]);
});
test('views per day and window sums', async () => {
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0);
await P.addView('vid00000001', T0 - 40 * 86400_000);
expect(await P.viewsSince('vid00000001', T0 - 30 * 86400_000)).toBe(2);
expect(await P.viewsSince('vid00000001', T0 - 50 * 86400_000)).toBe(3);
const s = await P.p2pStats();
expect(s.content).toBe(1);
expect(s.devices).toBe(1);
});
```

273
plans/queue/009-server-content-hash-186e7f.md Normal file
View File

@@ -0,0 +1,273 @@
---
id: 009-server-content-hash-186e7f
title: Hash every validated server copy and register it as verified content
created: 2026-09-29
depends_on: [008-p2p-schema-and-config-127966]
est_files: 7
---
# 009 — Server content hashes → verified P2P content
## Objective
Implements flow 1 of `docs/p2p-architecture.md`. After this plan:
- Every copy the media cache promotes (fetch lane AND compression lane) gets a
server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are
hashed by a background backfill 30 s after boot.
- Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only
when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted.
- `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: <cid>`.
- `/api/streams`' cached payload gets an additive `data.cid` (web-only, like
`data.serverCached`).
The media-cache edits were written and tested ahead of time (25/25 media-cache
tests pass, incl. 2 new ones); they ship as patch files.
## Context the executor must NOT rediscover
- Patches (made against the current tree; `media-cache.js` and its test are untouched by
plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`.
They add to `createMediaCache()` the options `hashFile` (default `sha256File` from
`./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the
file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call
`onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export
`backfillHashes()`.
- `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last
property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`.
- `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers:
```js
headers: {
'Content-Type': 'video/mp4',
'Content-Length': String(file.size),
'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
'Cache-Control': 'no-store',
'Access-Control-Allow-Origin': '*',
},
```
- `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns
`{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`.
- Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`).
## Steps
1. Create `server/hash.js` with exactly:
```js
/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
* The hex digest of a validated file is its P2P content id (cid). */
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
export function sha256File(path) {
return new Promise((resolve, reject) => {
const h = createHash('sha256');
createReadStream(path, { highWaterMark: 1024 * 1024 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
// Hash of bytes [offset, offset+length) — used for holder range challenges.
export function sha256Range(path, offset, length) {
return new Promise((resolve, reject) => {
if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
const h = createHash('sha256');
createReadStream(path, { start: offset, end: offset + length - 1 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
```
2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B.
4. Apply the patches from the repo root:
```bash
git apply plans/patches/009-media-cache.diff
git apply plans/patches/009-media-cache-test.diff
```
If either fails, STOP and report the error (do not hand-edit).
5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`.
6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from
plan 008 may already exist; merge into it):
```js
import { admitFile } from './p2p-admit.js';
import { P2P } from './p2p-config.js';
import * as p2pDb from './p2p-db.js';
```
If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in
`main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import.
7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add:
```js
// P2P (docs/p2p-architecture.md): every validated copy's server-computed
// hash becomes verified content, after the optional malware scan.
onReady: (info) => admitFile(
{ ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
{ cfg: P2P, upsertContent: p2pDb.upsertContent },
),
```
8. `server/server.js` `cachedDownloadResponse` — add to the headers object:
```js
...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),
```
9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add
`cid: row.sha256 || null,` and add a comment above the return:
`// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.`
10. `Dockerfile` — optional ClamAV, off by default. After the existing
`RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add:
```dockerfile
# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
ARG INSTALL_CLAMAV=0
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
fi
```
## Out of scope / do NOT touch
- `validateMedia()` itself, the eviction logic, any frontend file.
- Never serve anything from the intake dir; no new routes in this plan.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
bun test ./p2p-admit.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js
```
Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail`
(media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-admit.js
```js
/* ============================================================================
* p2p-admit.js — the ONLY way a content id enters p2p_content
* (docs/p2p-architecture.md, "Security rules").
*
* Callers must already have: the complete file on the server's own disk, its
* SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
* optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
* row. The scan command gets the path as its last argument: exit 0 = clean,
* 1 = infected (rejected), anything else = scanner error (not admitted now).
* ========================================================================== */
import { spawn } from 'node:child_process';
export function scanFile(path, cmd) {
const parts = String(cmd).split(/\s+/).filter(Boolean);
return new Promise((resolve) => {
let child;
try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
catch (e) { resolve({ result: 'error', detail: e.message }); return; }
let out = '';
child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
child.on('close', (code) => resolve(
code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
));
});
}
const CID_RE = /^[0-9a-f]{64}$/;
// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
// → { ok: true, scan } | { ok: false, reason }
export async function admitFile(info, deps) {
const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
let scanResult = 'skipped';
if (cfg.malwareScan) {
const r = await scan(info.path, cfg.scanCmd);
if (r.result !== 'clean') {
log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
return { ok: false, reason: 'scan ' + r.result };
}
scanResult = 'clean';
}
await upsertContent({
cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
scan: scanResult, now: now(),
});
return { ok: true, scan: scanResult };
}
```
## Appendix B — server/p2p-admit.test.js
```js
import { test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { admitFile, scanFile } from './p2p-admit.js';
import { sha256File, sha256Range } from './hash.js';
const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
const file = join(dir, 'f.bin');
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const quiet = { warn() {}, info() {} };
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };
test('sha256File / sha256Range match node:crypto', async () => {
expect(await sha256File(file)).toBe(CID);
const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
expect(await sha256Range(file, 1000, 65536)).toBe(want);
});
test('scan off by default: admitted with scan=skipped', async () => {
const rows = [];
const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
expect(r).toEqual({ ok: true, scan: 'skipped' });
expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
});
test('scan on: clean admits, infected and scanner errors do not', async () => {
for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
const rows = [];
const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
expect(r.ok).toBe(ok);
expect(rows.length).toBe(ok ? 1 : 0);
}
});
test('disabled P2P or a malformed cid never admits', async () => {
const rows = [];
expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect(rows.length).toBe(0);
});
test('scanFile maps exit codes', async () => {
expect((await scanFile(file, 'true')).result).toBe('clean');
expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
});
```

168
plans/queue/010-views-and-retention-d0c6ca.md Normal file
View File

@@ -0,0 +1,168 @@
---
id: 010-views-and-retention-d0c6ca
title: Count views and evict server copies by retention criteria before LRU
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f]
est_files: 5
---
# 010 — Views + retention-ordered eviction
## Objective
Implements flow 9 of `docs/p2p-architecture.md`. After this plan:
- Every play (`GET /api/streams`) and save (`GET /api/download/:id`) counts one view in
`video_views` (deduped per client + video for 30 min; warm-ups don't count).
- When the media cache needs room it evicts copies that are neither **top**
(≥ `P2P_KEEP_MIN_VIEWS` views in `P2P_KEEP_DAYS`) nor **recent** (played within
`P2P_KEEP_RECENT_DAYS`) first — fewest views, then oldest — and only then qualifying
ones by LRU. Budget, disk guard and 10-min play protection are unchanged.
- Eviction never touches `p2p_content` / holders (the catalog keeps growing).
The query + media-cache change were tested ahead of time (retention test 1/1,
media-cache 26/26).
## Context the executor must NOT rediscover
- `plans/patches/010-retention-eviction.diff` patches `server/media-cache.js` (`makeRoom` uses
`db.listMediaEvictionOrder()` when present, else `db.listMediaLru()`) and adds one test to
`server/media-cache.test.js`. It applies on top of plan 009's patches.
- `server/p2p-db.js` (plan 008) has `dayKey`, `rowsOf`, `addView(videoId, now)`.
- `server/server.js` `createMediaCache({ dir: MEDIA_DIR, db: { getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats }, …` (~line 983).
- `app.get('/api/streams', async (c) => {` (~line 532): first lines validate `videoId` and
return 400 when empty. `app.get('/api/download/:videoId', async (c) => {` (~line 1311) does the same.
- Plan 009 imported `P2P` and `* as p2pDb` in server.js.
## Steps
1. `server/p2p-db.js` — append at the end of the file:
```js
// ---- retention (plan 010) -------------------------------------------------------
// Server copies in the order they should be evicted when the cache needs room:
// first the ones that are neither "top" (≥ keepMinViews views in keepDays) nor
// "recent" (played within keepRecentDays) — fewest views, then oldest — and
// only then the qualifying ones, least recently played first.
export async function listMediaEvictionOrder({ now, keepMinViews, keepDays, keepRecentDays }) {
const r = await db.execute({
sql: `SELECT m.video_id, m.size, m.last_access,
COALESCE((SELECT SUM(v.n) FROM video_views v
WHERE v.video_id = m.video_id AND v.day >= ?), 0) AS views
FROM media_cache m WHERE m.status = 'ready'`,
args: [dayKey(now - keepDays * 86400_000)],
});
const recentCut = now - keepRecentDays * 86400_000;
const rows = rowsOf(r).map((x) => ({
...x, views: Number(x.views) || 0,
qualifies: (Number(x.views) || 0) >= keepMinViews || Number(x.last_access) >= recentCut,
}));
rows.sort((a, b) => (a.qualifies - b.qualifies)
|| (a.qualifies ? 0 : a.views - b.views)
|| (a.last_access - b.last_access));
return rows;
}
```
2. Create `server/p2p-retention.test.js` — copy VERBATIM from the Appendix.
3. `server/package.json` "test" script — append ` && bun test ./p2p-retention.test.js`.
4. From the repo root: `git apply plans/patches/010-retention-eviction.diff` (STOP and report on failure).
5. `server/server.js` `createMediaCache({ … db: { … } …` — replace the `db:` line with:
```js
db: {
getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats,
// Retention (docs/p2p-architecture.md flow 9): cold copies go before popular ones.
listMediaEvictionOrder: () => p2pDb.listMediaEvictionOrder({
now: Date.now(), keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
}),
},
```
6. `server/server.js` — directly above `app.get('/api/streams', …)` (and above the plan-005
warm route if present) add:
```js
// One view per client per video per 30 min (a play and its save count once).
const viewSeen = new Map(); // `${who}|${id}` -> ms
function countView(c, videoId) {
const who = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || c.req.query('fp') || 'local';
const key = who + '|' + videoId;
const now = Date.now();
if (now - (viewSeen.get(key) || 0) < 30 * 60_000) return;
viewSeen.set(key, now);
if (viewSeen.size > 20000) viewSeen.clear();
p2pDb.addView(videoId, now).catch(() => {});
}
```
7. In `/api/streams`, directly after `if (!videoId) return c.json({ ok: false, error: 'missing videoId' }, 400);`
add `countView(c, videoId);`. Do the same in `/api/download/:videoId` after its
`missing videoId` guard.
## Out of scope / do NOT touch
- `MEDIA_CACHE_MAX_BYTES`, `EVICT_PROTECT_MS`, the disk guard, `listMediaLru` itself.
- Do not delete server files on a timer — eviction stays budget-driven.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-retention.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "countView(c, videoId)" server.js
```
Expected: `1 pass`; media-cache `26 pass 0 fail`; every file `0 fail`; `SERVER_OK`; grep `2`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — server/p2p-retention.test.js
```js
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'ytp-retention-test-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const P = await import('./p2p-db.js');
const DAY = 86400_000;
const NOW = Date.UTC(2026, 8, 29, 12);
const opts = { now: NOW, keepMinViews: 3, keepDays: 30, keepRecentDays: 14 };
beforeAll(async () => {
await dbmod.initDb();
await P.initP2pSchema();
const add = (id, lastAccessDaysAgo) => dbmod.upsertMedia(id, { status: 'ready', size: 100, last_access: NOW - lastAccessDaysAgo * DAY });
await add('topOld00001', 60); // 5 views in window → top, but old
await add('recent00001', 2); // 0 views, played 2 days ago → recent
await add('cold0000001', 40); // 1 view in window, old → evict first-ish
await add('cold0000002', 20); // 0 views, 20 days → evict first
await add('cold0000003', 50); // 0 views, 50 days → evict first (older than cold2)
for (let i = 0; i < 5; i++) await P.addView('topOld00001', NOW - 5 * DAY);
await P.addView('cold0000001', NOW - 3 * DAY);
await P.addView('cold0000002', NOW - 45 * DAY); // outside the 30-day window
});
test('non-qualifying copies go first (fewest views, then oldest); qualifying ones by LRU', async () => {
const order = (await P.listMediaEvictionOrder(opts)).map((r) => [r.video_id, r.qualifies]);
expect(order).toEqual([
['cold0000003', false],
['cold0000002', false],
['cold0000001', false],
['topOld00001', true],
['recent00001', true],
]);
});
```

242
plans/queue/011-browser-sha256-e1793d.md Normal file
View File

@@ -0,0 +1,242 @@
---
id: 011-browser-sha256-e1793d
title: Add an incremental SHA-256 library for the browser and node tests
created: 2026-09-29
depends_on: []
est_files: 5
---
# 011 — Incremental SHA-256 for the browser
## Objective
Devices must compute a file's content id (SHA-256) while bytes stream past —
during a download, a peer transfer, or a chunked OPFS read — without holding the
file in memory. WebCrypto's `digest()` needs the whole input at once, so add a
small pure-JS incremental hasher, usable from the page, from Web Workers
(`importScripts('/sha256.js')`) and from node tests. Pre-tested: all vectors and
every length 0–200 match `node:crypto`; random chunking of a 3 MiB buffer matches;
~116 MB/s in Node.
## Context the executor must NOT rediscover
- Module style to copy: `frontend/stats-core.js` — an IIFE `(function (root) { … })(typeof globalThis !== 'undefined' ? globalThis : this);`
that sets `module.exports` under node and `root.StatsCore` in the browser. Tests are
CommonJS `node:test` (`frontend/stats-core.test.js`).
- `frontend/index.html:562-569` loads scripts in this order:
```html
<script src="fingerprint.js"></script>
<script src="opfs.js"></script>
<script src="video-edit.js"></script>
<script src="lyrics-core.js"></script>
<script src="stats-core.js"></script>
<script src="async-guard.js"></script>
<script src="sw-update.js"></script>
<script src="app.js"></script>
```
- `frontend/sw.js` `SHELL` array (~line 55) must list every shell file.
## Steps
1. Create `frontend/sha256.js` — copy VERBATIM from Appendix A.
2. Create `frontend/sha256.test.js` — copy VERBATIM from Appendix B.
3. `frontend/index.html` — add ` <script src="sha256.js"></script>` directly after the
`stats-core.js` script line.
4. `frontend/sw.js` `SHELL` — add `'/sha256.js',` directly after `'/stats-core.js',`.
5. `CLAUDE.md` "Testing" section — in the list `(sw, sw-update, async-guard, video-edit, lyrics-core, stats-core)`
add `, sha256`.
## Out of scope / do NOT touch
- No callers yet (plans 012/013/017 use it). Do not touch `app.js`.
## Verification
```bash
cd /home/user/ytplayer && node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
node -e "const S=require('./frontend/sha256');console.log(S.hex(new TextEncoder().encode('abc')))"
grep -c "sha256.js" frontend/index.html frontend/sw.js
```
Expected: `# fail 0`; `ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad`; each grep `1`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — frontend/sha256.js
```js
/* ============================================================================
* sha256.js — incremental SHA-256 (pure JS; window.Sha256 / worker / node)
*
* WebCrypto's digest() needs the whole input at once, which would pull a
* multi-hundred-MB video into memory. This hasher takes chunks as they stream
* past (downloads, peer transfers, OPFS reads) and keeps ~100 bytes of state.
*
* const h = Sha256.create(); h.update(u8); …; const hex = h.hex();
* Sha256.hex(u8) // one-shot convenience
* ========================================================================== */
(function (root) {
'use strict';
const K = new Uint32Array([
0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2,
]);
function create() {
const H = new Uint32Array([
0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19,
]);
const W = new Uint32Array(64);
const block = new Uint8Array(64);
let blockLen = 0;
let total = 0; // bytes hashed so far (safe to 2^53)
let done = false;
function compress(buf, off) {
for (let i = 0; i < 16; i++) {
const j = off + i * 4;
W[i] = (buf[j] << 24) | (buf[j + 1] << 16) | (buf[j + 2] << 8) | buf[j + 3];
}
for (let i = 16; i < 64; i++) {
const w15 = W[i - 15], w2 = W[i - 2];
const s0 = ((w15 >>> 7) | (w15 << 25)) ^ ((w15 >>> 18) | (w15 << 14)) ^ (w15 >>> 3);
const s1 = ((w2 >>> 17) | (w2 << 15)) ^ ((w2 >>> 19) | (w2 << 13)) ^ (w2 >>> 10);
W[i] = (W[i - 16] + s0 + W[i - 7] + s1) | 0;
}
let a = H[0], b = H[1], c = H[2], d = H[3], e = H[4], f = H[5], g = H[6], h = H[7];
for (let i = 0; i < 64; i++) {
const S1 = ((e >>> 6) | (e << 26)) ^ ((e >>> 11) | (e << 21)) ^ ((e >>> 25) | (e << 7));
const ch = (e & f) ^ (~e & g);
const t1 = (h + S1 + ch + K[i] + W[i]) | 0;
const S0 = ((a >>> 2) | (a << 30)) ^ ((a >>> 13) | (a << 19)) ^ ((a >>> 22) | (a << 10));
const maj = (a & b) ^ (a & c) ^ (b & c);
const t2 = (S0 + maj) | 0;
h = g; g = f; f = e; e = (d + t1) | 0;
d = c; c = b; b = a; a = (t1 + t2) | 0;
}
H[0] = (H[0] + a) | 0; H[1] = (H[1] + b) | 0; H[2] = (H[2] + c) | 0; H[3] = (H[3] + d) | 0;
H[4] = (H[4] + e) | 0; H[5] = (H[5] + f) | 0; H[6] = (H[6] + g) | 0; H[7] = (H[7] + h) | 0;
}
function update(data) {
if (done) throw new Error('sha256: update() after digest');
const u8 = data instanceof Uint8Array ? data : new Uint8Array(data);
let i = 0;
total += u8.length;
if (blockLen) {
const take = Math.min(64 - blockLen, u8.length);
block.set(u8.subarray(0, take), blockLen);
blockLen += take;
i = take;
if (blockLen === 64) { compress(block, 0); blockLen = 0; }
}
for (; i + 64 <= u8.length; i += 64) compress(u8, i);
if (i < u8.length) { block.set(u8.subarray(i), 0); blockLen = u8.length - i; }
return api;
}
function digest() {
if (!done) {
done = true;
const bits = total * 8;
block[blockLen++] = 0x80;
if (blockLen > 56) { block.fill(0, blockLen); compress(block, 0); blockLen = 0; }
block.fill(0, blockLen, 56);
const hi = Math.floor(bits / 0x100000000), lo = bits >>> 0;
block[56] = hi >>> 24; block[57] = hi >>> 16; block[58] = hi >>> 8; block[59] = hi;
block[60] = lo >>> 24; block[61] = lo >>> 16; block[62] = lo >>> 8; block[63] = lo;
compress(block, 0);
}
const out = new Uint8Array(32);
for (let i = 0; i < 8; i++) {
out[i * 4] = H[i] >>> 24; out[i * 4 + 1] = H[i] >>> 16; out[i * 4 + 2] = H[i] >>> 8; out[i * 4 + 3] = H[i];
}
return out;
}
function hex() {
let s = '';
for (const b of digest()) s += (b < 16 ? '0' : '') + b.toString(16);
return s;
}
const api = { update, digest, hex, get bytes() { return total; } };
return api;
}
const Sha256 = {
create,
hex: (data) => create().update(data).hex(),
isHex: (s) => typeof s === 'string' && /^[0-9a-f]{64}$/.test(s),
};
if (typeof module !== 'undefined' && module.exports) module.exports = Sha256;
else root.Sha256 = Sha256;
})(typeof globalThis !== 'undefined' ? globalThis : this);
```
## Appendix B — frontend/sha256.test.js
```js
'use strict';
const { test } = require('node:test');
const assert = require('node:assert');
const crypto = require('node:crypto');
const Sha256 = require('./sha256');
const ref = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
test('known vectors', () => {
assert.strictEqual(Sha256.hex(new Uint8Array(0)), 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855');
assert.strictEqual(Sha256.hex(new TextEncoder().encode('abc')), 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad');
assert.strictEqual(
Sha256.hex(new TextEncoder().encode('abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq')),
'248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1');
});
test('every length around block boundaries matches node:crypto', () => {
for (let n = 0; n <= 200; n++) {
const buf = crypto.randomBytes(n);
assert.strictEqual(Sha256.hex(new Uint8Array(buf)), ref(buf), 'length ' + n);
}
});
test('random chunk boundaries give the same digest as one shot', () => {
const buf = crypto.randomBytes(3 * 1024 * 1024 + 17);
for (let round = 0; round < 5; round++) {
const h = Sha256.create();
let pos = 0;
while (pos < buf.length) {
const n = Math.min(buf.length - pos, 1 + Math.floor(Math.random() * 200000));
h.update(new Uint8Array(buf.buffer, buf.byteOffset + pos, n));
pos += n;
}
assert.strictEqual(h.hex(), ref(buf));
assert.strictEqual(h.bytes, buf.length);
}
});
test('update after digest throws; isHex', () => {
const h = Sha256.create();
h.hex();
assert.throws(() => h.update(new Uint8Array(1)));
assert.ok(Sha256.isHex(ref(Buffer.from('x'))));
assert.ok(!Sha256.isHex('ABC'));
});
```

215
plans/queue/012-device-file-registry-288d55.md Normal file
View File

@@ -0,0 +1,215 @@
---
id: 012-device-file-registry-288d55
title: Add the on-device IndexedDB file registry and hash saves while downloading
created: 2026-09-29
depends_on: [009-server-content-hash-186e7f, 011-browser-sha256-e1793d]
est_files: 6
---
# 012 — On-device file registry + hash while saving
## Objective
Implements flow 2 of `docs/p2p-architecture.md`. After this plan:
- `frontend/device-db.js` (`window.DeviceDB`) keeps one IndexedDB record per saved
video: `{ videoId, cid, size, savedAt, lastCheckedAt, state }`.
- The OPFS download worker hashes bytes as it writes them. If the server sent
`X-Content-SHA256` (plan 009) and the hash differs, the save FAILS and the partial
file is deleted ("integrity check failed").
- `preload()` records the file (`state: 'verified'` when the hashes matched,
`'unverified'` when the server sent none, `'unhashed'` for the main-thread fallback);
deleting / clearing saved videos removes the records.
Pre-tested in Chromium with the harness in `plans/harness/` (good hash → saved and
`verified`; wrong hash → rejected, no file left; no header → saved with a local hash).
## Context the executor must NOT rediscover
- `plans/patches/012-opfs-hash.diff` patches `frontend/opfs-worker.js` (importScripts
`/sha256.js`, hash in the write loop, compare with `x-content-sha256`, post
`{ type:'done', ext, sha256, expectedSha, size }`) and `frontend/opfs.js`
(`downloadVideo` resolves `{ ok:true, sha256, expectedSha, size }`).
- `frontend/app.js:157-198` `async function opfsDownload(videoId, { mux = false } = {})` — worker path:
```js
if (typeof window.OPFS.downloadVideo === 'function' && typeof Worker !== 'undefined') {
const w = await window.OPFS.downloadVideo(videoId, url);
if (w.ok) return { ok: true, cached: true };
workerError = w.error || null;
}
```
- `frontend/app.js:258-268`:
```js
async function opfsDelete(videoId) {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.deleteVideo(videoId); } catch { /* ignore */ }
return { ok: true };
}
async function opfsClear() {
if (!window.OPFS || !window.OPFS.isSupported()) return { ok: true };
try { await window.OPFS.clearAll(); } catch { /* ignore */ }
return { ok: true };
}
```
- `frontend/app.js` `async function preload(video, …)` (~line 1237), success branch:
```js
const res = await API.cacheDownload(id, { mux });
if (res && res.ok && res.cached) {
cachedIds.add(id);
cacheMutations++;
warmThumb(thumbUrlFor(id, video));
```
- Script order in `frontend/index.html` after plan 011: `… stats-core.js, sha256.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` after plan 011 contains `'/sha256.js',`.
## Steps
1. Create `frontend/device-db.js` — copy VERBATIM from the Appendix.
2. From the repo root: `git apply plans/patches/012-opfs-hash.diff` (STOP and report on failure).
3. `frontend/index.html` — add ` <script src="device-db.js"></script>` directly after the `sha256.js` line.
4. `frontend/sw.js` `SHELL` — add `'/device-db.js',` directly after `'/sha256.js',`.
5. `frontend/app.js` `opfsDownload` — the line `if (w.ok) return { ok: true, cached: true };` appears
TWICE in app.js; change ONLY the first one (inside `async function opfsDownload`, ~line 180),
NOT the one inside `opfsDownloadEdited` (edited cuts are never shared). Change it to:
```js
if (w.ok) return { ok: true, cached: true, sha256: w.sha256 || null, expectedSha: w.expectedSha || null, size: w.size || 0 };
```
(The main-thread fallback below keeps returning `{ ok: true, cached: true }` — no hash.)
6. `frontend/app.js` — directly ABOVE `async function preload(video, …)` add:
```js
// This device's record of what it holds and each file's content id
// (docs/p2p-architecture.md flow 2). The P2P client reports these.
function recordDeviceFile(id, res) {
if (!WEB || !window.DeviceDB) return;
const cid = res && window.Sha256 && window.Sha256.isHex(res.sha256) ? res.sha256 : null;
const state = cid ? (res.expectedSha === cid ? 'verified' : 'unverified') : 'unhashed';
const now = Date.now();
window.DeviceDB.putFile({ videoId: id, cid, size: (res && res.size) || 0, savedAt: now, lastCheckedAt: now, state })
.then(() => { if (window.P2PClient) window.P2PClient.changed(); })
.catch(() => {});
}
```
7. `frontend/app.js` `preload` success branch — after `cacheMutations++;` add
`recordDeviceFile(id, res);`
8. `frontend/app.js` `opfsDelete` — after the `try { await window.OPFS.deleteVideo(videoId); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.deleteFile(videoId); if (window.P2PClient) window.P2PClient.changed(); }
```
`opfsClear` — after its `try { await window.OPFS.clearAll(); } …` line add:
```js
if (window.DeviceDB) { await window.DeviceDB.clear(); if (window.P2PClient) window.P2PClient.changed(); }
```
## Out of scope / do NOT touch
- `opfsDownloadEdited` (edited cuts are device-only, never shared).
- No server calls here (plan 013 reports holdings). Do not hash old files here (plan 013).
- Do not migrate `_ytpdata` / playlists to IndexedDB.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/opfs.js frontend/opfs-worker.js frontend/device-db.js && echo SYNTAX_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "device-db.js" frontend/index.html frontend/sw.js
# Browser check (Chromium). Skip with a note in Findings if no Chromium/playwright is available.
cd plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1)
bun device-db-server.js >/tmp/ytp012.log 2>&1 & SRV=$!; sleep 2; timeout 60 node device-db-check.mjs; kill $SRV; true
```
Expected: `SYNTAX_OK`; `# fail 0`; grep `1` each; the browser check prints JSON containing
`"shaOk":true,"goodExpected":true`, `"bad":{"ok":false,"error":"integrity check failed (content hash mismatch)"}`,
`"list":["good","nohash"]`, `"rec":"verified"`, `"after":0`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix — frontend/device-db.js
```js
/* ============================================================================
* device-db.js — this device's own file registry (IndexedDB "ytp-device")
*
* One record per saved video, next to the bytes in OPFS:
* { videoId, cid, size, savedAt, lastCheckedAt, state }
* cid SHA-256 of the stored file (P2P content id) or null
* state 'verified' hash computed on save and equal to the server's
* 'unverified' hash computed, but the server sent none to compare
* 'unhashed' saved before hashing existed / main-thread fallback
* The P2P client (p2p-client.js) reads this to report holdings; OPFS stays the
* source of truth for what is playable (a record without a file is ignored).
* Every call resolves (null / [] on failure) — private windows can block IDB.
* See docs/p2p-architecture.md.
* ========================================================================== */
(function () {
'use strict';
const DB_NAME = 'ytp-device';
const VERSION = 1;
let _open = null;
function open() {
if (!_open) {
_open = new Promise((resolve, reject) => {
const r = indexedDB.open(DB_NAME, VERSION);
r.onupgradeneeded = () => {
const d = r.result;
if (!d.objectStoreNames.contains('files')) {
const s = d.createObjectStore('files', { keyPath: 'videoId' });
s.createIndex('cid', 'cid', { unique: false });
}
};
r.onsuccess = () => resolve(r.result);
r.onerror = () => reject(r.error);
r.onblocked = () => reject(new Error('device-db blocked'));
});
_open.catch(() => { _open = null; });
}
return _open;
}
const done = (req) => new Promise((resolve, reject) => {
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
async function store(mode) {
const d = await open();
return d.transaction('files', mode).objectStore('files');
}
async function safe(fn, fallback) {
try { return await fn(); } catch { return fallback; }
}
window.DeviceDB = {
isSupported: () => typeof indexedDB !== 'undefined',
putFile: (rec) => safe(async () => {
if (!rec || !rec.videoId) return null;
await done((await store('readwrite')).put({
videoId: String(rec.videoId),
cid: rec.cid || null,
size: Number(rec.size) || 0,
savedAt: Number(rec.savedAt) || Date.now(),
lastCheckedAt: Number(rec.lastCheckedAt) || Date.now(),
state: rec.state || 'unhashed',
}));
return true;
}, null),
getFile: (videoId) => safe(async () => (await done((await store('readonly')).get(String(videoId)))) || null, null),
getByCid: (cid) => safe(async () => (await done((await store('readonly')).index('cid').get(String(cid)))) || null, null),
listFiles: () => safe(async () => (await done((await store('readonly')).getAll())) || [], []),
deleteFile: (videoId) => safe(async () => { await done((await store('readwrite')).delete(String(videoId))); return true; }, null),
clear: () => safe(async () => { await done((await store('readwrite')).clear()); return true; }, null),
};
}());
```

622
plans/queue/013-device-identity-and-holdings-3ba493.md Normal file
View File

@@ -0,0 +1,622 @@
---
id: 013-device-identity-and-holdings-3ba493
title: Register devices and report verified holdings to the server
created: 2026-09-29
depends_on: [010-views-and-retention-d0c6ca, 012-device-file-registry-288d55]
est_files: 11
---
# 013 — Device identity + holdings sync
## Objective
Implements flow 3 of `docs/p2p-architecture.md`. After this plan, with default
settings (P2P ON):
- A device registers once (`POST /api/p2p/device` → `deviceId` + `secret`, kept in
`localStorage.ytpDevice`; only `sha256(secret)` is stored server-side).
- `P2PClient` reconciles `DeviceDB` with the files really in OPFS, hashes old/unhashed
saves in a worker, reports the FULL holdings list, answers the server's range
challenges, and marks accepted files `verified`.
- The server keeps holder rows forever (no TTL): refreshed `last_verified_at` on every
report, `removed` when a full report no longer lists them or a challenge fails.
- Turning sharing off (`data.settings.p2pShare === false`) withdraws every holding.
Pre-tested: server routes 6/6 (`bun:test`), and the whole client flow in Chromium via
`plans/harness/p2p-client-*.js` (ghost record dropped, legacy file hashed, verified
file accepted + challenge passed → `trust: challenged`, share off → 0 holders).
## Context the executor must NOT rediscover
- Plans 008–012 are applied: `server/p2p-db.js`, `p2p-config.js` (`P2P`), `hash.js`
(`sha256Range`), `p2p-admit.js`; `frontend/sha256.js`, `frontend/device-db.js`, and
`opfs.js`/`opfs-worker.js` hash on save. `server/server.js` imports `P2P` and `* as p2pDb`.
- `plans/patches/013-opfs-readrange.diff` adds `OPFS.readRange(videoId, offset, length)` to
`frontend/opfs.js` (applies on top of plan 012's patch).
- `server/server.js`: `const MEDIA_DIR = process.env.MEDIA_DIR || './data/media';` (~line 980).
Plan 009's `const notes = registerNoteRoutes(app, {…});` (~line 1823) is a good neighbourhood
for another `register…Routes(app, …)` call — add ours right after the uploads block
(`const isUpload = (id) => uploads.isUploadId(id);`).
- `frontend/app.js`:
- `const DEFAULT_SETTINGS = {` (~line 326), first line
`quality: 'auto', volume: 1, audioOnly: false, autoPreload: true,`.
- end of `async function boot()` (~line 9822):
```js
// Learn what's already cached, then top up any playlist videos that aren't.
await refreshCachedIds();
startCacheResyncWatch();
```
- `data.profile` is `{ name, syncedAt }` or null.
- Script order in `frontend/index.html` after plan 012:
`… stats-core.js, sha256.js, device-db.js, async-guard.js, sw-update.js, app.js`.
## Steps
1. `server/p2p-db.js`:
a. In `initP2pSchema()`, directly after the `ALTER TABLE media_cache ADD COLUMN sha256` try/catch add:
` await db.execute('CREATE INDEX IF NOT EXISTS idx_media_sha256 ON media_cache (sha256)');`
b. Append at the end of the file:
```js
// The server's own ready copy whose mp4 has this content id (or null).
export async function findMediaByCid(cid) {
const r = await db.execute({
sql: "SELECT video_id, gen FROM media_cache WHERE sha256 = ? AND status = 'ready' LIMIT 1",
args: [cid],
});
return rowsOf(r)[0] || null;
}
```
2. Create `server/p2p-routes.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-routes.test.js` — copy VERBATIM from Appendix B.
4. `server/package.json` "test" script — append ` && bun test ./p2p-routes.test.js`.
5. `server/server.js`:
a. imports: `import { registerP2pRoutes } from './p2p-routes.js';` and add `sha256Range` to an
import from `./hash.js`: `import { sha256Range } from './hash.js';`
b. after `const isUpload = (id) => uploads.isUploadId(id);` add:
```js
// ============================================================================
// Peer-to-peer sharing — devices + holdings (docs/p2p-architecture.md)
// ============================================================================
async function fileForCid(cid) {
const row = await p2pDb.findMediaByCid(cid);
if (!row) return null;
const path = `${MEDIA_DIR}/${row.video_id}.${row.gen}.mp4`;
const f = Bun.file(path);
return (await f.exists()) ? { path, size: f.size } : null;
}
const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });
```
(`p2p` is used by plan 014.)
6. Create `frontend/hash-worker.js` — copy VERBATIM from Appendix C.
7. Create `frontend/p2p-client.js` — copy VERBATIM from Appendix D.
8. From the repo root: `git apply plans/patches/013-opfs-readrange.diff` (STOP on failure).
9. `frontend/index.html` — add ` <script src="p2p-client.js"></script>` directly after the `device-db.js` line.
10. `frontend/sw.js` `SHELL` — add `'/hash-worker.js',` and `'/p2p-client.js',` directly after `'/device-db.js',`.
11. `frontend/app.js` `DEFAULT_SETTINGS` — add a new line after the first line:
```js
p2pShare: true, // P2P (docs/p2p-architecture.md): share my saved videos with other devices — ON by default
p2pReceive: true, // fetch from other devices when YouTube and the server can't — ON by default
```
12. `frontend/app.js` `boot()` — directly after `startCacheResyncWatch();` add:
```js
// Peer-to-peer: report what this device holds (on by default; settings.p2pShare).
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
## Out of scope / do NOT touch
- Presence/websocket, holders endpoint, UI (plans 014/015). Intake of unknown files (016).
- Do not change how saves download or play.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-routes.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js frontend/hash-worker.js frontend/opfs.js frontend/app.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
# Browser check (Chromium + playwright; see plans/harness/device-db-check.mjs header).
cd server
bun ../plans/harness/p2p-client-server.js >/tmp/ytp013.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node p2p-client-check.mjs
kill $SRV; true
```
Expected: routes `6 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; `# fail 0`; browser JSON
`{"accepted":1,"unknown":1,"challenges":1,"recs":[["goodAAAAAAA","verified",true],["legacyAAAAA","unverified",true]],"trust":["challenged"],"afterShareOff":0,"device":true}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes (new files may be summarised as "verbatim from appendix").
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-routes.js
```js
/* ============================================================================
* p2p-routes.js — device registration + holdings (docs/p2p-architecture.md
* flows 3 and 5). Mounted by server.js; every route answers 404 when
* P2P_ENABLED=0.
*
* GET /api/p2p/config { ok, enabled, staleDays }
* POST /api/p2p/device { fingerprint?, profile? } → { ok, deviceId, secret }
* POST /api/p2p/holdings (device) { items:[{cid,size,videoId}], share } → { ok, accepted, unknown, challenges }
* POST /api/p2p/challenge (device) { answers:[{cid,offset,length,sha256}] } → { ok, passed, failed }
* GET /api/p2p/holders?v=<id>|cid= availability — added by plan 014
*
* Device auth: header `X-Device: <deviceId>.<secret>`; only sha256(secret) is
* stored. A holdings report is always the device's FULL list: anything it held
* before and no longer lists is marked removed. Holder rows never expire by
* time — last_verified_at is refreshed by each report (see the architecture doc).
* ========================================================================== */
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto';
const CID_RE = /^[0-9a-f]{64}$/;
const DEV_RE = /^dev_[0-9a-f]{16}$/;
const MAX_ITEMS = 5000;
const MAX_CHALLENGES = 5;
const CHALLENGE_LEN = 64 * 1024;
const sha = (s) => createHash('sha256').update(String(s)).digest('hex');
const same = (a, b) => { const x = Buffer.from(String(a)), y = Buffer.from(String(b)); return x.length === y.length && timingSafeEqual(x, y); };
export const peerIdOf = (deviceId) => sha('peer:' + deviceId).slice(0, 12);
export function registerP2pRoutes(app, deps) {
const { cfg, p2pDb, fileForCid, sha256Range, now = () => Date.now(), log = console } = deps;
const pending = new Map(); // `${deviceId}|${cid}` -> { offset, length, at }
const regLog = new Map(); // ip -> [ms]
const gate = async (c, next) => {
if (!cfg.enabled) return c.json({ ok: false, error: 'p2p disabled' }, 404);
await next();
};
async function deviceOf(c) {
const m = String(c.req.header('x-device') || '').match(/^(dev_[0-9a-f]{16})\.([0-9a-f]{64})$/);
if (!m) return null;
const d = await p2pDb.getDevice(m[1]);
if (!d || !same(d.secret_hash, sha(m[2]))) return null;
return d;
}
const requireDevice = async (c, next) => {
const d = await deviceOf(c);
if (!d) return c.json({ ok: false, error: 'unknown device' }, 401);
c.set('device', d);
await next();
};
app.get('/api/p2p/config', (c) => c.json({ ok: true, enabled: cfg.enabled, staleDays: cfg.staleDays }));
app.post('/api/p2p/device', gate, async (c) => {
const ip = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || 'local';
const t = now();
const recent = (regLog.get(ip) || []).filter((x) => t - x < 3600_000);
if (recent.length >= 20) return c.json({ ok: false, error: 'too many registrations' }, 429);
recent.push(t);
regLog.set(ip, recent);
if (regLog.size > 10000) regLog.clear();
const body = await c.req.json().catch(() => ({}));
const deviceId = 'dev_' + randomBytes(8).toString('hex');
const secret = randomBytes(32).toString('hex');
await p2pDb.createDevice({
deviceId, secretHash: sha(secret),
fingerprint: String(body.fingerprint || '').slice(0, 128) || null,
profile: String(body.profile || '').slice(0, 64) || null,
now: t,
});
return c.json({ ok: true, deviceId, secret });
});
app.post('/api/p2p/holdings', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const share = body.share !== false;
await p2pDb.touchDevice(d.device_id, { now: t, share, profile: String(body.profile || '').slice(0, 64) || undefined });
const raw = Array.isArray(body.items) ? body.items.slice(0, MAX_ITEMS) : [];
const items = share ? raw.filter((x) => x && CID_RE.test(String(x.cid))) : [];
const known = await p2pDb.knownCids(items.map((x) => x.cid));
const accepted = [];
const unknown = [];
for (const it of items) {
if (!known.has(it.cid)) { unknown.push(it.cid); continue; }
await p2pDb.upsertHolder({ cid: it.cid, deviceId: d.device_id, now: t });
accepted.push(it.cid);
}
await p2pDb.removeHoldersExcept({ deviceId: d.device_id, keep: accepted, now: t });
// Spot-check a few holdings against the server's own copy when it has one.
const challenges = [];
for (const cid of accepted) {
if (challenges.length >= MAX_CHALLENGES) break;
const f = await fileForCid(cid);
if (!f || !(f.size > CHALLENGE_LEN)) continue;
const offset = Math.floor(Math.random() * (f.size - CHALLENGE_LEN));
pending.set(d.device_id + '|' + cid, { offset, length: CHALLENGE_LEN, at: t });
challenges.push({ cid, offset, length: CHALLENGE_LEN });
}
if (pending.size > 50000) pending.clear();
return c.json({ ok: true, accepted, unknown, challenges });
});
app.post('/api/p2p/challenge', gate, requireDevice, async (c) => {
const d = c.get('device');
const body = await c.req.json().catch(() => ({}));
const t = now();
const passed = [];
const failed = [];
for (const a of (Array.isArray(body.answers) ? body.answers : []).slice(0, MAX_CHALLENGES)) {
const key = d.device_id + '|' + a.cid;
const p = pending.get(key);
if (!p || p.offset !== a.offset || p.length !== a.length || t - p.at > 10 * 60_000) continue;
pending.delete(key);
const f = await fileForCid(a.cid);
if (!f) continue;
const want = await sha256Range(f.path, p.offset, p.length);
if (String(a.sha256) === want) {
await p2pDb.setHolderTrust({ cid: a.cid, deviceId: d.device_id, trust: 'challenged', now: t });
passed.push(a.cid);
} else {
await p2pDb.removeHolder({ cid: a.cid, deviceId: d.device_id, now: t });
failed.push(a.cid);
log.warn?.(`[p2p] ${d.device_id} failed challenge for ${a.cid.slice(0, 12)}`);
}
}
return c.json({ ok: true, passed, failed });
});
return { deviceOf, peerIdOf, requireDevice, gate };
}
```
## Appendix B — server/p2p-routes.test.js
```js
// Device registration, holdings reports and range challenges (plan 013).
import { test, expect, beforeAll } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { Hono } from 'hono';
const root = mkdtempSync(join(tmpdir(), 'ytp-p2p-routes-'));
process.env.DB_PATH = join(root, 'test.db');
const dbmod = await import('./db.js');
const p2pDb = await import('./p2p-db.js');
const { registerP2pRoutes } = await import('./p2p-routes.js');
const { sha256Range } = await import('./hash.js');
const file = join(root, 'copy.mp4');
const bytes = Buffer.from(Array.from({ length: 400000 }, (_, i) => (i * 13) % 256));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const OTHER = 'c'.repeat(64); // verified but the server has no file
const UNKNOWN = 'd'.repeat(64); // not in p2p_content
const quiet = { warn() {}, info() {} };
let app;
let enabled = true;
const req = (path, { method = 'GET', body, dev } = {}) => app.request(path, {
method,
headers: { 'Content-Type': 'application/json', ...(dev ? { 'X-Device': dev.deviceId + '.' + dev.secret } : {}) },
body: body ? JSON.stringify(body) : undefined,
});
beforeAll(async () => {
await dbmod.initDb();
await p2pDb.initP2pSchema();
await p2pDb.upsertContent({ cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server', now: 1 });
await p2pDb.upsertContent({ cid: OTHER, videoId: 'dQw4w9WgXcQ', size: 10, origin: 'server', now: 1 });
app = new Hono();
registerP2pRoutes(app, {
cfg: { get enabled() { return enabled; }, staleDays: 7 },
p2pDb,
fileForCid: async (cid) => (cid === CID ? { path: file, size: bytes.length } : null),
sha256Range,
log: quiet,
});
});
async function newDevice() {
const r = await req('/api/p2p/device', { method: 'POST', body: { fingerprint: 'fp1' } });
expect(r.status).toBe(200);
const j = await r.json();
expect(j.deviceId).toMatch(/^dev_[0-9a-f]{16}$/);
expect(j.secret).toMatch(/^[0-9a-f]{64}$/);
return j;
}
test('config is public and says enabled + staleDays', async () => {
expect(await (await req('/api/p2p/config')).json()).toEqual({ ok: true, enabled: true, staleDays: 7 });
});
test('holdings need a valid device secret', async () => {
const dev = await newDevice();
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] } })).status).toBe(401);
expect((await req('/api/p2p/holdings', { method: 'POST', body: { items: [] }, dev: { ...dev, secret: 'e'.repeat(64) } })).status).toBe(401);
});
test('report accepts verified cids, returns unknown ones, and challenges the server-held file', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: true, items: [
{ cid: CID, size: bytes.length, videoId: 'dQw4w9WgXcQ' }, { cid: OTHER, size: 10 }, { cid: UNKNOWN, size: 5 }, { cid: 'nothex' },
] } })).json();
expect(r.accepted.sort()).toEqual([CID, OTHER].sort());
expect(r.unknown).toEqual([UNKNOWN]);
expect(r.challenges.length).toBe(1);
const ch = r.challenges[0];
expect(ch.cid).toBe(CID);
const good = createHash('sha256').update(bytes.subarray(ch.offset, ch.offset + ch.length)).digest('hex');
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: good }] } })).json();
expect(a).toEqual({ ok: true, passed: [CID], failed: [] });
const hs = await p2pDb.listHolders(CID);
expect(hs.find((h) => h.device_id === dev.deviceId).trust).toBe('challenged');
});
test('a wrong challenge answer removes the holder; a replayed answer is ignored', async () => {
const dev = await newDevice();
const r = await (await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID, size: bytes.length }] } })).json();
const ch = r.challenges[0];
const a = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(a.failed).toEqual([CID]);
expect((await p2pDb.listHolders(CID)).some((h) => h.device_id === dev.deviceId)).toBe(false);
const again = await (await req('/api/p2p/challenge', { method: 'POST', dev, body: { answers: [{ ...ch, sha256: '0'.repeat(64) }] } })).json();
expect(again).toEqual({ ok: true, passed: [], failed: [] });
});
test('a full report without an item removes it; share=false withdraws everything', async () => {
const dev = await newDevice();
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: CID }, { cid: OTHER }] } });
expect((await p2pDb.activeHoldingsOf(dev.deviceId)).sort()).toEqual([CID, OTHER].sort());
await req('/api/p2p/holdings', { method: 'POST', dev, body: { items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([OTHER]);
await req('/api/p2p/holdings', { method: 'POST', dev, body: { share: false, items: [{ cid: OTHER }] } });
expect(await p2pDb.activeHoldingsOf(dev.deviceId)).toEqual([]);
expect((await p2pDb.getDevice(dev.deviceId)).share).toBe(0);
});
test('P2P_ENABLED=0 → routes answer 404 (config still says disabled)', async () => {
enabled = false;
try {
expect((await req('/api/p2p/device', { method: 'POST', body: {} })).status).toBe(404);
expect((await (await req('/api/p2p/config')).json()).enabled).toBe(false);
} finally { enabled = true; }
});
```
## Appendix C — frontend/hash-worker.js
```js
/* ============================================================================
* hash-worker.js — SHA-256 of a file already saved in OPFS, off the main
* thread, read in 4 MiB slices (never the whole video in memory).
*
* In: { name } file name under OPFS videos/ (e.g. "abc.mp4")
* Out: { ok: true, sha256, size } | { ok: false, error }
* ========================================================================== */
'use strict';
importScripts('/sha256.js');
self.onmessage = async (e) => {
const { name } = e.data || {};
try {
const root = await navigator.storage.getDirectory();
const dir = await root.getDirectoryHandle('videos');
const file = await (await dir.getFileHandle(String(name))).getFile();
const h = self.Sha256.create();
const STEP = 4 * 1024 * 1024;
for (let pos = 0; pos < file.size; pos += STEP) {
h.update(new Uint8Array(await file.slice(pos, pos + STEP).arrayBuffer()));
}
self.postMessage({ ok: true, sha256: h.hex(), size: file.size });
} catch (err) {
self.postMessage({ ok: false, error: err && err.message ? err.message : String(err) });
}
};
```
## Appendix D — frontend/p2p-client.js
```js
/* ============================================================================
* p2p-client.js — this device's side of peer-to-peer sharing
* (docs/p2p-architecture.md flows 2–3). window.P2PClient.
*
* start({ getSettings, getProfile }) called once from app.js boot()
* changed() a save/delete happened → re-report soon
* device() { deviceId, secret } or null
* authHeaders() { 'X-Device': … } for other P2P calls
*
* What it does, in order, each sync:
* 1. registers the device once (localStorage ytpDevice)
* 2. reconciles DeviceDB with the files really in OPFS (a record without a
* file is dropped; a file without a record is added as 'unhashed')
* 3. hashes 'unhashed' files and files not re-checked for 30 days, one at a
* time in hash-worker.js
* 4. reports the FULL holdings list (empty when sharing is off), answers the
* server's range challenges, and marks accepted files 'verified'
* P2P is ON by default: sharing runs unless settings.p2pShare === false or the
* server says P2P is disabled.
* ========================================================================== */
(function () {
'use strict';
const KEY = 'ytpDevice';
const REHASH_MS = 30 * 24 * 3600_000;
const RESYNC_MS = 6 * 3600_000;
let hooks = { getSettings: () => ({}), getProfile: () => '' };
let serverCfg = null;
let running = null;
let again = false;
let timer = null;
let lastSync = 0;
function device() {
try {
const d = JSON.parse(localStorage.getItem(KEY) || 'null');
return d && /^dev_[0-9a-f]{16}$/.test(d.deviceId) && /^[0-9a-f]{64}$/.test(d.secret) ? d : null;
} catch { return null; }
}
const authHeaders = () => { const d = device(); return d ? { 'X-Device': d.deviceId + '.' + d.secret } : {}; };
async function config() {
if (serverCfg) return serverCfg;
try {
const j = await (await fetch('/api/p2p/config')).json();
if (j && j.ok) serverCfg = j;
} catch { /* offline — try again next sync */ }
return serverCfg;
}
async function ensureDevice() {
const have = device();
if (have) return have;
const r = await fetch('/api/p2p/device', {
method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ fingerprint: window.getFingerprint ? window.getFingerprint() : '', profile: hooks.getProfile() || '' }),
});
const j = await r.json();
if (!j || !j.ok) throw new Error((j && j.error) || 'device registration failed');
const d = { deviceId: j.deviceId, secret: j.secret };
try { localStorage.setItem(KEY, JSON.stringify(d)); } catch { /* storage blocked */ }
return d;
}
function hashInWorker(name) {
return new Promise((resolve) => {
let w;
try { w = new Worker('/hash-worker.js'); } catch { resolve(null); return; }
w.onmessage = (e) => { w.terminate(); resolve(e.data && e.data.ok ? e.data : null); };
w.onerror = () => { w.terminate(); resolve(null); };
w.postMessage({ name });
});
}
async function sha256Range(videoId, offset, length) {
const bytes = window.OPFS && window.OPFS.readRange ? await window.OPFS.readRange(videoId, offset, length) : null;
if (!bytes) return null;
return window.Sha256.hex(bytes);
}
// DeviceDB ⇄ OPFS. Returns the records that describe a real file.
async function reconcile() {
const files = await window.OPFS.listVideos({ strict: true });
const byId = new Map(files.map((f) => [f.id, f]));
const recs = await window.DeviceDB.listFiles();
const out = [];
for (const r of recs) {
const f = byId.get(r.videoId);
if (!f) { await window.DeviceDB.deleteFile(r.videoId); continue; }
if (f.size !== r.size && r.size) { r.cid = null; r.state = 'unhashed'; r.size = f.size; await window.DeviceDB.putFile(r); }
r.name = f.name;
out.push(r);
byId.delete(r.videoId);
}
for (const f of byId.values()) {
if (String(f.id).startsWith('edit_')) continue; // edited cuts are never shared
const r = { videoId: f.id, cid: null, size: f.size, savedAt: Date.now(), lastCheckedAt: 0, state: 'unhashed' };
await window.DeviceDB.putFile(r);
out.push({ ...r, name: f.name });
}
return out;
}
async function hashPending(recs) {
const now = Date.now();
for (const r of recs) {
if (r.state !== 'unhashed' && now - (r.lastCheckedAt || 0) < REHASH_MS) continue;
const h = await hashInWorker(r.name);
if (!h) continue;
const changedCid = h.sha256 !== r.cid;
r.cid = h.sha256;
r.size = h.size;
r.lastCheckedAt = Date.now();
if (changedCid || r.state === 'unhashed') r.state = 'unverified';
await window.DeviceDB.putFile(r);
}
}
async function report(recs, dev) {
const share = hooks.getSettings().p2pShare !== false;
const items = recs.filter((r) => r.cid).map((r) => ({ cid: r.cid, size: r.size, videoId: r.videoId }));
const r = await fetch('/api/p2p/holdings', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ share, items: share ? items : [], profile: hooks.getProfile() || '' }),
});
if (r.status === 401) { try { localStorage.removeItem(KEY); } catch { /* ignore */ } return null; }
const j = await r.json();
if (!j || !j.ok) return null;
const accepted = new Set(j.accepted || []);
for (const rec of recs) {
if (rec.cid && accepted.has(rec.cid) && rec.state !== 'verified') { rec.state = 'verified'; await window.DeviceDB.putFile(rec); }
}
const answers = [];
for (const ch of j.challenges || []) {
const rec = recs.find((x) => x.cid === ch.cid);
if (!rec) continue;
const hex = await sha256Range(rec.videoId, ch.offset, ch.length);
if (hex) answers.push({ ...ch, sha256: hex });
}
if (answers.length) {
await fetch('/api/p2p/challenge', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Device': dev.deviceId + '.' + dev.secret },
body: JSON.stringify({ answers }),
}).catch(() => {});
}
return j;
}
async function syncOnce() {
if (!window.OPFS || !window.OPFS.isSupported() || !window.DeviceDB || !window.Sha256) return null;
if (navigator.onLine === false) return null;
const cfg = await config();
if (!cfg || !cfg.enabled) return null;
const dev = await ensureDevice();
const recs = await reconcile();
await hashPending(recs);
const res = await report(recs, dev);
lastSync = Date.now();
return res;
}
// One sync at a time; a request during a sync schedules exactly one more.
function sync() {
if (running) { again = true; return running; }
running = syncOnce().catch(() => null).finally(() => {
running = null;
if (again) { again = false; sync(); }
});
return running;
}
function changed() {
clearTimeout(timer);
timer = setTimeout(sync, 5000);
}
function start(h) {
hooks = { ...hooks, ...(h || {}) };
const idle = window.requestIdleCallback || ((fn) => setTimeout(fn, 1));
setTimeout(() => idle(() => sync()), 8000);
document.addEventListener('visibilitychange', () => {
if (document.visibilityState === 'visible' && Date.now() - lastSync > RESYNC_MS) sync();
});
}
window.P2PClient = { start, changed, sync, device, authHeaders, config };
}());
```

122
plans/queue/014-p2p-presence-hub-ceced8.md Normal file
View File

@@ -0,0 +1,122 @@
---
id: 014-p2p-presence-hub-ceced8
title: Add the /ws/p2p presence and signalling hub and the holders endpoint
created: 2026-09-29
depends_on: [013-device-identity-and-holdings-3ba493]
est_files: 6
---
# 014 — Presence + signalling hub, holders endpoint
## Objective
Implements flows 4–5 of `docs/p2p-architecture.md`. After this plan:
- `/ws/p2p` is a websocket hub (`server/p2p-hub.js`). A device authenticates with its
FIRST message `{type:'auth', device, secret}` (never in the URL — proxies log URLs),
gets `{type:'hello', peer}`, and is **online** while the socket is open (memory only).
It relays `{type:'signal', to, data}` between two authenticated online devices.
- `GET /api/p2p/holders?v=<videoId>` (or `?cid=`) lists each verified cid with its
holders: opaque `peer`, `online`, `lastVerifiedAt`, `stale`, `trust`, plus `serverHas`
and counts. Holder rows are NEVER hidden for age — `stale` only flags them.
Devices with sharing off are not listed. Device ids never appear in the payload.
- `P2PClient` keeps the socket open while sharing or receiving is on (both default ON),
with a 50 s keepalive (the server closes idle sockets after 120 s), and exposes
`onMessage(type, fn)`, `signal(to, data)`, `peer()`, `isConnected()`.
Pre-tested: hub/payload 3/3 (`bun:test`); two real browsers exchanged a signal through a
real Bun server (`plans/harness/presence-*.js`).
## Context the executor must NOT rediscover
- `plans/patches/014-p2p-hub-new.diff` creates `server/p2p-hub.js` (exports `createP2pHub`,
`holdersPayload`, `peerIdOf`) and `server/p2p-hub.test.js`.
- `plans/patches/014-p2p-client-presence.diff` patches `frontend/p2p-client.js` (from plan 013).
- `server/server.js` (~line 1778-1790):
```js
const remote = createRemoteHub({ requireSameNetwork: process.env.REMOTE_SAME_NETWORK === '1' });
const party = createPartyHub();
// Bun allows ONE websocket handler per server: party sockets are tagged
// (ws.data.hub === 'party'), everything else belongs to the remote relay.
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket : remote.websocket);
```
- `server/server.js` `Bun.serve({ … fetch(req, server) {` (~line 1926):
```js
const path = new URL(req.url).pathname;
if (path === '/ws/remote') return remote.upgrade(req, server, clientIpOf(req, server));
if (path === '/ws/party') return party.upgrade(req, server, clientIpOf(req, server));
return app.fetch(req, server);
```
- Plan 013 added in server.js: `async function fileForCid(cid)` and
`const p2p = registerP2pRoutes(app, { cfg: P2P, p2pDb, fileForCid, sha256Range });`
(`p2p.gate` is a Hono middleware answering 404 when P2P is disabled).
## Steps
1. From the repo root:
```bash
git apply plans/patches/014-p2p-hub-new.diff
git apply plans/patches/014-p2p-client-presence.diff
```
STOP and report if either fails.
2. `server/package.json` "test" script — append ` && bun test ./p2p-hub.test.js`.
3. `server/server.js` — add import `import { createP2pHub, holdersPayload } from './p2p-hub.js';`
4. `server/server.js` — directly after `const p2p = registerP2pRoutes(app, { … });` add:
```js
const p2pHub = createP2pHub({ getDevice: p2pDb.getDevice, enabled: () => P2P.enabled });
// GET /api/p2p/holders?v=<videoId>|cid=<sha256> — who holds a copy. Rows are
// persistent; `stale` flags a holder not re-verified for P2P_STALE_DAYS.
app.get('/api/p2p/holders', p2p.gate, async (c) => {
const v = (c.req.query('v') || '').trim();
const cid = (c.req.query('cid') || '').trim().toLowerCase();
const hasCid = /^[0-9a-f]{64}$/.test(cid);
if (!hasCid && !/^[A-Za-z0-9_-]{6,64}$/.test(v)) return c.json({ ok: false, error: 'missing v or cid' }, 400);
const payload = await holdersPayload({
videoId: v, cid: hasCid ? cid : null, p2pDb, isOnline: p2pHub.isOnline,
staleDays: P2P.staleDays, serverHas: fileForCid,
});
return c.json(payload, 200, { 'Cache-Control': 'no-store' });
});
```
5. `server/server.js` — replace the `pickHub` line with:
```js
// …and P2P sockets are tagged ws.data.hub === 'p2p' (p2p-hub.js).
const pickHub = (ws) => (ws.data && ws.data.hub === 'party' ? party.websocket
: ws.data && ws.data.hub === 'p2p' ? p2pHub.websocket : remote.websocket);
```
(`p2pHub` is declared later in the file; that is fine — `pickHub` only runs once sockets exist.)
6. `server/server.js` `Bun.serve` fetch — after the `/ws/party` line add:
` if (path === '/ws/p2p') return p2pHub.upgrade(req, server);`
7. Copy nothing else; `frontend/index.html` / `sw.js` need no change (no new frontend files).
## Out of scope / do NOT touch
- `remote.js`, `party.js` and their behaviour. No UI (plan 015). No file transfer (plan 017).
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/presence-server.js >/tmp/ytp014.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node presence-check.mjs
kill $SRV; true
```
Expected: hub `3 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"peers":[true,true],"distinct":true,"sent":true,"got":[{"from":true,"data":{"hi":1}}]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

184
plans/queue/015-availability-ui-and-settings-3b9397.md Normal file
View File

@@ -0,0 +1,184 @@
---
id: 015-availability-ui-and-settings-3b9397
title: Show peer availability with stale markers and add Sharing settings
created: 2026-09-29
depends_on: [014-p2p-presence-hub-ceced8]
est_files: 6
---
# 015 — Availability line + Sharing settings
## Objective
Implements the UI half of flow 5 and the settings of `docs/p2p-architecture.md`.
After this plan:
- Under the now-playing title a small line shows peer availability, e.g.
`📡 On 3 devices · 1 online now · last checked 2d ago`. Hovering lists each holder
(`peer · online/offline · checked Xd ago (stale) · spot-checked`). When EVERY holder
is older than the server's `staleDays`, the line says `(not checked recently)` and is
styled dimmer — holders are **never hidden for age**. No holders → no line.
- Settings gets a "Sharing (peer-to-peer)" group: **Share my saved videos** and
**Get videos from other devices**, both ON by default, plus a status row
(registered / online peer id, verified/saved counts).
Pre-tested: `frontend/p2p-core.test.js` 5/5.
## Context the executor must NOT rediscover
- `plans/patches/015-p2p-core-new.diff` creates `frontend/p2p-core.js` (`window.P2PCore`:
`ago`, `formatAvailability(payload, now)`, `pickPeers(payload, cid)`) and its node test.
- `frontend/index.html:223-227`:
```html
<div id="nowPlayingMeta" class="now-meta hidden">
<div class="np-text">
<div class="np-title" id="npTitle"></div>
<div class="np-channel" id="npChannel"></div>
</div>
```
- `frontend/app.js` `function updateNowPlayingActions()` (~line 2389) begins
`if (!current || !current.meta) return; const id = current.meta.id;` and ends with the
save-button `if/else` chain followed by `}`.
- `frontend/app.js` `async function renderSettings()` (~line 7321). Its template contains:
```html
<div class="set-group">
<div class="set-group-title">Offline cache</div>
```
and further down the handlers:
```js
$('setAutoPreload').addEventListener('change', (e) => {
data.settings.autoPreload = e.target.checked;
persist();
if (e.target.checked) data.playlists.forEach(preloadPlaylist);
});
```
- `frontend/styles.css:737` — `.np-actions { display: flex; gap: 8px; flex-shrink: 0; }`;
the dim text colour token is `var(--text-dim)`.
- `window.P2PClient` (plans 013/014): `device()`, `isConnected()`, `peer()`, `sync()`.
`window.DeviceDB.listFiles()` (plan 012).
## Steps
1. From the repo root: `git apply plans/patches/015-p2p-core-new.diff` (STOP on failure).
2. `frontend/index.html` — after `<div class="np-channel" id="npChannel"></div>` add:
` <div class="np-avail hidden" id="npAvail" aria-live="polite"></div>`
and add ` <script src="p2p-core.js"></script>` directly after the `p2p-client.js` script line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-core.js',` directly after `'/p2p-client.js',`.
4. `frontend/styles.css` — directly after the `.np-actions { … }` line add:
```css
.np-avail { font-size: 12px; color: var(--text-dim); margin-top: 4px; cursor: default; }
.np-avail.stale { opacity: 0.6; font-style: italic; }
```
5. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` add:
```js
// P2P availability under the title (docs/p2p-architecture.md flow 5).
// Cached 60 s per video so the frequent updateNowPlayingActions() calls
// don't hammer the server.
const availCache = new Map(); // videoId -> { at, payload }
async function renderAvailability(id) {
const el = $('npAvail');
if (!el) return;
const off = data.settings.p2pShare === false && data.settings.p2pReceive === false;
if (!WEB || !window.P2PCore || off || navigator.onLine === false || String(id).startsWith('edit_')) {
el.classList.add('hidden');
return;
}
let hit = availCache.get(id);
if (!hit || Date.now() - hit.at > 60_000) {
hit = { at: Date.now(), payload: null };
availCache.set(id, hit);
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
hit.payload = r.ok ? await r.json() : null;
} catch { /* offline / disabled — no line */ }
if (availCache.size > 200) availCache.delete(availCache.keys().next().value);
}
if (!current || !current.meta || current.meta.id !== id) return; // moved on meanwhile
const f = window.P2PCore.formatAvailability(hit.payload, Date.now());
if (!f) { el.classList.add('hidden'); return; }
el.textContent = f.text;
el.title = f.title;
el.classList.toggle('stale', f.stale);
el.classList.remove('hidden');
}
```
6. `frontend/app.js` `updateNowPlayingActions` — directly after `const id = current.meta.id;` add
`renderAvailability(id);`
7. `frontend/app.js` `renderSettings` template — directly BEFORE the
`<div class="set-group">` whose title is `Offline cache`, insert:
```html
<div class="set-group">
<div class="set-group-title">Sharing (peer-to-peer)</div>
<label class="set-row">
<span>
Share my saved videos
<small>Other devices can download videos you saved when YouTube and the server can't provide them. Files are verified by their hash. On by default.</small>
</span>
<input id="setP2pShare" type="checkbox" ${data.settings.p2pShare !== false ? 'checked' : ''} />
</label>
<label class="set-row">
<span>
Get videos from other devices
<small>When a video is gone from YouTube and the server, download a verified copy from a device that has it. On by default.</small>
</span>
<input id="setP2pReceive" type="checkbox" ${data.settings.p2pReceive !== false ? 'checked' : ''} />
</label>
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
8. `frontend/app.js` `renderSettings` — directly after the `$('setAutoPreload').addEventListener(…);`
block add:
```js
$('setP2pShare').addEventListener('change', (e) => {
data.settings.p2pShare = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync(); // share off → withdraws every holding
});
$('setP2pReceive').addEventListener('change', (e) => {
data.settings.p2pReceive = e.target.checked;
persist();
if (window.P2PClient) window.P2PClient.sync();
});
(async () => {
const info = $('p2pDeviceInfo');
const cnt = $('p2pHoldingCount');
if (!info || !cnt) return;
const P = window.P2PClient;
const d = P && P.device();
info.textContent = !d ? 'Not registered yet'
: P.isConnected() ? `Online · peer ${P.peer()}` : 'Registered · not connected';
const files = window.DeviceDB ? await window.DeviceDB.listFiles() : [];
cnt.textContent = `${files.filter((f) => f.state === 'verified').length} verified / ${files.length} saved`;
})();
```
## Out of scope / do NOT touch
- No download button yet (plan 017 adds "Get from a device"). Do not change `p2p-client.js`.
- Do not show availability for edited cuts (`edit_…`).
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-core.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-core.js" frontend/index.html frontend/sw.js
grep -c "setP2pShare\|setP2pReceive\|renderAvailability(id)" frontend/app.js
```
Expected: `FRONT_OK`; `# fail 0`; `1` and `1`; app.js grep ≥ `5`.
Manual (optional, needs a running server with P2P data): open the app, play a video that has holders,
see the 📡 line; Settings shows the Sharing group with both boxes ticked.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

150
plans/queue/016-intake-and-server-verification-cfe031.md Normal file
View File

@@ -0,0 +1,150 @@
---
id: 016-intake-and-server-verification-cfe031
title: Let a device hand a file to the server for hashing and validation
created: 2026-09-29
depends_on: [015-availability-ui-and-settings-3b9397]
est_files: 9
---
# 016 — Intake: server-side verification of device files
## Objective
Implements flow 7 of `docs/p2p-architecture.md` — the owner's rule "a file must first
be downloaded by the server and checked before its hash is added to the server DB".
After this plan:
- `POST /api/p2p/intake` (device auth) opens a 30-min ticket; `PUT /api/p2p/intake/:ticket`
streams the bytes into `P2P_INTAKE_DIR` (never served). The SERVER hashes while writing,
a claimed cid must match, `validateMedia()` must pass, then `admitFile()` (malware scan
only when `P2P_MALWARE_SCAN=1`, off by default). Only then does the cid enter
`p2p_content` (`origin 'intake'`), and the uploader becomes a holder (`trust 'challenged'`).
- If the server has no copy of that video, the file is adopted into the media cache
byte-for-byte (`media.adoptFile`, new) so it can be streamed again. Otherwise deleted.
- Settings → Sharing shows "Verify & share N saved videos" for saves whose hash the server
doesn't know yet (old saves, fallback-path saves); one tap uploads them one by one.
Nothing uploads automatically in this plan.
Pre-tested: intake 4/4 (valid, hash-mismatch, truncated, non-media, size limits, ticket
reuse, scan-infected), media-cache 27/27 (incl. `adoptFile`), and the browser flow
(unknown → contribute → accepted) in Chromium.
## Context the executor must NOT rediscover
- Patches (apply in this order from the repo root):
- `plans/patches/016-p2p-intake-new.diff` → new `server/p2p-intake.js` (`registerIntakeRoutes`) + `server/p2p-intake.test.js`
- `plans/patches/016-media-cache-adopt.diff` → `adoptFile(id, src, { sha256, probe, meta })` in `server/media-cache.js` + a test
- `plans/patches/016-client-intake.diff` → `P2PClient.contribute(videoId, { cid, title, channel })`,
`P2PClient.unknownVideos()` in `frontend/p2p-client.js`; `OPFS.getFileObject(videoId)` in `frontend/opfs.js`
- `server/server.js` imports `{ createMediaCache, HIGH, LOW } from './media-cache.js'`; `FFMPEG`
const exists; ffprobe path is `process.env.FFPROBE_PATH || 'ffprobe'` (used in createMediaCache).
- Plan 013/014 added in server.js: `const p2p = registerP2pRoutes(app, …)` (exposes `gate`,
`requireDevice`) and `const p2pHub = createP2pHub(…)`. Plan 009 imported `admitFile`, `P2P`, `* as p2pDb`.
- Plan 015 added to `renderSettings()` the Sharing group ending with:
```html
<div class="set-row">
<span>This device<small id="p2pDeviceInfo">…</small></span>
<span id="p2pHoldingCount" class="set-stat">…</span>
</div>
</div>
```
and an `(async () => { const info = $('p2pDeviceInfo'); … cnt.textContent = …; })();` block.
- Settings buttons use `class="btn"`. `videoById(id)` (app.js ~7301) returns the known video object
(title/channel) or undefined. `toast(msg)` shows a toast.
## Steps
1. Apply the three patches (STOP and report on any failure):
```bash
git apply plans/patches/016-p2p-intake-new.diff
git apply plans/patches/016-media-cache-adopt.diff
git apply plans/patches/016-client-intake.diff
```
2. `server/package.json` "test" script — append ` && bun test --timeout 60000 ./p2p-intake.test.js`.
3. `server/server.js`:
a. change the media-cache import to `import { createMediaCache, HIGH, LOW, validateMedia } from './media-cache.js';`
b. add `import { registerIntakeRoutes } from './p2p-intake.js';`
c. directly after the `/api/p2p/holders` route (plan 014) add:
```js
// Device → server intake: the server hashes, validates (and scans when
// P2P_MALWARE_SCAN=1) before a cid is admitted. docs/p2p-architecture.md flow 7.
const intake = registerIntakeRoutes(app, {
cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile,
validateMedia: (path, expected, opts) => validateMedia(path, expected,
{ ...opts, ffmpeg: FFMPEG, ffprobe: process.env.FFPROBE_PATH || 'ffprobe' }),
adopt: (videoId, path, info) => media.adoptFile(videoId, path, info),
});
```
4. `docker-compose.yml` — under the P2P env lines from plan 008 add the commented lines:
```yaml
# P2P_INTAKE_DIR: "/app/data/p2p-intake" # quarantine for device uploads (never served)
# P2P_INTAKE_MAX_BYTES: "3221225472" # 3 GiB
```
5. `frontend/app.js` `renderSettings` template — directly BEFORE the closing `</div>` of the
Sharing group (i.e. after the `This device` row), insert:
```html
<div class="set-row hidden" id="p2pContributeRow">
<span>
Saved videos the server can't verify yet
<small>Uploading lets the server check them (hash + media validation) so other devices can get them. Uses your upload bandwidth.</small>
</span>
<button id="p2pContributeBtn" class="btn">Verify &amp; share</button>
</div>
```
6. `frontend/app.js` `renderSettings` — inside the plan-015 `(async () => { … })();` block, after the
`cnt.textContent = …;` line, add:
```js
const unknown = window.P2PClient ? await window.P2PClient.unknownVideos() : [];
const row = $('p2pContributeRow');
const btn = $('p2pContributeBtn');
if (row && btn && unknown.length) {
row.classList.remove('hidden');
btn.textContent = `Verify & share ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`;
btn.onclick = async () => {
btn.disabled = true;
let ok = 0;
for (let i = 0; i < unknown.length; i++) {
const v = videoById(unknown[i]) || {};
btn.textContent = `Uploading ${i + 1} of ${unknown.length}…`;
const r = await window.P2PClient.contribute(unknown[i], { title: v.title || '', channel: v.channel || '' });
if (r && r.ok) ok++;
}
toast(`Verified ${ok} of ${unknown.length} saved video${unknown.length === 1 ? '' : 's'}`);
btn.disabled = false;
row.classList.add('hidden');
};
}
```
## Out of scope / do NOT touch
- No automatic uploads (plan 018 adds server-requested ones). No UI for the admin (plan 019).
- Never serve files from `P2P_INTAKE_DIR`; never add a `/api/p2p/intake` GET.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — install it (apt-get install -y ffmpeg); intake/media tests need it"
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/app.js frontend/p2p-client.js frontend/opfs.js && echo FRONT_OK
cd server
bun ../plans/harness/intake-server.js >/tmp/ytp016.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 90 node intake-check.mjs
kill $SRV; true
```
Expected: intake `4 pass`; media-cache `27 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`;
browser JSON `{"firstUnknown":1,"unknown":["upAAAAAAAA9"],"contribute":{"ok":true,"adopted":false,"cidOk":true},"secondAccepted":1,"secondUnknown":0}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

179
plans/queue/017-peer-transfer-1faaa7.md Normal file
View File

@@ -0,0 +1,179 @@
---
id: 017-peer-transfer-1faaa7
title: Download a verified file from another device over WebRTC
created: 2026-09-29
depends_on: [016-intake-and-server-verification-cfe031]
est_files: 6
---
# 017 — Peer download over WebRTC
## Objective
Implements flow 6 of `docs/p2p-architecture.md`. After this plan:
- `window.P2PTransfer` serves this device's saved files to other devices (only while
`settings.p2pShare !== false`, one upload at a time) and downloads from them over a
WebRTC data channel, signalled through `/ws/p2p` (plan 014). STUN only.
- The receiver writes through `p2p-recv-worker.js` into `videos/<id>.p2p.part` while
hashing, and renames to `<id>.mp4` ONLY when the SHA-256 equals the cid. A wrong,
corrupted or short file leaves nothing behind.
- When a video fails to load (YouTube and the server both failing), and
`settings.p2pReceive !== false`, the player shows **📡 Get it from a device (N online)**
next to **↻ Retry**. On success the video is saved offline, registered as `verified`
(so this device becomes a holder at its next report) and plays.
Pre-tested in Chromium with two browser contexts and a real hub
(`plans/harness/transfer-*.js`): 5 MiB file delivered and verified in ~2.5 s; unknown cid →
`peer declined: not here`; holder with a corrupted same-size file → `content hash mismatch`
and no file kept.
## Context the executor must NOT rediscover
- `plans/patches/017-p2p-transfer-new.diff` creates `frontend/p2p-transfer.js` and
`frontend/p2p-recv-worker.js`.
- Script order in `frontend/index.html` after plan 015:
`… sha256.js, device-db.js, p2p-client.js, p2p-core.js, async-guard.js, sw-update.js, app.js`.
- `frontend/sw.js` `SHELL` has `'/p2p-core.js',` (plan 015).
- `frontend/app.js`:
- `Player.loadVideo` catch block (~line 1735):
```js
} catch (err) {
showSpinner(false);
toast('⚠ ' + err.message);
// Show retry button in the player pane
const retry = els.playerPane.querySelector('.retry-btn');
if (retry) retry.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn';
btn.textContent = '↻ Retry';
btn.addEventListener('click', () => {
btn.remove();
Player.loadVideo(videoObj, { preferStream, resume, reveal, nocache });
});
els.playerPane.appendChild(btn);
}
```
- globals/helpers: `downloading` (Set, ~line 372), `cachedIds`, `cacheMutations`,
`markCardCacheState(id, state)` (~1351), `updateDownloadBadge()` (~1313), `renderSidebar()`,
`recordDeviceFile(id, res)` (plan 012), `toast()`.
- `boot()` has (plan 013):
```js
if (WEB && window.P2PClient) {
window.P2PClient.start({ getSettings: () => data.settings, getProfile: () => (data.profile && data.profile.name) || '' });
}
```
- `window.P2PCore.pickPeers(payload, cid)` → `[{ peer, cid, size }]` online holders (plan 015).
## Steps
1. From the repo root: `git apply plans/patches/017-p2p-transfer-new.diff` (STOP on failure).
2. `frontend/index.html` — add ` <script src="p2p-transfer.js"></script>` directly after the `p2p-core.js` line.
3. `frontend/sw.js` `SHELL` — add `'/p2p-transfer.js',` and `'/p2p-recv-worker.js',` directly after `'/p2p-core.js',`.
4. `frontend/styles.css` — append at the end of the file:
```css
/* P2P: "Get it from a device" next to Retry (plan 017) */
.peer-btn { margin-left: 8px; }
```
5. `frontend/app.js` `boot()` — inside the `if (WEB && window.P2PClient) { … }` block, after the
`start(...)` call add:
```js
if (window.P2PTransfer) window.P2PTransfer.start({ canShare: () => data.settings.p2pShare !== false });
```
6. `frontend/app.js` — directly ABOVE `function updateNowPlayingActions() {` (next to plan 015's
`renderAvailability`) add:
```js
// Fetch a verified copy from another device (docs/p2p-architecture.md flow 6).
// Download-then-play: on success the file is saved offline like any save.
async function getFromPeers(videoObj, c, peers, onProgress) {
const id = videoObj.id;
downloading.add(id);
markCardCacheState(id, 'downloading');
updateDownloadBadge();
try {
const res = await window.P2PTransfer.download({ videoId: id, cid: c.cid, size: c.size, peers, onProgress });
if (res.ok) {
cachedIds.add(id);
cacheMutations++;
recordDeviceFile(id, { sha256: res.sha256, expectedSha: c.cid, size: res.size });
toast(`Saved “${videoObj.title || id}” from another device ✓`);
}
return res;
} finally {
downloading.delete(id);
markCardCacheState(id, cachedIds.has(id) ? 'cached' : 'none');
updateDownloadBadge();
renderSidebar();
}
}
// Offered when a video won't load: only if some device holding it is online now.
async function offerPeerDownload(videoObj) {
if (!WEB || !window.P2PTransfer || !window.P2PCore || data.settings.p2pReceive === false) return;
const id = videoObj && videoObj.id;
if (!id || videoObj.custom || cachedIds.has(id)) return;
let payload = null;
try {
const r = await fetch(`/api/p2p/holders?v=${encodeURIComponent(id)}`);
payload = r.ok ? await r.json() : null;
} catch { return; }
const c = payload && Array.isArray(payload.cids) ? payload.cids[0] : null;
const peers = c ? window.P2PCore.pickPeers(payload, c.cid) : [];
if (!peers.length || !els.playerPane.querySelector('.retry-btn')) return; // nobody online / user moved on
const old = els.playerPane.querySelector('.peer-btn');
if (old) old.remove();
const btn = document.createElement('button');
btn.className = 'retry-btn peer-btn';
btn.textContent = `📡 Get it from a device (${peers.length} online)`;
btn.addEventListener('click', async () => {
btn.disabled = true;
const res = await getFromPeers(videoObj, c, peers, (got, total) => {
btn.textContent = `📡 ${Math.min(99, Math.round((got / total) * 100))}%…`;
});
if (res.ok) {
els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());
Player.loadVideo(videoObj);
} else {
btn.disabled = false;
btn.textContent = '📡 Try again';
toast('⚠ ' + res.error);
}
});
els.playerPane.appendChild(btn);
}
```
7. `frontend/app.js` `Player.loadVideo` catch block — directly after `els.playerPane.appendChild(btn);`
add `offerPeerDownload(videoObj);`. Also, in the Retry button's click handler, change
`btn.remove();` to
`els.playerPane.querySelectorAll('.retry-btn').forEach((b) => b.remove());` so the peer button
goes away with it.
## Out of scope / do NOT touch
- No TURN server, no progressive (streaming) playback from peers, no partial seeding.
- Don't change `p2p-client.js` or the server.
## Verification
```bash
cd /home/user/ytplayer && node --check frontend/app.js frontend/p2p-transfer.js frontend/p2p-recv-worker.js && echo FRONT_OK
node --test frontend/*.test.js 2>&1 | grep -E "^# (pass|fail)"
grep -c "p2p-transfer.js\|p2p-recv-worker.js" frontend/index.html frontend/sw.js
cd server && bun install >/dev/null 2>&1
bun ../plans/harness/transfer-server.js >/tmp/ytp017.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node transfer-check.mjs 2>&1 | grep -v "status of 500"
kill $SRV; true
```
Expected: `FRONT_OK`; `# fail 0`; index.html `1`, sw.js `2`; browser JSON containing
`"good":{"ok":true`, `"list":[["gotAAAAAAAA",5243657]]`, `"bad":{"ok":false,"error":"peer declined: not here"}`,
`"corrupt":{"r":{"ok":false,"error":"content hash mismatch"},"names":["gotAAAAAAAA.mp4"]}`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

120
plans/queue/018-server-rehydrate-from-peer-4fb8bd.md Normal file
View File

@@ -0,0 +1,120 @@
---
id: 018-server-rehydrate-from-peer-4fb8bd
title: Restore an evicted server copy from an online holder
created: 2026-09-29
depends_on: [017-peer-transfer-1faaa7]
est_files: 6
---
# 018 — Rehydrate the server from a device
## Objective
Implements flow 8 of `docs/p2p-architecture.md`: the owner's "the original source can go
offline, the video stays reachable from devices". After this plan, when `/api/streams`
fails at the source (removed/private video, yt-dlp failing) AND the server has no ready
copy AND a device holding a verified cid of that video is online with sharing on:
- the server sends that ONE device `{type:'upload-request', videoId, cid}` over `/ws/p2p`
(at most once per video per 10 min) and answers
`503 { ok:false, restoring:true, error:'…try again in a minute.' }`;
- the device uploads the file through intake with `restore: true`; intake accepts a known
cid again ONLY when the server no longer holds those bytes, re-hashes + re-validates,
and adopts it into the media cache — the next play is served from the server again.
Pre-tested: rehydrator + restore intake (`bun:test` hub 4/4, intake 5/5) and the whole loop in
Chromium (`plans/harness/rehydrate-*.js`: first request → `restoring:true`, device uploads,
server adopts the exact cid).
## Context the executor must NOT rediscover
- Patches (apply in order):
- `plans/patches/018-server-rehydrate.diff` — `server/p2p-hub.js` gains
`createRehydrator({ p2pDb, hub, hasServerCopy, enabled, now })`; `server/p2p-intake.js` gains the
`restore` body flag and a `serverHasCid(cid)` dep; tests for both.
- `plans/patches/018-client-restore.diff` — `frontend/p2p-client.js` answers `upload-request`
(only when sharing is on, only for the exact cid it holds, one at a time) and `contribute()`
accepts `restore`.
- `server/server.js` `/api/streams` handler ends (~line 623):
```js
} catch (err) {
return c.json({ ok: false, error: err.message }, 500);
}
});
```
(this catch covers the yt-dlp resolve path; the cached-copy path returned earlier).
- Plans 013–016 created in server.js: `fileForCid(cid)`, `const p2p = registerP2pRoutes(…)`,
`const p2pHub = createP2pHub(…)`, the `/api/p2p/holders` route, and
`const intake = registerIntakeRoutes(app, { cfg: P2P, p2pDb, gate: p2p.gate, requireDevice: p2p.requireDevice, admitFile, validateMedia: …, adopt: … });`
- `media.getReady(videoId)` resolves the ready row or null.
## Steps
1. From the repo root (STOP on failure):
```bash
git apply plans/patches/018-server-rehydrate.diff
git apply plans/patches/018-client-restore.diff
```
2. `server/server.js` — change the import from `./p2p-hub.js` to
`import { createP2pHub, holdersPayload, createRehydrator } from './p2p-hub.js';`
3. `server/server.js` — in the `registerIntakeRoutes(app, { … })` options add
`serverHasCid: async (cid) => !!(await fileForCid(cid)),`
4. `server/server.js` — directly after the `const intake = registerIntakeRoutes(…);` statement add:
```js
// Source gone + server copy evicted → ask one online holder to send it back
// through intake (docs/p2p-architecture.md flow 8).
const p2pRehydrate = createRehydrator({
p2pDb, hub: p2pHub, enabled: () => P2P.enabled,
hasServerCopy: async (id) => !!(await media.getReady(id).catch(() => null)),
});
```
5. `server/server.js` `/api/streams` — replace the final catch block shown in Context with:
```js
} catch (err) {
// The source failed. If a device holds a verified copy, ask it to send one
// to the server so the video comes back (P2P flow 8).
let restoring = false;
try { restoring = await p2pRehydrate(videoId); } catch { /* best effort */ }
if (restoring) {
return c.json({
ok: false, restoring: true,
error: 'This video is unavailable at the source — a device that has it is sending a copy to the server. Try again in a minute.',
}, 503);
}
return c.json({ ok: false, error: err.message }, 500);
}
```
(`p2pRehydrate` is declared further down the file; it is only called at request time, after startup.)
## Out of scope / do NOT touch
- The client error UI (the existing toast + Retry + plan-017 peer button already cover it).
- Do not ask more than one device per video per 10 minutes; do not upload without sharing on.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
bun test ./p2p-hub.test.js 2>&1 | tail -3
bun test --timeout 60000 ./p2p-intake.test.js 2>&1 | tail -3
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
cd .. && node --check frontend/p2p-client.js && echo FRONT_OK
cd server
bun ../plans/harness/rehydrate-server.js >/tmp/ytp018.log 2>&1 & SRV=$!; sleep 3
cd ../plans/harness && (npm ls playwright >/dev/null 2>&1 || npm i --no-save playwright >/dev/null 2>&1); timeout 120 node rehydrate-check.mjs
kill $SRV; true
```
Expected: hub `4 pass`; intake `5 pass`; every file `0 fail`; `SERVER_OK`; `FRONT_OK`; browser JSON
`{"accepted":1,"first":{"ok":false,"restoring":true,"error":"restoring from a device"},"adopted":[["goneAAAAAAA",true]],"second":{"ok":false,"error":"source unavailable"}}`
(the harness's stand-in `/api/streams` has no media cache, so the second answer is expected).
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

169
plans/queue/019-admin-p2p-panel-4dc623.md Normal file
View File

@@ -0,0 +1,169 @@
---
id: 019-admin-p2p-panel-4dc623
title: Add a P2P panel to the admin page
created: 2026-09-29
depends_on: [018-server-rehydrate-from-peer-4fb8bd]
est_files: 3
---
# 019 — Admin P2P panel
## Objective
Give the admin one place to see and steer peer-to-peer sharing
(`docs/p2p-architecture.md`). After this plan `/admin` has a **Peer-to-peer** section showing:
- config: P2P on/off, malware scan on/off (with a note that it is OFF by default and hashing +
validation always run), stale-after days, retention thresholds;
- counts: verified content, revoked, devices, active holder rows, cids with ≥1 holder,
devices online now, open/active intake uploads;
- the 30 most recently verified files (video, cid prefix, origin, scan, size, verified at,
holder count) with a **Revoke** button (revoked cids are never offered to or accepted
from devices again — plan 013 only accepts `verified` cids).
## Context the executor must NOT rediscover
- Admin auth middleware: `notes.requireAdminOrToken` (returned by `registerNoteRoutes`,
`server/server.js` ~line 1823). Used like `app.get('/api/admin/media', requireAdminOrToken, async (c) => …)`.
- In server.js (plans 008–018): `P2P` (config), `p2pDb` (module), `p2pHub.onlineCount()`,
`intake.openTickets()`, `intake.active()`.
- `server/p2p-db.js` has `rowsOf`, `p2pStats()`, `revokeContent(cid)`.
- `frontend/admin.html`:
- sections are `<section>…</section>` inside `<div id="app">`; the "Recent edits" section
(~line 190) starts with
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Recent edits</h2>
```
- helpers inside the script IIFE: `$(id)`, `esc(s)`, `api(path, opts)` (JSON in/out,
`opts.body` is JSON-encoded), classes `tbl`, `scroll`, `muted`, `row`, `spacer`.
- `async function boot()` calls `loadTokens(); loadRevs(); loadUploads();`.
## Steps
1. `server/p2p-db.js` — append:
```js
// Admin panel (plan 019): newest verified/revoked files with their holder counts.
export async function recentContent(limit = 30) {
const r = await db.execute({
sql: `SELECT c.cid, c.video_id, c.size, c.height, c.vcodec, c.origin, c.status, c.scan, c.verified_at, c.meta,
(SELECT COUNT(*) FROM p2p_holders h WHERE h.cid = c.cid AND h.status = 'active') AS holders
FROM p2p_content c ORDER BY c.verified_at DESC LIMIT ?`,
args: [limit],
});
return rowsOf(r).map((x) => ({ ...x, holders: Number(x.holders) || 0 }));
}
```
2. `server/server.js` — directly after the `const p2pRehydrate = createRehydrator({…});` statement add:
```js
// Admin: P2P overview + revoke (frontend/admin.html → Peer-to-peer).
app.get('/api/admin/p2p', notes.requireAdminOrToken, async (c) => c.json({
ok: true,
config: {
enabled: P2P.enabled, malwareScan: P2P.malwareScan, staleDays: P2P.staleDays,
keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
},
stats: { ...(await p2pDb.p2pStats()), online: p2pHub.onlineCount(), intakeOpen: intake.openTickets(), intakeActive: intake.active() },
recent: await p2pDb.recentContent(30),
}, 200, { 'Cache-Control': 'no-store' }));
app.post('/api/admin/p2p/revoke', notes.requireAdminOrToken, async (c) => {
const body = await c.req.json().catch(() => ({}));
const cid = String(body.cid || '').toLowerCase();
if (!/^[0-9a-f]{64}$/.test(cid)) return c.json({ ok: false, error: 'bad cid' }, 400);
await p2pDb.revokeContent(cid);
console.warn(`[p2p] admin revoked ${cid.slice(0, 12)}`);
return c.json({ ok: true });
});
```
3. `frontend/admin.html` — directly BEFORE the "Recent edits" `<section>` insert:
```html
<section>
<div class="row" style="margin-bottom:10px">
<h2 style="margin:0">Peer-to-peer</h2>
<span class="spacer"></span>
<button id="p2pRefreshBtn">↻ Refresh</button>
</div>
<p class="muted">Devices share verified copies with each other (docs/p2p-architecture.md). A file's hash is only
added after the server itself hashed and validated it. The malware scan is off by default
(<code>P2P_MALWARE_SCAN=1</code> to enable). Holders are never expired — ones not re-checked for the
stale period are only marked stale.</p>
<div id="p2pSummary" class="msg"></div>
<div class="scroll"><table class="tbl" id="p2pTable"></table></div>
</section>
```
4. `frontend/admin.html` script — directly ABOVE `async function boot() {` add:
```js
async function loadP2p() {
const j = await api('/api/admin/p2p');
if (!j.ok) { $('p2pSummary').textContent = j.error || 'P2P unavailable'; return; }
const c = j.config, s = j.stats;
$('p2pSummary').textContent =
`P2P ${c.enabled ? 'ON' : 'OFF'} · malware scan ${c.malwareScan ? 'ON' : 'off'} · stale after ${c.staleDays} d · ` +
`keep ≥${c.keepMinViews} views/${c.keepDays} d or played in ${c.keepRecentDays} d — ` +
`${s.content} verified (${s.revoked} revoked) · ${s.devices} devices, ${s.online} online · ` +
`${s.holders} holdings over ${s.heldCids} files · intake ${s.intakeActive} running / ${s.intakeOpen} open`;
const mb = (b) => (Number(b) / 1048576).toFixed(1) + ' MB';
$('p2pTable').innerHTML = '<tr><th>Video</th><th>cid</th><th>Origin</th><th>Scan</th><th>Size</th><th>Verified</th><th>Holders</th><th></th></tr>' +
j.recent.map((r) => {
let title = '';
try { title = JSON.parse(r.meta || '{}').title || ''; } catch { /* no meta */ }
return `<tr${r.status === 'revoked' ? ' class="muted"' : ''}><td>${esc(title || r.video_id)}<br><small>${esc(r.video_id)}</small></td>` +
`<td><code>${esc(r.cid.slice(0, 12))}</code></td><td>${esc(r.origin)}</td><td>${esc(r.scan)}</td><td>${mb(r.size)}</td>` +
`<td>${esc(new Date(Number(r.verified_at)).toLocaleString())}</td><td>${r.holders}</td>` +
`<td>${r.status === 'verified' ? `<button data-revoke="${esc(r.cid)}">Revoke</button>` : 'revoked'}</td></tr>`;
}).join('');
}
$('p2pRefreshBtn').addEventListener('click', loadP2p);
$('p2pTable').addEventListener('click', async (e) => {
const b = e.target.closest('[data-revoke]');
if (!b || !confirm('Revoke this file? Devices will stop sharing it and the server will never accept it again.')) return;
const j = await api('/api/admin/p2p/revoke', { method: 'POST', body: { cid: b.dataset.revoke } });
if (!j.ok) alert(j.error || 'failed');
loadP2p();
});
```
5. `frontend/admin.html` `boot()` — `loadUploads();` appears 3 times in the file; use ONLY the one
inside `async function boot()`, i.e. this exact pair of lines:
```js
loadUploads();
const want = videoIdFrom(new URLSearchParams(location.search).get('v') || '');
```
and insert ` loadP2p();` between them.
## Out of scope / do NOT touch
- No editing of config from the page (env vars stay the source of truth).
- `admin.html` is never cached by the SW — no `sw.js` change.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1 && [ -e public ] || ln -s ../frontend public
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
DBDIR=$(mktemp -d); DB_PATH=$DBDIR/t.db MEDIA_DIR=$DBDIR/media ADMIN_PASSWORD=test-pass PORT=3995 bun server.js >/tmp/ytp019.log 2>&1 & SRV=$!; sleep 4
curl -s -c /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"password":"test-pass"}' http://localhost:3995/api/admin/login
echo
curl -s -b /tmp/ytp019.jar http://localhost:3995/api/admin/p2p | head -c 400; echo
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3995/api/admin/p2p
curl -s -b /tmp/ytp019.jar -H 'Content-Type: application/json' -d '{"cid":"nope"}' http://localhost:3995/api/admin/p2p/revoke
echo
kill $SRV; true
grep -c "loadP2p" ../frontend/admin.html
```
Expected: `SERVER_OK`; every test file `0 fail`; login `{"ok":true…}`; the p2p call returns
`{"ok":true,"config":{"enabled":true,"malwareScan":false,"staleDays":7,…},"stats":{"content":0,…},"recent":[]}`;
without the cookie `401`; bad cid `{"ok":false,"error":"bad cid"}`; grep ≥ `3`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.

166
server/fixtures/innertube-search.json Normal file
View File

@@ -0,0 +1,166 @@
{
"contents": {
"twoColumnSearchResultsRenderer": {
"primaryContents": {
"sectionListRenderer": {
"contents": [
{
"itemSectionRenderer": {
"contents": [
{
"videoRenderer": {
"videoId": "nQWFzMvCfLE",
"title": {
"runs": [
{
"text": "What A Beautiful Name - Hillsong Worship"
}
],
"accessibility": {
"accessibilityData": {
"label": "What A Beautiful Name - Hillsong Worship 5 minutes, 43 seconds"
}
}
},
"ownerText": {
"runs": [
{
"text": "Hillsong Worship",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UC4q12NoPNySbVqwpw4iO5Vg",
"canonicalBaseUrl": "/channel/UC4q12NoPNySbVqwpw4iO5Vg"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "5 minutes, 43 seconds"
}
},
"simpleText": "5:43"
}
}
},
{
"shelfRenderer": {
"title": {
"simpleText": "ignored"
}
}
},
{
"videoRenderer": {
"videoId": "8YKJVx4qxcs",
"title": {
"runs": [
{
"text": "Best Hillsong Worship Songs of All Time | Nonstop Christian Music Playlist"
}
],
"accessibility": {
"accessibilityData": {
"label": "Best Hillsong Worship Songs of All Time | Nonstop Christian Music Playlist 1 hour, 59 minutes"
}
}
},
"ownerText": {
"runs": [
{
"text": "Top Praise and Worship Songs",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UCNp42By0x1viJ1XSCHYV38g",
"canonicalBaseUrl": "/@TopPraiseAndWorshipSongs7"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "1 hour, 59 minutes, 47 seconds"
}
},
"simpleText": "1:59:47"
}
}
},
{
"videoRenderer": {
"videoId": "_1HGZ_9aRhI",
"title": {
"runs": [
{
"text": "Hillsong Worship Best Praise Songs Collection 2023 \u2013 Gospel Christian Songs Of Hillsong Worship"
}
],
"accessibility": {
"accessibilityData": {
"label": "Hillsong Worship Best Praise Songs Collection 2023 \u2013 Gospel Christian Songs Of Hillsong Worship 1 hour, 17 minutes"
}
}
},
"ownerText": {
"runs": [
{
"text": "Hallelujah Harmony",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UC1dEcyKOYiqdHwcVUmbPudw",
"canonicalBaseUrl": "/@HallelujahHarmony68"
}
}
}
]
},
"lengthText": {
"accessibility": {
"accessibilityData": {
"label": "1 hour, 17 minutes, 50 seconds"
}
},
"simpleText": "1:17:50"
}
}
},
{
"videoRenderer": {
"videoId": "LIVEid12345",
"title": {
"runs": [
{
"text": "Live now"
}
]
},
"ownerText": {
"runs": [
{
"text": "Some Church",
"navigationEndpoint": {
"browseEndpoint": {
"browseId": "UCxxxxxxxxxxxxxxxxxxxxxx"
}
}
}
]
}
}
}
]
}
},
{
"continuationItemRenderer": {}
}
]
}
}
}
}
}