--- id: 009-server-content-hash-186e7f title: Hash every validated server copy and register it as verified content created: 2026-09-29 depends_on: [008-p2p-schema-and-config-127966] est_files: 7 --- # 009 — Server content hashes → verified P2P content ## Objective Implements flow 1 of `docs/p2p-architecture.md`. After this plan: - Every copy the media cache promotes (fetch lane AND compression lane) gets a server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are hashed by a background backfill 30 s after boot. - Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted. - `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: `. - `/api/streams`' cached payload gets an additive `data.cid` (web-only, like `data.serverCached`). The media-cache edits were written and tested ahead of time (25/25 media-cache tests pass, incl. 2 new ones); they ship as patch files. ## Context the executor must NOT rediscover - Patches (made against the current tree; `media-cache.js` and its test are untouched by plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`. They add to `createMediaCache()` the options `hashFile` (default `sha256File` from `./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call `onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export `backfillHashes()`. - `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`. - `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers: ```js headers: { 'Content-Type': 'video/mp4', 'Content-Length': String(file.size), 'Content-Disposition': `attachment; filename="${videoId}.mp4"`, 'Cache-Control': 'no-store', 'Access-Control-Allow-Origin': '*', }, ``` - `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns `{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`. - Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`). ## Steps 1. Create `server/hash.js` with exactly: ```js /* hash.js — streaming SHA-256 of files on disk (never loads a whole video). * The hex digest of a validated file is its P2P content id (cid). */ import { createHash } from 'node:crypto'; import { createReadStream } from 'node:fs'; export function sha256File(path) { return new Promise((resolve, reject) => { const h = createHash('sha256'); createReadStream(path, { highWaterMark: 1024 * 1024 }) .on('data', (d) => h.update(d)) .on('error', reject) .on('end', () => resolve(h.digest('hex'))); }); } // Hash of bytes [offset, offset+length) — used for holder range challenges. export function sha256Range(path, offset, length) { return new Promise((resolve, reject) => { if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; } const h = createHash('sha256'); createReadStream(path, { start: offset, end: offset + length - 1 }) .on('data', (d) => h.update(d)) .on('error', reject) .on('end', () => resolve(h.digest('hex'))); }); } ``` 2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A. 3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B. 4. Apply the patches from the repo root: ```bash git apply plans/patches/009-media-cache.diff git apply plans/patches/009-media-cache-test.diff ``` If either fails, STOP and report the error (do not hand-edit). 5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`. 6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from plan 008 may already exist; merge into it): ```js import { admitFile } from './p2p-admit.js'; import { P2P } from './p2p-config.js'; import * as p2pDb from './p2p-db.js'; ``` If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in `main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import. 7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add: ```js // P2P (docs/p2p-architecture.md): every validated copy's server-computed // hash becomes verified content, after the optional malware scan. onReady: (info) => admitFile( { ...info, cid: info.sha256, videoId: info.id, origin: 'server' }, { cfg: P2P, upsertContent: p2pDb.upsertContent }, ), ``` 8. `server/server.js` `cachedDownloadResponse` — add to the headers object: ```js ...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}), ``` 9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add `cid: row.sha256 || null,` and add a comment above the return: `// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.` 10. `Dockerfile` — optional ClamAV, off by default. After the existing `RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add: ```dockerfile # Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by # default: build with --build-arg INSTALL_CLAMAV=1 to include it. ARG INSTALL_CLAMAV=0 RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \ apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \ freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \ fi ``` ## Out of scope / do NOT touch - `validateMedia()` itself, the eviction logic, any frontend file. - Never serve anything from the intake dir; no new routes in this plan. ## Verification ```bash cd /home/user/ytplayer/server && bun install >/dev/null 2>&1 which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)" bun test ./p2p-admit.test.js 2>&1 | tail -4 bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4 bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)" bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js ``` Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail` (media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits. ## Report format (executor: follow exactly) Output ONLY the following, no other prose: 1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`. 2. Raw output of the Verification commands. 3. `Findings:` — max 10 lines. Do not commit. Do not push. Do not touch files outside the Steps. --- ## Appendix A — server/p2p-admit.js ```js /* ============================================================================ * p2p-admit.js — the ONLY way a content id enters p2p_content * (docs/p2p-architecture.md, "Security rules"). * * Callers must already have: the complete file on the server's own disk, its * SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the * optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the * row. The scan command gets the path as its last argument: exit 0 = clean, * 1 = infected (rejected), anything else = scanner error (not admitted now). * ========================================================================== */ import { spawn } from 'node:child_process'; export function scanFile(path, cmd) { const parts = String(cmd).split(/\s+/).filter(Boolean); return new Promise((resolve) => { let child; try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); } catch (e) { resolve({ result: 'error', detail: e.message }); return; } let out = ''; child.stdout.on('data', (d) => { out = (out + d).slice(-2000); }); child.stderr.on('data', (d) => { out = (out + d).slice(-2000); }); child.on('error', (e) => resolve({ result: 'error', detail: e.message })); child.on('close', (code) => resolve( code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code }, )); }); } const CID_RE = /^[0-9a-f]{64}$/; // info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin } // deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console } // → { ok: true, scan } | { ok: false, reason } export async function admitFile(info, deps) { const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps; if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' }; if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' }; let scanResult = 'skipped'; if (cfg.malwareScan) { const r = await scan(info.path, cfg.scanCmd); if (r.result !== 'clean') { log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`); return { ok: false, reason: 'scan ' + r.result }; } scanResult = 'clean'; } await upsertContent({ cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec, acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin, scan: scanResult, now: now(), }); return { ok: true, scan: scanResult }; } ``` ## Appendix B — server/p2p-admit.test.js ```js import { test, expect } from 'bun:test'; import { mkdtempSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { createHash } from 'node:crypto'; import { admitFile, scanFile } from './p2p-admit.js'; import { sha256File, sha256Range } from './hash.js'; const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-')); const file = join(dir, 'f.bin'); const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251)); writeFileSync(file, bytes); const CID = createHash('sha256').update(bytes).digest('hex'); const quiet = { warn() {}, info() {} }; const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o }); const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' }; test('sha256File / sha256Range match node:crypto', async () => { expect(await sha256File(file)).toBe(CID); const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex'); expect(await sha256Range(file, 1000, 65536)).toBe(want); }); test('scan off by default: admitted with scan=skipped', async () => { const rows = []; const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet }); expect(r).toEqual({ ok: true, scan: 'skipped' }); expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' }); }); test('scan on: clean admits, infected and scanner errors do not', async () => { for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) { const rows = []; const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet }); expect(r.ok).toBe(ok); expect(rows.length).toBe(ok ? 1 : 0); } }); test('disabled P2P or a malformed cid never admits', async () => { const rows = []; expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false); expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false); expect(rows.length).toBe(0); }); test('scanFile maps exit codes', async () => { expect((await scanFile(file, 'true')).result).toBe('clean'); expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1 expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error'); expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error'); }); ``` ## Execution log - Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 0. - Orchestrator re-ran Verification: `server/media-cache.js` and `media-cache.test.js` byte-identical to the pre-tested versions; `hash.js`, `p2p-admit.js`, `p2p-admit.test.js` identical; p2p-admit 5 pass; media-cache 25 pass; all 9 server test files 0 fail; `SERVER_OK`; the 3 server.js edits present. - Executor Findings (verbatim): Plan 009 executed successfully. All steps completed: created server/hash.js, server/p2p-admit.js, server/p2p-admit.test.js; applied both media-cache patches without errors (25/25 tests pass); updated package.json test script; updated server.js imports to namespace import for p2pDb; added onReady callback to createMediaCache; added X-Content-SHA256 header to cachedDownloadResponse; added cid field to cachedStreamsPayload; updated Dockerfile with optional ClamAV. Server builds successfully; all 74 tests pass (9 test suites).