josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plan: start 009-server-content-hash-186e7f

Browse Source
This commit is contained in:
Claude
2026-09-30 07:11:55 +00:00
parent 167df35a97
commit b19193841c
2 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

273
plans/queue/009-server-content-hash-186e7f.md
View File

@@ -1,273 +0,0 @@
---
id: 009-server-content-hash-186e7f
title: Hash every validated server copy and register it as verified content
created: 2026-09-29
depends_on: [008-p2p-schema-and-config-127966]
est_files: 7
---
# 009 — Server content hashes → verified P2P content
## Objective
Implements flow 1 of `docs/p2p-architecture.md`. After this plan:
- Every copy the media cache promotes (fetch lane AND compression lane) gets a
server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are
hashed by a background backfill 30 s after boot.
- Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only
when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted.
- `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: <cid>`.
- `/api/streams`' cached payload gets an additive `data.cid` (web-only, like
`data.serverCached`).
The media-cache edits were written and tested ahead of time (25/25 media-cache
tests pass, incl. 2 new ones); they ship as patch files.
## Context the executor must NOT rediscover
- Patches (made against the current tree; `media-cache.js` and its test are untouched by
plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`.
They add to `createMediaCache()` the options `hashFile` (default `sha256File` from
`./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the
file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call
`onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export
`backfillHashes()`.
- `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last
property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`.
- `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers:
```js
headers: {
'Content-Type': 'video/mp4',
'Content-Length': String(file.size),
'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
'Cache-Control': 'no-store',
'Access-Control-Allow-Origin': '*',
},
```
- `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns
`{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`.
- Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`).
## Steps
1. Create `server/hash.js` with exactly:
```js
/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
* The hex digest of a validated file is its P2P content id (cid). */
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
export function sha256File(path) {
return new Promise((resolve, reject) => {
const h = createHash('sha256');
createReadStream(path, { highWaterMark: 1024 * 1024 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
// Hash of bytes [offset, offset+length) — used for holder range challenges.
export function sha256Range(path, offset, length) {
return new Promise((resolve, reject) => {
if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
const h = createHash('sha256');
createReadStream(path, { start: offset, end: offset + length - 1 })
.on('data', (d) => h.update(d))
.on('error', reject)
.on('end', () => resolve(h.digest('hex')));
});
}
```
2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A.
3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B.
4. Apply the patches from the repo root:
```bash
git apply plans/patches/009-media-cache.diff
git apply plans/patches/009-media-cache-test.diff
```
If either fails, STOP and report the error (do not hand-edit).
5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`.
6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from
plan 008 may already exist; merge into it):
```js
import { admitFile } from './p2p-admit.js';
import { P2P } from './p2p-config.js';
import * as p2pDb from './p2p-db.js';
```
If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in
`main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import.
7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add:
```js
// P2P (docs/p2p-architecture.md): every validated copy's server-computed
// hash becomes verified content, after the optional malware scan.
onReady: (info) => admitFile(
{ ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
{ cfg: P2P, upsertContent: p2pDb.upsertContent },
),
```
8. `server/server.js` `cachedDownloadResponse` — add to the headers object:
```js
...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),
```
9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add
`cid: row.sha256 || null,` and add a comment above the return:
`// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.`
10. `Dockerfile` — optional ClamAV, off by default. After the existing
`RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add:
```dockerfile
# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
ARG INSTALL_CLAMAV=0
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
fi
```
## Out of scope / do NOT touch
- `validateMedia()` itself, the eviction logic, any frontend file.
- Never serve anything from the intake dir; no new routes in this plan.
## Verification
```bash
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
bun test ./p2p-admit.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js
```
Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail`
(media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
---
## Appendix A — server/p2p-admit.js
```js
/* ============================================================================
* p2p-admit.js — the ONLY way a content id enters p2p_content
* (docs/p2p-architecture.md, "Security rules").
*
* Callers must already have: the complete file on the server's own disk, its
* SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
* optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
* row. The scan command gets the path as its last argument: exit 0 = clean,
* 1 = infected (rejected), anything else = scanner error (not admitted now).
* ========================================================================== */
import { spawn } from 'node:child_process';
export function scanFile(path, cmd) {
const parts = String(cmd).split(/\s+/).filter(Boolean);
return new Promise((resolve) => {
let child;
try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
catch (e) { resolve({ result: 'error', detail: e.message }); return; }
let out = '';
child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
child.on('close', (code) => resolve(
code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
));
});
}
const CID_RE = /^[0-9a-f]{64}$/;
// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
// → { ok: true, scan } | { ok: false, reason }
export async function admitFile(info, deps) {
const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
let scanResult = 'skipped';
if (cfg.malwareScan) {
const r = await scan(info.path, cfg.scanCmd);
if (r.result !== 'clean') {
log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
return { ok: false, reason: 'scan ' + r.result };
}
scanResult = 'clean';
}
await upsertContent({
cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
scan: scanResult, now: now(),
});
return { ok: true, scan: scanResult };
}
```
## Appendix B — server/p2p-admit.test.js
```js
import { test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { admitFile, scanFile } from './p2p-admit.js';
import { sha256File, sha256Range } from './hash.js';
const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
const file = join(dir, 'f.bin');
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const quiet = { warn() {}, info() {} };
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };
test('sha256File / sha256Range match node:crypto', async () => {
expect(await sha256File(file)).toBe(CID);
const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
expect(await sha256Range(file, 1000, 65536)).toBe(want);
});
test('scan off by default: admitted with scan=skipped', async () => {
const rows = [];
const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
expect(r).toEqual({ ok: true, scan: 'skipped' });
expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
});
test('scan on: clean admits, infected and scanner errors do not', async () => {
for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
const rows = [];
const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
expect(r.ok).toBe(ok);
expect(rows.length).toBe(ok ? 1 : 0);
}
});
test('disabled P2P or a malformed cid never admits', async () => {
const rows = [];
expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
expect(rows.length).toBe(0);
});
test('scanFile maps exit codes', async () => {
expect((await scanFile(file, 'true')).result).toBe('clean');
expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
});
```