274 lines
12 KiB
Markdown
274 lines
12 KiB
Markdown
---
|
||
id: 009-server-content-hash-186e7f
|
||
title: Hash every validated server copy and register it as verified content
|
||
created: 2026-09-29
|
||
depends_on: [008-p2p-schema-and-config-127966]
|
||
est_files: 7
|
||
---
|
||
|
||
# 009 — Server content hashes → verified P2P content
|
||
|
||
## Objective
|
||
|
||
Implements flow 1 of `docs/p2p-architecture.md`. After this plan:
|
||
- Every copy the media cache promotes (fetch lane AND compression lane) gets a
|
||
server-computed SHA-256 stored in `media_cache.sha256`; copies cached earlier are
|
||
hashed by a background backfill 30 s after boot.
|
||
- Each hashed copy is admitted to `p2p_content` via `admitFile()` (malware scan only
|
||
when `P2P_MALWARE_SCAN=1`; OFF by default). P2P disabled → nothing is admitted.
|
||
- `GET /api/download/:id` (server-cache path) sends `X-Content-SHA256: <cid>`.
|
||
- `/api/streams`' cached payload gets an additive `data.cid` (web-only, like
|
||
`data.serverCached`).
|
||
|
||
The media-cache edits were written and tested ahead of time (25/25 media-cache
|
||
tests pass, incl. 2 new ones); they ship as patch files.
|
||
|
||
## Context the executor must NOT rediscover
|
||
|
||
- Patches (made against the current tree; `media-cache.js` and its test are untouched by
|
||
plans 001–008): `plans/patches/009-media-cache.diff`, `plans/patches/009-media-cache-test.diff`.
|
||
They add to `createMediaCache()` the options `hashFile` (default `sha256File` from
|
||
`./hash.js`), `onReady(info)` and `backfillDelayMs` (default 30 000; tests pass -1), hash the
|
||
file before promotion in `runFetch` and `runOptimize`, store `sha256` in the row, call
|
||
`onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta })`, and export
|
||
`backfillHashes()`.
|
||
- `server/server.js:983-1012` — the `createMediaCache({ … transcode: { … }, })` call; its last
|
||
property is `transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },`.
|
||
- `server/server.js` `cachedDownloadResponse(videoId, fp, row)` (~line 1290) builds headers:
|
||
```js
|
||
headers: {
|
||
'Content-Type': 'video/mp4',
|
||
'Content-Length': String(file.size),
|
||
'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
|
||
'Cache-Control': 'no-store',
|
||
'Access-Control-Allow-Origin': '*',
|
||
},
|
||
```
|
||
- `server/server.js` `cachedStreamsPayload(videoId, row)` (~line 1024) returns
|
||
`{ meta: {…}, audioUrl, qualities: [...], serverCached: true }`.
|
||
- Plan 008 created `server/p2p-config.js` (`P2P`) and `server/p2p-db.js` (`upsertContent`).
|
||
|
||
## Steps
|
||
|
||
1. Create `server/hash.js` with exactly:
|
||
```js
|
||
/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
|
||
* The hex digest of a validated file is its P2P content id (cid). */
|
||
import { createHash } from 'node:crypto';
|
||
import { createReadStream } from 'node:fs';
|
||
|
||
export function sha256File(path) {
|
||
return new Promise((resolve, reject) => {
|
||
const h = createHash('sha256');
|
||
createReadStream(path, { highWaterMark: 1024 * 1024 })
|
||
.on('data', (d) => h.update(d))
|
||
.on('error', reject)
|
||
.on('end', () => resolve(h.digest('hex')));
|
||
});
|
||
}
|
||
|
||
// Hash of bytes [offset, offset+length) — used for holder range challenges.
|
||
export function sha256Range(path, offset, length) {
|
||
return new Promise((resolve, reject) => {
|
||
if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
|
||
const h = createHash('sha256');
|
||
createReadStream(path, { start: offset, end: offset + length - 1 })
|
||
.on('data', (d) => h.update(d))
|
||
.on('error', reject)
|
||
.on('end', () => resolve(h.digest('hex')));
|
||
});
|
||
}
|
||
```
|
||
2. Create `server/p2p-admit.js` — copy VERBATIM from Appendix A.
|
||
3. Create `server/p2p-admit.test.js` — copy VERBATIM from Appendix B.
|
||
4. Apply the patches from the repo root:
|
||
```bash
|
||
git apply plans/patches/009-media-cache.diff
|
||
git apply plans/patches/009-media-cache-test.diff
|
||
```
|
||
If either fails, STOP and report the error (do not hand-edit).
|
||
5. `server/package.json` "test" script — append ` && bun test ./p2p-admit.test.js`.
|
||
6. `server/server.js` imports — add next to the other local imports (the `p2p-db.js` import from
|
||
plan 008 may already exist; merge into it):
|
||
```js
|
||
import { admitFile } from './p2p-admit.js';
|
||
import { P2P } from './p2p-config.js';
|
||
import * as p2pDb from './p2p-db.js';
|
||
```
|
||
If plan 008 added `import { initP2pSchema } from './p2p-db.js';`, keep it and change the call in
|
||
`main()` from `initP2pSchema()` to `p2pDb.initP2pSchema()` only if you removed the named import.
|
||
7. `server/server.js` `createMediaCache({...})` — after the closing `},` of `transcode: {…},` add:
|
||
```js
|
||
// P2P (docs/p2p-architecture.md): every validated copy's server-computed
|
||
// hash becomes verified content, after the optional malware scan.
|
||
onReady: (info) => admitFile(
|
||
{ ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
|
||
{ cfg: P2P, upsertContent: p2pDb.upsertContent },
|
||
),
|
||
```
|
||
8. `server/server.js` `cachedDownloadResponse` — add to the headers object:
|
||
```js
|
||
...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),
|
||
```
|
||
9. `server/server.js` `cachedStreamsPayload` — after `serverCached: true,` add
|
||
`cid: row.sha256 || null,` and add a comment above the return:
|
||
`// data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.`
|
||
10. `Dockerfile` — optional ClamAV, off by default. After the existing
|
||
`RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/*` block add:
|
||
```dockerfile
|
||
# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
|
||
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
|
||
ARG INSTALL_CLAMAV=0
|
||
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
|
||
apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
|
||
freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
|
||
fi
|
||
```
|
||
|
||
## Out of scope / do NOT touch
|
||
|
||
- `validateMedia()` itself, the eviction logic, any frontend file.
|
||
- Never serve anything from the intake dir; no new routes in this plan.
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
|
||
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
|
||
bun test ./p2p-admit.test.js 2>&1 | tail -4
|
||
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
|
||
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
|
||
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
|
||
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js
|
||
```
|
||
|
||
Expected: admit tests `5 pass`; content-hash tests `2 pass`; every file `0 fail`
|
||
(media-cache: 25 pass); `SERVER_OK`; the grep shows the 3 edits.
|
||
|
||
## Report format (executor: follow exactly)
|
||
|
||
Output ONLY the following, no other prose:
|
||
|
||
1. `git diff --stat` and the unified diff of `server/server.js`, `server/package.json`, `Dockerfile`.
|
||
2. Raw output of the Verification commands.
|
||
3. `Findings:` — max 10 lines.
|
||
|
||
Do not commit. Do not push. Do not touch files outside the Steps.
|
||
|
||
---
|
||
|
||
## Appendix A — server/p2p-admit.js
|
||
|
||
```js
|
||
/* ============================================================================
|
||
* p2p-admit.js — the ONLY way a content id enters p2p_content
|
||
* (docs/p2p-architecture.md, "Security rules").
|
||
*
|
||
* Callers must already have: the complete file on the server's own disk, its
|
||
* SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
|
||
* optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
|
||
* row. The scan command gets the path as its last argument: exit 0 = clean,
|
||
* 1 = infected (rejected), anything else = scanner error (not admitted now).
|
||
* ========================================================================== */
|
||
import { spawn } from 'node:child_process';
|
||
|
||
export function scanFile(path, cmd) {
|
||
const parts = String(cmd).split(/\s+/).filter(Boolean);
|
||
return new Promise((resolve) => {
|
||
let child;
|
||
try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
|
||
catch (e) { resolve({ result: 'error', detail: e.message }); return; }
|
||
let out = '';
|
||
child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
|
||
child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
|
||
child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
|
||
child.on('close', (code) => resolve(
|
||
code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
|
||
));
|
||
});
|
||
}
|
||
|
||
const CID_RE = /^[0-9a-f]{64}$/;
|
||
|
||
// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
|
||
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
|
||
// → { ok: true, scan } | { ok: false, reason }
|
||
export async function admitFile(info, deps) {
|
||
const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
|
||
if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
|
||
if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
|
||
let scanResult = 'skipped';
|
||
if (cfg.malwareScan) {
|
||
const r = await scan(info.path, cfg.scanCmd);
|
||
if (r.result !== 'clean') {
|
||
log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
|
||
return { ok: false, reason: 'scan ' + r.result };
|
||
}
|
||
scanResult = 'clean';
|
||
}
|
||
await upsertContent({
|
||
cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
|
||
acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
|
||
scan: scanResult, now: now(),
|
||
});
|
||
return { ok: true, scan: scanResult };
|
||
}
|
||
```
|
||
|
||
## Appendix B — server/p2p-admit.test.js
|
||
|
||
```js
|
||
import { test, expect } from 'bun:test';
|
||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||
import { tmpdir } from 'node:os';
|
||
import { join } from 'node:path';
|
||
import { createHash } from 'node:crypto';
|
||
import { admitFile, scanFile } from './p2p-admit.js';
|
||
import { sha256File, sha256Range } from './hash.js';
|
||
|
||
const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
|
||
const file = join(dir, 'f.bin');
|
||
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
|
||
writeFileSync(file, bytes);
|
||
const CID = createHash('sha256').update(bytes).digest('hex');
|
||
const quiet = { warn() {}, info() {} };
|
||
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
|
||
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };
|
||
|
||
test('sha256File / sha256Range match node:crypto', async () => {
|
||
expect(await sha256File(file)).toBe(CID);
|
||
const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
|
||
expect(await sha256Range(file, 1000, 65536)).toBe(want);
|
||
});
|
||
|
||
test('scan off by default: admitted with scan=skipped', async () => {
|
||
const rows = [];
|
||
const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
|
||
expect(r).toEqual({ ok: true, scan: 'skipped' });
|
||
expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
|
||
});
|
||
|
||
test('scan on: clean admits, infected and scanner errors do not', async () => {
|
||
for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
|
||
const rows = [];
|
||
const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
|
||
expect(r.ok).toBe(ok);
|
||
expect(rows.length).toBe(ok ? 1 : 0);
|
||
}
|
||
});
|
||
|
||
test('disabled P2P or a malformed cid never admits', async () => {
|
||
const rows = [];
|
||
expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
|
||
expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
|
||
expect(rows.length).toBe(0);
|
||
});
|
||
|
||
test('scanFile maps exit codes', async () => {
|
||
expect((await scanFile(file, 'true')).result).toBe('clean');
|
||
expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
|
||
expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
|
||
expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
|
||
});
|
||
```
|