josh/ytplayer

Fork 0

Files

Claude b19193841c plan: start 009-server-content-hash-186e7f

2026-09-30 07:11:55 +00:00

12 KiB

Raw Blame History

id, title, created, depends_on, est_files

title

created

depends_on

est_files

009-server-content-hash-186e7f

Hash every validated server copy and register it as verified content

2026-09-29

008-p2p-schema-and-config-127966

009 — Server content hashes → verified P2P content

Objective

Implements flow 1 of docs/p2p-architecture.md. After this plan:

Every copy the media cache promotes (fetch lane AND compression lane) gets a server-computed SHA-256 stored in media_cache.sha256; copies cached earlier are hashed by a background backfill 30 s after boot.
Each hashed copy is admitted to p2p_content via admitFile() (malware scan only when P2P_MALWARE_SCAN=1; OFF by default). P2P disabled → nothing is admitted.
GET /api/download/:id (server-cache path) sends X-Content-SHA256: <cid>.
/api/streams' cached payload gets an additive data.cid (web-only, like data.serverCached).

The media-cache edits were written and tested ahead of time (25/25 media-cache tests pass, incl. 2 new ones); they ship as patch files.

Context the executor must NOT rediscover

Patches (made against the current tree; media-cache.js and its test are untouched by plans 001–008): plans/patches/009-media-cache.diff, plans/patches/009-media-cache-test.diff. They add to createMediaCache() the options hashFile (default sha256File from ./hash.js), onReady(info) and backfillDelayMs (default 30 000; tests pass -1), hash the file before promotion in runFetch and runOptimize, store sha256 in the row, call onReady({ id, gen, path, sha256, size, height, vcodec, acodec, duration, meta }), and export backfillHashes().
server/server.js:983-1012 — the createMediaCache({ … transcode: { … }, }) call; its last property is transcode: { enabled: …, maxSeconds: envNum('MEDIA_OPT_MAX_SECONDS', 3600), },.

server/server.js cachedDownloadResponse(videoId, fp, row) (~line 1290) builds headers:

    headers: {
      'Content-Type':        'video/mp4',
      'Content-Length':      String(file.size),
      'Content-Disposition': `attachment; filename="${videoId}.mp4"`,
      'Cache-Control':       'no-store',
      'Access-Control-Allow-Origin': '*',
    },

server/server.js cachedStreamsPayload(videoId, row) (~line 1024) returns { meta: {…}, audioUrl, qualities: [...], serverCached: true }.
Plan 008 created server/p2p-config.js (P2P) and server/p2p-db.js (upsertContent).

Steps

Create server/hash.js with exactly:

/* hash.js — streaming SHA-256 of files on disk (never loads a whole video).
 * The hex digest of a validated file is its P2P content id (cid). */
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';

export function sha256File(path) {
  return new Promise((resolve, reject) => {
    const h = createHash('sha256');
    createReadStream(path, { highWaterMark: 1024 * 1024 })
      .on('data', (d) => h.update(d))
      .on('error', reject)
      .on('end', () => resolve(h.digest('hex')));
  });
}

// Hash of bytes [offset, offset+length) — used for holder range challenges.
export function sha256Range(path, offset, length) {
  return new Promise((resolve, reject) => {
    if (!(length > 0)) { resolve(createHash('sha256').digest('hex')); return; }
    const h = createHash('sha256');
    createReadStream(path, { start: offset, end: offset + length - 1 })
      .on('data', (d) => h.update(d))
      .on('error', reject)
      .on('end', () => resolve(h.digest('hex')));
  });
}

Create server/p2p-admit.js — copy VERBATIM from Appendix A.
Create server/p2p-admit.test.js — copy VERBATIM from Appendix B.
Apply the patches from the repo root:
```
git apply plans/patches/009-media-cache.diff
git apply plans/patches/009-media-cache-test.diff
```
If either fails, STOP and report the error (do not hand-edit).
server/package.json "test" script — append && bun test ./p2p-admit.test.js.
server/server.js imports — add next to the other local imports (the p2p-db.js import from plan 008 may already exist; merge into it):
```
import { admitFile } from './p2p-admit.js';
import { P2P } from './p2p-config.js';
import * as p2pDb from './p2p-db.js';
```
If plan 008 added import { initP2pSchema } from './p2p-db.js';, keep it and change the call in main() from initP2pSchema() to p2pDb.initP2pSchema() only if you removed the named import.

server/server.js createMediaCache({...}) — after the closing }, of transcode: {…}, add:

  // P2P (docs/p2p-architecture.md): every validated copy's server-computed
  // hash becomes verified content, after the optional malware scan.
  onReady: (info) => admitFile(
    { ...info, cid: info.sha256, videoId: info.id, origin: 'server' },
    { cfg: P2P, upsertContent: p2pDb.upsertContent },
  ),

server/server.js cachedDownloadResponse — add to the headers object:

      ...(row.sha256 ? { 'X-Content-SHA256': row.sha256, 'Access-Control-Expose-Headers': 'X-Content-SHA256' } : {}),

server/server.js cachedStreamsPayload — after serverCached: true, add cid: row.sha256 || null, and add a comment above the return: // data.cid is additive and web-only (like serverCached) — the Tauri bridge ignores it.

Dockerfile — optional ClamAV, off by default. After the existing RUN apt-get update -qq && … rm -rf /var/lib/apt/lists/* block add:

# Optional malware scanner for P2P admission (P2P_MALWARE_SCAN=1). Off by
# default: build with --build-arg INSTALL_CLAMAV=1 to include it.
ARG INSTALL_CLAMAV=0
RUN if [ "$INSTALL_CLAMAV" = "1" ]; then \
      apt-get update -qq && apt-get install -y --no-install-recommends clamav clamav-freshclam && \
      freshclam --quiet || true; rm -rf /var/lib/apt/lists/*; \
    fi

Out of scope / do NOT touch

validateMedia() itself, the eviction logic, any frontend file.
Never serve anything from the intake dir; no new routes in this plan.

Verification

cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
which ffmpeg ffprobe || echo "NO FFMPEG — media-cache tests need it (apt-get install ffmpeg)"
bun test ./p2p-admit.test.js 2>&1 | tail -4
bun test --timeout 60000 ./media-cache.test.js -t "content hashes" 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -n "X-Content-SHA256\|onReady: (info)\|cid: row.sha256" server.js

Expected: admit tests 5 pass; content-hash tests 2 pass; every file 0 fail (media-cache: 25 pass); SERVER_OK; the grep shows the 3 edits.

Report format (executor: follow exactly)

Output ONLY the following, no other prose:

git diff --stat and the unified diff of server/server.js, server/package.json, Dockerfile.
Raw output of the Verification commands.
Findings: — max 10 lines.

Do not commit. Do not push. Do not touch files outside the Steps.

Appendix A — server/p2p-admit.js

/* ============================================================================
 * p2p-admit.js — the ONLY way a content id enters p2p_content
 * (docs/p2p-architecture.md, "Security rules").
 *
 * Callers must already have: the complete file on the server's own disk, its
 * SHA-256 computed BY THE SERVER, and validateMedia() passed. This adds the
 * optional malware scan (P2P_MALWARE_SCAN=1, off by default) and writes the
 * row. The scan command gets the path as its last argument: exit 0 = clean,
 * 1 = infected (rejected), anything else = scanner error (not admitted now).
 * ========================================================================== */
import { spawn } from 'node:child_process';

export function scanFile(path, cmd) {
  const parts = String(cmd).split(/\s+/).filter(Boolean);
  return new Promise((resolve) => {
    let child;
    try { child = spawn(parts[0], [...parts.slice(1), path], { stdio: ['ignore', 'pipe', 'pipe'] }); }
    catch (e) { resolve({ result: 'error', detail: e.message }); return; }
    let out = '';
    child.stdout.on('data', (d) => { out = (out + d).slice(-2000); });
    child.stderr.on('data', (d) => { out = (out + d).slice(-2000); });
    child.on('error', (e) => resolve({ result: 'error', detail: e.message }));
    child.on('close', (code) => resolve(
      code === 0 ? { result: 'clean' } : code === 1 ? { result: 'infected', detail: out.trim() } : { result: 'error', detail: out.trim() || 'exit ' + code },
    ));
  });
}

const CID_RE = /^[0-9a-f]{64}$/;

// info: { path, cid, videoId, size, height, vcodec, acodec, duration, meta, origin }
// deps: { cfg (P2P config), upsertContent, scan = scanFile, now = Date.now, log = console }
// → { ok: true, scan } | { ok: false, reason }
export async function admitFile(info, deps) {
  const { cfg, upsertContent, scan = scanFile, now = Date.now, log = console } = deps;
  if (!cfg.enabled) return { ok: false, reason: 'p2p disabled' };
  if (!CID_RE.test(String(info.cid || ''))) return { ok: false, reason: 'bad cid' };
  let scanResult = 'skipped';
  if (cfg.malwareScan) {
    const r = await scan(info.path, cfg.scanCmd);
    if (r.result !== 'clean') {
      log.warn?.(`[p2p] ${info.videoId} ${info.cid.slice(0, 12)} not admitted: scan ${r.result} ${r.detail || ''}`);
      return { ok: false, reason: 'scan ' + r.result };
    }
    scanResult = 'clean';
  }
  await upsertContent({
    cid: info.cid, videoId: info.videoId, size: info.size, height: info.height, vcodec: info.vcodec,
    acodec: info.acodec, duration: info.duration, meta: info.meta || {}, origin: info.origin,
    scan: scanResult, now: now(),
  });
  return { ok: true, scan: scanResult };
}

Appendix B — server/p2p-admit.test.js

import { test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { admitFile, scanFile } from './p2p-admit.js';
import { sha256File, sha256Range } from './hash.js';

const dir = mkdtempSync(join(tmpdir(), 'ytp-admit-'));
const file = join(dir, 'f.bin');
const bytes = Buffer.from(Array.from({ length: 300000 }, (_, i) => i % 251));
writeFileSync(file, bytes);
const CID = createHash('sha256').update(bytes).digest('hex');
const quiet = { warn() {}, info() {} };
const cfg = (o = {}) => ({ enabled: true, malwareScan: false, scanCmd: 'true', ...o });
const info = { path: file, cid: CID, videoId: 'dQw4w9WgXcQ', size: bytes.length, origin: 'server' };

test('sha256File / sha256Range match node:crypto', async () => {
  expect(await sha256File(file)).toBe(CID);
  const want = createHash('sha256').update(bytes.subarray(1000, 1000 + 65536)).digest('hex');
  expect(await sha256Range(file, 1000, 65536)).toBe(want);
});

test('scan off by default: admitted with scan=skipped', async () => {
  const rows = [];
  const r = await admitFile(info, { cfg: cfg(), upsertContent: async (c) => rows.push(c), log: quiet });
  expect(r).toEqual({ ok: true, scan: 'skipped' });
  expect(rows[0]).toMatchObject({ cid: CID, videoId: 'dQw4w9WgXcQ', origin: 'server', scan: 'skipped' });
});

test('scan on: clean admits, infected and scanner errors do not', async () => {
  for (const [result, ok] of [['clean', true], ['infected', false], ['error', false]]) {
    const rows = [];
    const r = await admitFile(info, { cfg: cfg({ malwareScan: true }), upsertContent: async (c) => rows.push(c), scan: async () => ({ result }), log: quiet });
    expect(r.ok).toBe(ok);
    expect(rows.length).toBe(ok ? 1 : 0);
  }
});

test('disabled P2P or a malformed cid never admits', async () => {
  const rows = [];
  expect((await admitFile(info, { cfg: cfg({ enabled: false }), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
  expect((await admitFile({ ...info, cid: 'XYZ' }, { cfg: cfg(), upsertContent: async (c) => rows.push(c) })).ok).toBe(false);
  expect(rows.length).toBe(0);
});

test('scanFile maps exit codes', async () => {
  expect((await scanFile(file, 'true')).result).toBe('clean');
  expect((await scanFile(file, 'false')).result).toBe('infected'); // exit 1
  expect((await scanFile(file, 'sh -c "exit 2" --')).result).toBe('error');
  expect((await scanFile(file, '/nonexistent/scanner')).result).toBe('error');
});

12 KiB Raw Blame History Unescape Escape