plan: start 010-views-and-retention-d0c6ca

2026-09-30 07:18:52 +00:00
parent ea56d286b3
commit f12ca87131
2 changed files with 1 additions and 1 deletions
--- a/plans/active/010-views-and-retention-d0c6ca.md
+++ b/plans/active/010-views-and-retention-d0c6ca.md
@@ -0,0 +1,168 @@
+---
+id: 010-views-and-retention-d0c6ca
+title: Count views and evict server copies by retention criteria before LRU
+created: 2026-09-29
+depends_on: [009-server-content-hash-186e7f]
+est_files: 5
+---
+
+# 010 — Views + retention-ordered eviction
+
+## Objective
+
+Implements flow 9 of `docs/p2p-architecture.md`. After this plan:
+- Every play (`GET /api/streams`) and save (`GET /api/download/:id`) counts one view in
+  `video_views` (deduped per client + video for 30 min; warm-ups don't count).
+- When the media cache needs room it evicts copies that are neither **top**
+  (≥ `P2P_KEEP_MIN_VIEWS` views in `P2P_KEEP_DAYS`) nor **recent** (played within
+  `P2P_KEEP_RECENT_DAYS`) first — fewest views, then oldest — and only then qualifying
+  ones by LRU. Budget, disk guard and 10-min play protection are unchanged.
+- Eviction never touches `p2p_content` / holders (the catalog keeps growing).
+
+The query + media-cache change were tested ahead of time (retention test 1/1,
+media-cache 26/26).
+
+## Context the executor must NOT rediscover
+
+- `plans/patches/010-retention-eviction.diff` patches `server/media-cache.js` (`makeRoom` uses
+  `db.listMediaEvictionOrder()` when present, else `db.listMediaLru()`) and adds one test to
+  `server/media-cache.test.js`. It applies on top of plan 009's patches.
+- `server/p2p-db.js` (plan 008) has `dayKey`, `rowsOf`, `addView(videoId, now)`.
+- `server/server.js` `createMediaCache({ dir: MEDIA_DIR, db: { getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats }, …` (~line 983).
+- `app.get('/api/streams', async (c) => {` (~line 532): first lines validate `videoId` and
+  return 400 when empty. `app.get('/api/download/:videoId', async (c) => {` (~line 1311) does the same.
+- Plan 009 imported `P2P` and `* as p2pDb` in server.js.
+
+## Steps
+
+1. `server/p2p-db.js` — append at the end of the file:
+   ```js
+
+   // ---- retention (plan 010) -------------------------------------------------------
+   // Server copies in the order they should be evicted when the cache needs room:
+   // first the ones that are neither "top" (≥ keepMinViews views in keepDays) nor
+   // "recent" (played within keepRecentDays) — fewest views, then oldest — and
+   // only then the qualifying ones, least recently played first.
+   export async function listMediaEvictionOrder({ now, keepMinViews, keepDays, keepRecentDays }) {
+     const r = await db.execute({
+       sql: `SELECT m.video_id, m.size, m.last_access,
+                    COALESCE((SELECT SUM(v.n) FROM video_views v
+                               WHERE v.video_id = m.video_id AND v.day >= ?), 0) AS views
+               FROM media_cache m WHERE m.status = 'ready'`,
+       args: [dayKey(now - keepDays * 86400_000)],
+     });
+     const recentCut = now - keepRecentDays * 86400_000;
+     const rows = rowsOf(r).map((x) => ({
+       ...x, views: Number(x.views) || 0,
+       qualifies: (Number(x.views) || 0) >= keepMinViews || Number(x.last_access) >= recentCut,
+     }));
+     rows.sort((a, b) => (a.qualifies - b.qualifies)
+       || (a.qualifies ? 0 : a.views - b.views)
+       || (a.last_access - b.last_access));
+     return rows;
+   }
+   ```
+2. Create `server/p2p-retention.test.js` — copy VERBATIM from the Appendix.
+3. `server/package.json` "test" script — append ` && bun test ./p2p-retention.test.js`.
+4. From the repo root: `git apply plans/patches/010-retention-eviction.diff` (STOP and report on failure).
+5. `server/server.js` `createMediaCache({ … db: { … } …` — replace the `db:` line with:
+   ```js
+     db: {
+       getMedia, upsertMedia, deleteMedia, listMedia, listMediaLru, touchMedia, mediaStats,
+       // Retention (docs/p2p-architecture.md flow 9): cold copies go before popular ones.
+       listMediaEvictionOrder: () => p2pDb.listMediaEvictionOrder({
+         now: Date.now(), keepMinViews: P2P.keepMinViews, keepDays: P2P.keepDays, keepRecentDays: P2P.keepRecentDays,
+       }),
+     },
+   ```
+6. `server/server.js` — directly above `app.get('/api/streams', …)` (and above the plan-005
+   warm route if present) add:
+   ```js
+   // One view per client per video per 30 min (a play and its save count once).
+   const viewSeen = new Map(); // `${who}|${id}` -> ms
+   function countView(c, videoId) {
+     const who = (c.req.header('x-forwarded-for') || '').split(',')[0].trim() || c.req.query('fp') || 'local';
+     const key = who + '|' + videoId;
+     const now = Date.now();
+     if (now - (viewSeen.get(key) || 0) < 30 * 60_000) return;
+     viewSeen.set(key, now);
+     if (viewSeen.size > 20000) viewSeen.clear();
+     p2pDb.addView(videoId, now).catch(() => {});
+   }
+   ```
+7. In `/api/streams`, directly after `if (!videoId) return c.json({ ok: false, error: 'missing videoId' }, 400);`
+   add `countView(c, videoId);`. Do the same in `/api/download/:videoId` after its
+   `missing videoId` guard.
+
+## Out of scope / do NOT touch
+
+- `MEDIA_CACHE_MAX_BYTES`, `EVICT_PROTECT_MS`, the disk guard, `listMediaLru` itself.
+- Do not delete server files on a timer — eviction stays budget-driven.
+
+## Verification
+
+```bash
+cd /home/user/ytplayer/server && bun install >/dev/null 2>&1
+bun test ./p2p-retention.test.js 2>&1 | tail -4
+bun test --timeout 60000 ./media-cache.test.js 2>&1 | tail -4
+bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
+bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
+grep -c "countView(c, videoId)" server.js
+```
+
+Expected: `1 pass`; media-cache `26 pass 0 fail`; every file `0 fail`; `SERVER_OK`; grep `2`.
+
+## Report format (executor: follow exactly)
+
+Output ONLY the following, no other prose:
+
+1. `git diff` (unified) of all changes.
+2. Raw output of the Verification commands.
+3. `Findings:` — max 10 lines.
+
+Do not commit. Do not push. Do not touch files outside the Steps.
+
+---
+
+## Appendix — server/p2p-retention.test.js
+
+```js
+import { test, expect, beforeAll } from 'bun:test';
+import { mkdtempSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+const root = mkdtempSync(join(tmpdir(), 'ytp-retention-test-'));
+process.env.DB_PATH = join(root, 'test.db');
+const dbmod = await import('./db.js');
+const P = await import('./p2p-db.js');
+
+const DAY = 86400_000;
+const NOW = Date.UTC(2026, 8, 29, 12);
+const opts = { now: NOW, keepMinViews: 3, keepDays: 30, keepRecentDays: 14 };
+
+beforeAll(async () => {
+  await dbmod.initDb();
+  await P.initP2pSchema();
+  const add = (id, lastAccessDaysAgo) => dbmod.upsertMedia(id, { status: 'ready', size: 100, last_access: NOW - lastAccessDaysAgo * DAY });
+  await add('topOld00001', 60);   // 5 views in window → top, but old
+  await add('recent00001', 2);    // 0 views, played 2 days ago → recent
+  await add('cold0000001', 40);   // 1 view in window, old → evict first-ish
+  await add('cold0000002', 20);   // 0 views, 20 days → evict first
+  await add('cold0000003', 50);   // 0 views, 50 days → evict first (older than cold2)
+  for (let i = 0; i < 5; i++) await P.addView('topOld00001', NOW - 5 * DAY);
+  await P.addView('cold0000001', NOW - 3 * DAY);
+  await P.addView('cold0000002', NOW - 45 * DAY); // outside the 30-day window
+});
+
+test('non-qualifying copies go first (fewest views, then oldest); qualifying ones by LRU', async () => {
+  const order = (await P.listMediaEvictionOrder(opts)).map((r) => [r.video_id, r.qualifies]);
+  expect(order).toEqual([
+    ['cold0000003', false],
+    ['cold0000002', false],
+    ['cold0000001', false],
+    ['topOld00001', true],
+    ['recent00001', true],
+  ]);
+});
+```