Cache every played or saved video on the server via background jobs with a validation gate and a Broken re-download button

This commit is contained in:
Jonathan Sykes
2026-09-13 09:45:01 +08:00
parent 292a0c7e29
commit 5fb863ba25
8 changed files with 1240 additions and 20 deletions

View File

@@ -15,15 +15,44 @@ The PWA is what runs in production; `legacy/` holds the old native-only docs.
| `frontend/fingerprint.js` | Browser fingerprint used as the sync key |
| `frontend/async-guard.js` | Stale-async-response guard (unit-tested) |
| `server/server.js` | Bun + Hono backend — endpoint list is in its header comment |
| `server/db.js` | libsql schema/queries (`users`, `profiles`, `video_history`…) |
| `server/db.js` | libsql schema/queries (`users`, `profiles`, `video_history`, `media_cache`…) |
| `server/media-cache.js` | Server-side video cache: background fetch jobs, validation gate, x264 lane, LRU |
| `bin/yt-dlp` | Downloaded by `npm run setup`, gitignored |
| `scripts/` | icon generation, yt-dlp setup, push helper |
| `tests/` | Playwright e2e specs · unit tests live next to sources in `frontend/*.test.js` |
| `legacy/` | Old native-shell docs — do not treat as current |
API endpoints: `GET /api/search|channel|streams|download/:id|version|user/data|profile/load`,
`GET /api/media/:id|media/:id/status|media/stats`, `POST /api/media/:id/redownload`,
`POST /api/user/sync|profile/create|profile/save`, `GET /sw.js` (BUILD_TAG-injected), `GET /*` static.
JSON shapes mirror the Tauri Rust bridge exactly — don't change one side alone.
JSON shapes mirror the Tauri Rust bridge exactly — don't change one side alone
(`/api/streams`' `data.serverCached` is additive and web-only).
## Server media cache (`server/media-cache.js`)
- Every played (`/api/streams`, LOW priority, ≤ `MEDIA_AUTO_MAX_SECONDS`) or saved
(`/api/download`, HIGH, ≤ 3 h) video gets ONE copy: `$MEDIA_DIR/<id>.<gen>.mp4`
(≤720p H.264 8-bit + AAC, faststart) + `<id>.<gen>.m4a` audio sidecar for audio-only
mode. `MEDIA_DIR` defaults to `./data/media` (the `ytplayer-data` volume in prod).
- Jobs are server-owned — **never pass a request AbortSignal into them**; a client
closing its tab must not kill a fetch. Downloads still run inside `withSaveSlot`.
- **Validation gate** (`validateMedia`): h264 yuv420p + AAC, container/stream durations
cover the source, a full `-c copy` demux pass with zero stderr, head + tail decode.
The demux pass is what catches a faststart file whose index is intact but whose
data is cut. Nothing reaches the cache dir without passing it; failures back off
(15 min · 2^n, max 24 h) and saves fall back to the legacy per-request path.
- `gen` is in the filename and URL (`/api/media/<id>?g=<gen>`): a URL never serves
different bytes. A replaced gen is kept 30 min so in-flight Range playback survives.
- **Compression lane**: niced x264 `-preset slow -crf 28 -fpsmax 30`, audio copied,
kept only if ≥15% smaller. Measured: YouTube's 720p avc1 is already ~0.6–0.7 Mbps, so
CRF 24 made files *bigger*; CRF 28 ≈ −20% video at SSIM 0.994. Lyric videos usually
keep the original. HEVC/AV1/10-bit are smaller but don't play in iOS `<video>`.
- **Budget**: `MEDIA_CACHE_MAX_BYTES` (10 GiB) LRU by `last_access` (anything played in
the last 10 min is protected) + `MEDIA_MIN_FREE_BYTES` (5 GiB) disk guard → skip, stream.
- **Broken**: `POST /api/media/:id/redownload` (now-playing ⚠ Broken button). When a
client falls back from a cached copy (`/api/streams?nocache=1`) the server also
re-validates its copy and refetches it if it fails — never trusts one device's error.
- Ops: `curl https://worship.hesed.sbs/api/media/stats`; files under
`/app/data/media` in the container.
## Local dev
@@ -129,6 +158,8 @@ Local DB file: `server/data/ytplayer.db` (gitignored). `BUILD_TAG` is computed f
## Testing
- Unit: **`node --test frontend/*.test.js`** (sw, sw-update, async-guard, video-edit).
Server: **`cd server && bun test`** (media-cache: real ffmpeg fixtures + a temp libsql DB).
Local e2e needs a CURRENT yt-dlp — a 2-month-old one 403s on every download.
The directory form `node --test frontend/` fails on Node 22 with
`Cannot find module .../frontend` — it resolves the dir as a module, not a test
glob. That's the harness, not the tests.