Queue executable speed and peer-to-peer sharing plans with tested patches and harnesses
This commit is contained in:
Claude
@@ -1,7 +1,25 @@
|
||||
# ytplayer — Speed & Features Plan (2026-09-29)
|
||||
|
||||
Scope: **make the app faster** (startup, search, pressing play, moving around)
|
||||
and **add new features**. Hardening/refactor work is intentionally out of scope.
|
||||
and **add new features**, led by **peer-to-peer video sharing**. Hardening/refactor
|
||||
work is intentionally out of scope.
|
||||
|
||||
## Integrated roadmap (what is queued and executable today)
|
||||
|
||||
Everything below is broken into step-by-step plans a smaller model can execute
|
||||
without exploring the code — see `plans/INDEX.md`. All 19 were dry-run end to end.
|
||||
|
||||
| Order | Plans | What it delivers | Source |
|
||||
|-------|-------|------------------|--------|
|
||||
| 1 | `001` | Timing marks (`ytp:boot`, `ytp:search`, `ytp:tap-to-play`) + yt-dlp duration logs | A "Measuring" |
|
||||
| 2 | `002`–`003` | Compressed + ETagged shell (app.js 426 → 94 KB), self-hosted fonts | A1.1–A1.3 |
|
||||
| 3 | `004`–`005` | One yt-dlp resolve per video at a time; resolve likely next plays before the tap | A3.1–A3.2 |
|
||||
| 4 | `006`–`007` | Search via InnerTube (~0.8 s, yt-dlp fallback); long-lived yt-dlp workers | A2.1, A3.3 |
|
||||
| 5 | `008`–`019` | **Peer-to-peer sharing** — design and rules in `docs/p2p-architecture.md` | Part B0 |
|
||||
|
||||
Not queued yet (write plans for them with `/plan-queue` when reached): A1.4 minify,
|
||||
A1.5 deferred boot work, A1.6 lazy modules, A2.2 suggestions, A2.3/A2.4, A3.4/A3.5,
|
||||
A4 (VPS media cache), A5 polish, and the Part B feature list.
|
||||
|
||||
## Where the time goes today (measured against prod)
|
||||
|
||||
@@ -99,6 +117,21 @@ before the user taps.**
|
||||
|
||||
## Part B — New features
|
||||
|
||||
### B0. Peer-to-peer video sharing (queued: plans 008–019)
|
||||
|
||||
The server keeps the user list, the video list and metadata; every validated copy
|
||||
gets a content id = SHA-256 of its bytes; devices that save a video become persistent
|
||||
**holders**; when YouTube and the server copy are gone, devices serve each other over
|
||||
WebRTC and can restore the server's copy. Rules the owner set (2026-09-29):
|
||||
**P2P is on by default**, the **server malware scan is off by default** (hashing +
|
||||
media validation always run), and holder records are **persistent** — the UI shows
|
||||
when each was last verified and marks old ones **stale** instead of dropping them.
|
||||
Full design, data model, flows and security rules: `docs/p2p-architecture.md`.
|
||||
The earlier phase 02–06 draft is superseded; its differences are listed at the end of
|
||||
that document.
|
||||
|
||||
### Other features (not queued yet)
|
||||
|
||||
Ranked by fit with how the app is actually used (worship sets, sing-alongs,
|
||||
offline playback). Each is sized; most reuse machinery that already exists.
|
||||
|
||||
@@ -144,16 +177,15 @@ offline playback). Each is sized; most reuse machinery that already exists.
|
||||
|
||||
## Suggested order
|
||||
|
||||
1. **Week 1 — quick speed wins:** add the timing marks (Measuring), A1.1
|
||||
compression, A1.2 ETags, A1.3 fonts, A3.1 coalescing, A3.2 warm-on-intent,
|
||||
A3.4 instant UI. These are all small, and together they fix most of what
|
||||
feels slow today.
|
||||
2. **Week 2:** A2.1 InnerTube search + A2.2 suggestions, A1.4 defer/minify.
|
||||
3. **Week 3:** A3.3 persistent yt-dlp worker, A1.5 deferred boot work, then
|
||||
measure the WireGuard link (A4) and decide on the VPS media cache.
|
||||
4. **Then features**, starting with the small high-fit ones: section loops,
|
||||
1. **Run the queue** (`/run-queue`): plans `001`–`007` (speed), then `008`–`019`
|
||||
(peer-to-peer). Deploy after `007` and measure with the `ytp:*` marks and the
|
||||
`[ytdlp]` log lines before starting P2P.
|
||||
2. **Next speed plans to write:** A3.4 instant UI on tap, A2.2 suggestions,
|
||||
A1.5 deferred boot work, then measure the WireGuard link (A4) and decide on the
|
||||
VPS media cache.
|
||||
3. **Then features**, starting with the small high-fit ones: section loops,
|
||||
confidence monitor, count-in, lyrics search, stats wrap-up — then
|
||||
service plans and chord charts.
|
||||
|
||||
Each row is sized to be one commit (CLAUDE.md commit rules) and can be queued
|
||||
with the `plan-queue` skill.
|
||||
Each item is sized to be one commit (CLAUDE.md commit rules); queue new ones with
|
||||
the `plan-queue` skill.
|
||||
|
||||
155
docs/p2p-architecture.md
Normal file
155
docs/p2p-architecture.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Peer-to-peer video sharing — architecture (2026-09-29)
|
||||
|
||||
Source of truth for plans `008`–`019` in `plans/queue/`. Every P2P plan links here
|
||||
instead of repeating the rules. It replaces the earlier "phase 02–06" draft; the
|
||||
differences are listed at the end.
|
||||
|
||||
## What the owner asked for
|
||||
|
||||
> Peer-to-peer saving of videos. The server stores the user list, the video list and
|
||||
> metadata. While the original source is online, a video is available for streaming
|
||||
> and download. When it is downloaded to a device, the server records that device in
|
||||
> the list of holders, so the video stays reachable from devices after the source is
|
||||
> gone. Top or recent files stay on the server under a total space limit; files that
|
||||
> don't meet the criteria (e.g. number of views, also stored on the server) are
|
||||
> deleted first. The video id is the file hash of the highest-quality copy. The
|
||||
> database grows over time. Each device has its own database that can be synced or
|
||||
> added to the server's, with verification that the file exists. A file must first be
|
||||
> downloaded by the server and checked before its hash is added to the server DB.
|
||||
|
||||
Corrections to the earlier draft (owner, 2026-09-29):
|
||||
1. **P2P is ON by default** (server and every client).
|
||||
2. **The server's malware scan is OFF by default** (admin opt-in). Hashing and the
|
||||
media validation gate are ALWAYS on and cannot be turned off.
|
||||
3. **Holder records are persistent, not short-lived leases.** A device stays listed
|
||||
as a holder until it says the file is gone, fails a check, or its reported list no
|
||||
longer contains it. The UI shows when each holder was last verified and marks it
|
||||
**stale** when that is older than `P2P_STALE_DAYS` (default 7). Online-right-now is
|
||||
a separate, live signal.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **source** | Where bytes originally come from: YouTube (via yt-dlp) or a server upload (`upl_…`). |
|
||||
| **video id** | Existing ids (`dQw4w9WgXcQ`, `upl_…`). Still used everywhere in the app and API. |
|
||||
| **content id (`cid`)** | Lowercase hex SHA-256 of the exact file bytes. The P2P identity of a file. One video id can have several cids over time (a better master, the HEVC copy). |
|
||||
| **master** | The best copy the server keeps for a video: today the validated ≤720p H.264+AAC faststart MP4 in `MEDIA_DIR` (`<id>.<gen>.mp4`). The compression lane's HEVC copy is a second, separately hashed file. Raising the master quality later just creates new cids linked by `video_id`. |
|
||||
| **verified content** | A `p2p_content` row. Exists only after the server itself held the complete bytes, computed the SHA-256 itself and `validateMedia()` passed (plus the malware scan if enabled). |
|
||||
| **holder** | A device that reported holding a cid. Row in `p2p_holders`, never deleted by time. |
|
||||
| **online** | The device has an open `/ws/p2p` socket right now (in memory only). |
|
||||
| **stale** | `now - last_verified_at > P2P_STALE_DAYS`. Shown in the UI, still listed. |
|
||||
|
||||
## Data model (server, libsql — grows forever)
|
||||
|
||||
Added by plan 008 in `server/p2p-db.js` (`initP2pSchema()` runs after `initDb()`):
|
||||
|
||||
```sql
|
||||
p2p_content (cid PK, video_id, size, height, vcodec, acodec, duration, meta JSON,
|
||||
origin 'server'|'intake', status 'verified'|'revoked',
|
||||
scan 'skipped'|'clean', created_at ms, verified_at ms)
|
||||
p2p_devices (device_id PK 'dev_<16hex>', secret_hash, fingerprint, profile,
|
||||
share 0|1, created_at ms, last_seen_at ms)
|
||||
p2p_holders (cid, device_id, status 'active'|'removed', trust 'reported'|'challenged',
|
||||
first_reported_at ms, last_verified_at ms, removed_at ms NULL,
|
||||
PRIMARY KEY (cid, device_id))
|
||||
video_views (video_id, day 'YYYY-MM-DD', n, PRIMARY KEY (video_id, day))
|
||||
media_cache.sha256 -- new column: cid of the current <id>.<gen>.mp4
|
||||
```
|
||||
|
||||
"User list" = the existing `users` (fingerprints) and `profiles` tables plus
|
||||
`p2p_devices`. Nothing is ever deleted from `p2p_content`; a bad file is `revoked`.
|
||||
|
||||
## Configuration (`server/p2p-config.js`, plan 008)
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
| `P2P_ENABLED` | `1` (on) | `0` turns off every P2P route, the hub and client features. |
|
||||
| `P2P_MALWARE_SCAN` | `0` (off) | `1` runs `P2P_SCAN_CMD <file>` before admission; exit 0 = clean, 1 = infected (rejected), other = error (not admitted, retried later). |
|
||||
| `P2P_SCAN_CMD` | `clamscan --no-summary --infected` | Needs an image built with `--build-arg INSTALL_CLAMAV=1`. |
|
||||
| `P2P_STALE_DAYS` | `7` | Holder older than this is shown as stale. |
|
||||
| `P2P_KEEP_MIN_VIEWS` | `3` | Retention: views in the last `P2P_KEEP_DAYS` that make a server copy "top". |
|
||||
| `P2P_KEEP_DAYS` | `30` | Window for counting views. |
|
||||
| `P2P_KEEP_RECENT_DAYS` | `14` | Retention: played this recently = "recent". |
|
||||
| `P2P_INTAKE_DIR` | `<DB dir>/p2p-intake` | Quarantine for device uploads. Never served. |
|
||||
| `P2P_INTAKE_MAX_BYTES` | `3 GiB` | Largest accepted intake upload. |
|
||||
|
||||
Client settings (`data.settings`, per profile): `p2pShare: true` (let other devices
|
||||
download my saved videos, and report holdings), `p2pReceive: true` (fetch from other
|
||||
devices when YouTube and the server can't serve).
|
||||
|
||||
## Flows
|
||||
|
||||
1. **Server fetch (existing media cache) → verified content** (plan 009).
|
||||
`runFetch` / `runOptimize` hash the promoted file, store `media_cache.sha256`, run the
|
||||
scan if enabled, then upsert `p2p_content` (`origin 'server'`). `/api/download` sends
|
||||
`X-Content-SHA256`. A backfill hashes already-cached files at boot, one at a time.
|
||||
2. **Device save** (plan 012). The OPFS worker hashes while it writes. If the server
|
||||
sent `X-Content-SHA256` and the hash differs, the save fails (bonus integrity check).
|
||||
The device DB (`IndexedDB ytp-device`, store `files`) records `{videoId, cid, size,
|
||||
savedAt, lastCheckedAt, state}`; `state` is `verified` when the hashes matched,
|
||||
`unverified` when the server sent no hash, `unhashed` for old saves and the
|
||||
main-thread fallback path.
|
||||
3. **Holdings sync** (plan 013). Device registers once (`POST /api/p2p/device` →
|
||||
`deviceId` + `secret`, kept in `localStorage.ytpDevice`). It reports its holdings
|
||||
(`POST /api/p2p/holdings`, full list at launch, deltas after save/delete). The server
|
||||
accepts only cids in `p2p_content` with `status='verified'`; unknown cids come back
|
||||
in `unknown` (candidates for intake). When the server still has the file it returns
|
||||
up to 5 **range challenges**; a correct answer sets `trust='challenged'`, a wrong one
|
||||
removes the holder. `last_verified_at` = time of the last report where the device
|
||||
re-checked the file (exists, same size; full re-hash every 30 days). A full report
|
||||
marks every active holder row of that device that is missing from the list as
|
||||
`removed`. There is **no TTL**.
|
||||
4. **Presence** (plan 014). `/ws/p2p` socket per device, authenticated with the device
|
||||
secret. Online status lives only in memory. The hub also relays WebRTC signalling
|
||||
between two online devices and carries server → device requests (plan 018).
|
||||
5. **Availability** (plans 014/015). `GET /api/p2p/holders?v=<videoId>` lists cids and
|
||||
their holders: opaque peer id (never the fingerprint/profile), `online`,
|
||||
`lastVerifiedAt`, `stale`, `trust`, plus `serverHas`. The UI shows e.g.
|
||||
"📡 On 3 devices · 1 online now · last checked 2 d ago", with stale holders greyed.
|
||||
6. **Peer download** (plan 017). WebRTC data channel (STUN only, same ICE list as watch
|
||||
party), 64 KiB frames with `bufferedAmount` back-pressure, receiver writes through a
|
||||
worker into OPFS while hashing; only a matching SHA-256 is committed. The new copy
|
||||
is a holder at the next report. Download-then-play; no progressive peer streaming.
|
||||
Used when `/api/streams` fails and the device has no copy, and from a
|
||||
"Get from a device" button.
|
||||
7. **Intake** (plan 016). A device can hand a file to the server
|
||||
(`POST /api/p2p/intake` → ticket, `PUT` the bytes). The server writes it to the
|
||||
quarantine dir, hashes it, runs `validateMedia()`, runs the scan if enabled, and only
|
||||
then inserts `p2p_content` (`origin 'intake'`). If the server has no copy of that
|
||||
video it adopts the file into the media cache (budget permitting).
|
||||
8. **Rehydrate** (plan 018). When a video is requested, its source fails, the server
|
||||
evicted its copy, and a verified holder is online with `p2pShare` on, the hub asks
|
||||
that device to upload it through intake (known cid → quick accept).
|
||||
9. **Retention** (plan 010). Views are counted per video per day. When the media cache
|
||||
needs room it evicts in this order: copies that are neither "top"
|
||||
(`views in P2P_KEEP_DAYS ≥ P2P_KEEP_MIN_VIEWS`) nor "recent" (played within
|
||||
`P2P_KEEP_RECENT_DAYS`), fewest views first, then oldest; only then the qualifying
|
||||
ones by LRU. The 10-minute play protection and `MEDIA_CACHE_MAX_BYTES` stay.
|
||||
Evicting a server copy never deletes `p2p_content` or holder rows.
|
||||
|
||||
## Security rules every plan must keep
|
||||
|
||||
- No cid enters `p2p_content` unless the SERVER computed it over bytes it holds and
|
||||
`validateMedia` passed. Clients can never insert or edit content rows, views or trust.
|
||||
- Intake files live in `P2P_INTAKE_DIR`, never under `./public` or `MEDIA_DIR`, and
|
||||
are deleted on failure.
|
||||
- Device secrets: 32 random bytes, only `sha256(secret)` stored, compared with
|
||||
`timingSafeEqual`.
|
||||
- Holder lists never expose fingerprints, profile names or IPs; a peer id is
|
||||
`sha256('peer:' + device_id).slice(0, 12)`.
|
||||
- The hub relays signalling only between two authenticated, online devices, with a
|
||||
per-socket message budget.
|
||||
- `P2P_ENABLED=0` must leave the rest of the app working exactly as before.
|
||||
- Jobs stay server-owned; never pass a request `AbortSignal` into them (CLAUDE.md).
|
||||
|
||||
## Where this differs from the earlier phase 02–06 draft
|
||||
|
||||
| Earlier draft | Now |
|
||||
|---------------|-----|
|
||||
| "Default-off" P2P subsystem; "no inventory/upload from default settings" | P2P on by default; devices report holdings and seed by default (can be turned off). |
|
||||
| Mandatory scanner, "scan skip is failure" | Scanner off by default (`P2P_MALWARE_SCAN=0`); hash + `validateMedia` mandatory. |
|
||||
| Short-lived online leases; "cache availability only as an expiring hint" | Persistent holder rows with `last_verified_at` and a stale marker; online status is separate. |
|
||||
| Migrate all localStorage (`_ytpdata`) to IndexedDB first | Not now: the device DB holds files + cids only. `_ytpdata` stays in localStorage (lower risk). |
|
||||
| Collections, invitations, scoped principals, signed manifests, TURN, renditions lineage | Deferred. Scope is one shared catalog + device secrets; add later if needed. |
|
||||
| Separate `server/p2p/*` directory with migrations ledger | Flat files `server/p2p-*.js` matching the repo's style (`party.js`, `remote.js`). |
|
||||
Reference in New Issue
Block a user