40 lines
4.5 KiB
Markdown
40 lines
4.5 KiB
Markdown
# Plan: built-in search data in MySQL, and a Swoole port (benchmark first)
|
||
|
||
Owner request 2026-10-13. Nothing here is built yet. Phase 0 decides whether phases 2–3 happen at all.
|
||
|
||
## Why
|
||
- Cold search is ~12.6 s (yt-dlp) and every request pays the ~250 ms home -> VPS -> WireGuard path several times (see memory `worship-slow-cloudflare-plan`).
|
||
- Today's "quick search" is `search_cache` (query string -> JSON blob, libsql) plus `video_catalog` / `GET /api/search/local`. It only helps for a query someone already ran.
|
||
- Goal: answer *any* worship-song query from a local index in a few ms, before touching YouTube, and handle more concurrent users per box.
|
||
|
||
## Phase 0 - measure before committing (1 day, no production change)
|
||
1. Capture a baseline on the current Bun/Hono server: `/api/search/local`, cached `/api/search`, `/api/version`, static asset, `/api/media/<id>` range request. Tool: `bombardier`/`oha` at 1/10/50/200 connections, from the homelab VM itself (removes the ISP path from the numbers).
|
||
2. Build a throwaway Swoole (OpenSwoole/Swoole 5, PHP 8.3, coroutine MySQL pool) service that serves the same three endpoints from MySQL. Base it on `personal/AppStarterSwooleLite`. Same box, same data.
|
||
3. Compare req/s, p50/p95/p99, RSS, CPU at each concurrency level. Also compare streaming a 10 MB slice (proxy path), since that is where Bun may already win or tie.
|
||
4. Decision gate: port only if Swoole is clearly better (>= 1.5x req/s or half the p99) on the endpoints that matter. If the gap is small, keep Bun and only do the MySQL index (phase 1).
|
||
|
||
## Phase 1 - MySQL search index (independent of the port)
|
||
- Table `song_index(video_id PK, title, uploader, title_norm, uploader_norm, duration, views, published, thumb, lang, source, updated_at)` with `FULLTEXT(title_norm, uploader_norm)` (ngram parser for CJK / Tagalog / Cebuano titles) plus a prefix index on `title_norm` for as-you-type.
|
||
- Seed from data we already own: `video_catalog`, `search_cache` result blobs (de-duplicated by video id), saved/cached media ids, lyrics-library ids, admin-curated lists. Optionally a one-off yt-dlp crawl of known worship channels.
|
||
- Query path: normalise (lower-case, strip accents/punctuation) -> `MATCH ... AGAINST` in boolean mode with title boost -> rank by relevance, then views/recency -> return card JSON in the same shape as `/api/search`. Answer from MySQL first; if fewer than N good hits, fall through to yt-dlp and write what comes back into `song_index` (self-growing).
|
||
- Background refresh job: re-resolve stale rows, drop dead videos.
|
||
- API: keep `/api/search` contract unchanged; add `source: "index"|"live"` to the response so the UI can show instantly and then top up.
|
||
- MySQL runs on the homelab (same host as the app, local socket, no WireGuard hop). Needs a decision on engine: reuse an existing Dokploy MySQL service or add a dedicated one.
|
||
- Tests: ranking fixtures (exact title, partial, typo, non-Latin), fall-through behaviour, de-dupe, and a latency budget (< 30 ms p95 at 10k rows, < 80 ms at 1M).
|
||
|
||
## Phase 2 - Swoole port (only if phase 0 says yes)
|
||
- Strangler approach, not a rewrite: Swoole service takes over read-heavy, stateless routes first (`/api/search*`, `/api/version`, static + manifest delivery, notes read). Bun keeps the stateful parts (yt-dlp pool, media cache writes, party/remote WebSockets, p2p hub, transcription) behind it; a small router (Traefik path rules or Swoole reverse-proxy) splits traffic.
|
||
- Move route groups one by one behind a flag; each move must pass the existing server tests re-expressed against the new service plus the Playwright suites unchanged.
|
||
- Things that do not port cleanly and stay in Bun for now: `yt-dlp` worker pool, WebSocket party/remote hubs, OPFS-related admin tooling.
|
||
- Rollback = flip the router rule back; both services share the same MySQL.
|
||
|
||
## Phase 3 - cutover and capacity
|
||
- Beta first (beta.worship.hesed.sbs), 1 week soak, then main.
|
||
- Publish capacity numbers (users per box) from the phase-0 harness re-run against the final stack.
|
||
|
||
## Risks / open decisions for the owner
|
||
- Does the homelab have the headroom for MySQL + Swoole next to the app (the USB cache drive is currently unmounted and the host froze on past USB incidents)?
|
||
- Is it OK to run a PHP service in this repo, or should the port live in a separate repo (`ytplayer-swoole`)?
|
||
- Index scope: only songs users have searched, or a proactive crawl of worship channels (more storage and yt-dlp load)?
|
||
- Mostly a latency problem is the network path, not the runtime: a Swoole port will not fix the 250 ms RTT. The Cloudflare tunnel plan still stands and is complementary.
|