ytplayer/plans/mysql-search-swoole-plan.md

Plan: built-in search data in MySQL, and a Swoole port (benchmark first)

Owner request 2026-10-13. Nothing here is built yet. Phase 0 decides whether phases 2–3 happen at all.

Why

• Cold search is ~12.6 s (yt-dlp) and every request pays the ~250 ms home -> VPS -> WireGuard path several times (see memory worship-slow-cloudflare-plan).
• Today's "quick search" is search_cache (query string -> JSON blob, libsql) plus video_catalog / GET /api/search/local. It only helps for a query someone already ran.
• Goal: answer any worship-song query from a local index in a few ms, before touching YouTube, and handle more concurrent users per box.

Phase 0 - measure before committing (1 day, no production change)

1. Capture a baseline on the current Bun/Hono server: /api/search/local, cached /api/search, /api/version, static asset, /api/media/<id> range request. Tool: bombardier/oha at 1/10/50/200 connections, from the homelab VM itself (removes the ISP path from the numbers).
2. Build a throwaway Swoole (OpenSwoole/Swoole 5, PHP 8.3, coroutine MySQL pool) service that serves the same three endpoints from MySQL. Base it on personal/AppStarterSwooleLite. Same box, same data.
3. Compare req/s, p50/p95/p99, RSS, CPU at each concurrency level. Also compare streaming a 10 MB slice (proxy path), since that is where Bun may already win or tie.
4. Decision gate: port only if Swoole is clearly better (>= 1.5x req/s or half the p99) on the endpoints that matter. If the gap is small, keep Bun and only do the MySQL index (phase 1).

Phase 1 - MySQL search index (independent of the port)

• Table song_index(video_id PK, title, uploader, title_norm, uploader_norm, duration, views, published, thumb, lang, source, updated_at) with FULLTEXT(title_norm, uploader_norm) (ngram parser for CJK / Tagalog / Cebuano titles) plus a prefix index on title_norm for as-you-type.
• Seed from data we already own: video_catalog, search_cache result blobs (de-duplicated by video id), saved/cached media ids, lyrics-library ids, admin-curated lists. Optionally a one-off yt-dlp crawl of known worship channels.
• Query path: normalise (lower-case, strip accents/punctuation) -> MATCH ... AGAINST in boolean mode with title boost -> rank by relevance, then views/recency -> return card JSON in the same shape as /api/search. Answer from MySQL first; if fewer than N good hits, fall through to yt-dlp and write what comes back into song_index (self-growing).
• Background refresh job: re-resolve stale rows, drop dead videos.
• API: keep /api/search contract unchanged; add source: "index"|"live" to the response so the UI can show instantly and then top up.
• MySQL runs on the homelab (same host as the app, local socket, no WireGuard hop). Needs a decision on engine: reuse an existing Dokploy MySQL service or add a dedicated one.
• Tests: ranking fixtures (exact title, partial, typo, non-Latin), fall-through behaviour, de-dupe, and a latency budget (< 30 ms p95 at 10k rows, < 80 ms at 1M).

Phase 2 - Swoole port (only if phase 0 says yes)

• Strangler approach, not a rewrite: Swoole service takes over read-heavy, stateless routes first (/api/search*, /api/version, static + manifest delivery, notes read). Bun keeps the stateful parts (yt-dlp pool, media cache writes, party/remote WebSockets, p2p hub, transcription) behind it; a small router (Traefik path rules or Swoole reverse-proxy) splits traffic.
• Move route groups one by one behind a flag; each move must pass the existing server tests re-expressed against the new service plus the Playwright suites unchanged.
• Things that do not port cleanly and stay in Bun for now: yt-dlp worker pool, WebSocket party/remote hubs, OPFS-related admin tooling.
• Rollback = flip the router rule back; both services share the same MySQL.

Phase 3 - cutover and capacity

• Beta first (beta.worship.hesed.sbs), 1 week soak, then main.
• Publish capacity numbers (users per box) from the phase-0 harness re-run against the final stack.

Risks / open decisions for the owner

• Does the homelab have the headroom for MySQL + Swoole next to the app (the USB cache drive is currently unmounted and the host froze on past USB incidents)?
• Is it OK to run a PHP service in this repo, or should the port live in a separate repo (ytplayer-swoole)?
• Index scope: only songs users have searched, or a proactive crawl of worship channels (more storage and yt-dlp load)?
• Mostly a latency problem is the network path, not the runtime: a Swoole port will not fix the 250 ms RTT. The Cloudflare tunnel plan still stands and is complementary.