From f465cc86d04838e8141f73acf74531166e9c0e75 Mon Sep 17 00:00:00 2001 From: Jonathan Sykes Date: Tue, 13 Oct 2026 22:42:12 +0800 Subject: [PATCH] Plan built-in MySQL search index and a Swoole benchmark --- plans/mysql-search-swoole-plan.md | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 plans/mysql-search-swoole-plan.md diff --git a/plans/mysql-search-swoole-plan.md b/plans/mysql-search-swoole-plan.md new file mode 100644 index 0000000..abba478 --- /dev/null +++ b/plans/mysql-search-swoole-plan.md @@ -0,0 +1,39 @@ +# Plan: built-in search data in MySQL, and a Swoole port (benchmark first) + +Owner request 2026-10-13. Nothing here is built yet. Phase 0 decides whether phases 2–3 happen at all. + +## Why +- Cold search is ~12.6 s (yt-dlp) and every request pays the ~250 ms home -> VPS -> WireGuard path several times (see memory `worship-slow-cloudflare-plan`). +- Today's "quick search" is `search_cache` (query string -> JSON blob, libsql) plus `video_catalog` / `GET /api/search/local`. It only helps for a query someone already ran. +- Goal: answer *any* worship-song query from a local index in a few ms, before touching YouTube, and handle more concurrent users per box. + +## Phase 0 - measure before committing (1 day, no production change) +1. Capture a baseline on the current Bun/Hono server: `/api/search/local`, cached `/api/search`, `/api/version`, static asset, `/api/media/` range request. Tool: `bombardier`/`oha` at 1/10/50/200 connections, from the homelab VM itself (removes the ISP path from the numbers). +2. Build a throwaway Swoole (OpenSwoole/Swoole 5, PHP 8.3, coroutine MySQL pool) service that serves the same three endpoints from MySQL. Base it on `personal/AppStarterSwooleLite`. Same box, same data. +3. Compare req/s, p50/p95/p99, RSS, CPU at each concurrency level. Also compare streaming a 10 MB slice (proxy path), since that is where Bun may already win or tie. +4. Decision gate: port only if Swoole is clearly better (>= 1.5x req/s or half the p99) on the endpoints that matter. If the gap is small, keep Bun and only do the MySQL index (phase 1). + +## Phase 1 - MySQL search index (independent of the port) +- Table `song_index(video_id PK, title, uploader, title_norm, uploader_norm, duration, views, published, thumb, lang, source, updated_at)` with `FULLTEXT(title_norm, uploader_norm)` (ngram parser for CJK / Tagalog / Cebuano titles) plus a prefix index on `title_norm` for as-you-type. +- Seed from data we already own: `video_catalog`, `search_cache` result blobs (de-duplicated by video id), saved/cached media ids, lyrics-library ids, admin-curated lists. Optionally a one-off yt-dlp crawl of known worship channels. +- Query path: normalise (lower-case, strip accents/punctuation) -> `MATCH ... AGAINST` in boolean mode with title boost -> rank by relevance, then views/recency -> return card JSON in the same shape as `/api/search`. Answer from MySQL first; if fewer than N good hits, fall through to yt-dlp and write what comes back into `song_index` (self-growing). +- Background refresh job: re-resolve stale rows, drop dead videos. +- API: keep `/api/search` contract unchanged; add `source: "index"|"live"` to the response so the UI can show instantly and then top up. +- MySQL runs on the homelab (same host as the app, local socket, no WireGuard hop). Needs a decision on engine: reuse an existing Dokploy MySQL service or add a dedicated one. +- Tests: ranking fixtures (exact title, partial, typo, non-Latin), fall-through behaviour, de-dupe, and a latency budget (< 30 ms p95 at 10k rows, < 80 ms at 1M). + +## Phase 2 - Swoole port (only if phase 0 says yes) +- Strangler approach, not a rewrite: Swoole service takes over read-heavy, stateless routes first (`/api/search*`, `/api/version`, static + manifest delivery, notes read). Bun keeps the stateful parts (yt-dlp pool, media cache writes, party/remote WebSockets, p2p hub, transcription) behind it; a small router (Traefik path rules or Swoole reverse-proxy) splits traffic. +- Move route groups one by one behind a flag; each move must pass the existing server tests re-expressed against the new service plus the Playwright suites unchanged. +- Things that do not port cleanly and stay in Bun for now: `yt-dlp` worker pool, WebSocket party/remote hubs, OPFS-related admin tooling. +- Rollback = flip the router rule back; both services share the same MySQL. + +## Phase 3 - cutover and capacity +- Beta first (beta.worship.hesed.sbs), 1 week soak, then main. +- Publish capacity numbers (users per box) from the phase-0 harness re-run against the final stack. + +## Risks / open decisions for the owner +- Does the homelab have the headroom for MySQL + Swoole next to the app (the USB cache drive is currently unmounted and the host froze on past USB incidents)? +- Is it OK to run a PHP service in this repo, or should the port live in a separate repo (`ytplayer-swoole`)? +- Index scope: only songs users have searched, or a proactive crawl of worship channels (more storage and yt-dlp load)? +- Mostly a latency problem is the network path, not the runtime: a Swoole port will not fix the 250 ms RTT. The Cloudflare tunnel plan still stands and is complementary.