Answer searches from YouTube InnerTube directly with yt-dlp fallback

2026-09-30 06:58:05 +00:00
parent d61809f5c4
commit 556436da34
7 changed files with 104 additions and 8 deletions
--- a/plans/done/006-innertube-search-48066b.md
+++ b/plans/done/006-innertube-search-48066b.md
@@ -0,0 +1,219 @@
+---
+id: 006-innertube-search-48066b
+title: Answer searches from YouTube InnerTube directly with yt-dlp fallback
+created: 2026-09-29
+depends_on: [001-perf-timing-marks-105acc]
+est_files: 4
+---
+
+# 006 — InnerTube search with yt-dlp fallback
+
+## Objective
+
+`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process
+(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search
+API returns the same data in a few hundred ms. After this plan `/api/search` tries
+InnerTube first (6 s timeout), maps results to the existing card shape, and falls
+back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos.
+Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri
+bridge contract are untouched.
+
+## Context the executor must NOT rediscover
+
+A trimmed real response is committed at `server/fixtures/innertube-search.json`
+(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`,
+and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29):
+
+```
+contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
+  .itemSectionRenderer.contents[].videoRenderer:
+     videoId                                   -> id
+     title.runs[0].text                        -> title
+     ownerText.runs[0].text                    -> channel
+     ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId          -> channelId (UC…)
+     ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl  -> '/@handle' or '/channel/UC…'
+     lengthText.simpleText  "5:43" | "1:59:47" | absent (live)             -> duration seconds (0 if absent)
+```
+
+Request that works (no key needed):
+```
+POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
+Content-Type: application/json
+{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}
+```
+The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable.
+Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first
+cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the
+fallback is for. Do not "fix" the 403 by adding cookies/keys.
+
+Existing card shape — `server/server.js:293-305` `slimEntry`:
+```js
+return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };
+```
+`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`.
+
+Current route `server/server.js:365-395` (inside it):
+```js
+  let mine = [];
+  try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
+  try {
+    const out = await runYtdlpResilient([
+      `ytsearch${SEARCH_LIMIT}:${q}`,
+      '--dump-json', '--flat-playlist',
+      '--no-warnings', '--ignore-errors',
+    ]);
+    const results = [...mine, ...parseCards(out)];
+```
+
+Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script
+runs each file separately joined by `&&`.
+
+## Steps
+
+1. Create `server/innertube.js`:
+   ```js
+   /* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
+    * parseSearch() is pure (tested against fixtures/innertube-search.json);
+    * search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
+   const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
+
+   export function lengthToSeconds(s) {
+     if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
+     return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
+   }
+
+   export function parseSearch(json) {
+     const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
+       ?.sectionListRenderer?.contents;
+     if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
+     const out = [];
+     for (const s of sections) {
+       for (const it of s?.itemSectionRenderer?.contents || []) {
+         const v = it && it.videoRenderer;
+         if (!v || typeof v.videoId !== 'string') continue;
+         const owner = v.ownerText?.runs?.[0] || {};
+         const be = owner.navigationEndpoint?.browseEndpoint || {};
+         const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
+         out.push({
+           id: v.videoId,
+           title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
+           channel: owner.text || '',
+           channelId: be.browseId || '',
+           channelUrl: path ? `https://www.youtube.com${path}` : '',
+           duration: lengthToSeconds(v.lengthText?.simpleText),
+           thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
+         });
+       }
+     }
+     return out;
+   }
+
+   export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
+     const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
+       method: 'POST',
+       headers: { 'Content-Type': 'application/json' },
+       body: JSON.stringify({ context: { client: CLIENT }, query: q }),
+       signal: AbortSignal.timeout(timeoutMs),
+     });
+     if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
+     return parseSearch(await res.json());
+   }
+   ```
+2. Create `server/innertube.test.js`:
+   ```js
+   import { test, expect } from 'bun:test';
+   import { readFileSync } from 'node:fs';
+   import { parseSearch, lengthToSeconds, search } from './innertube.js';
+   const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
+
+   test('parses video renderers into slim cards', () => {
+     const r = parseSearch(fixture);
+     expect(r.length).toBe(4);
+     expect(r[0]).toEqual({
+       id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
+       channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
+       duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
+     });
+     expect(r[1].duration).toBe(7187);
+     expect(r[3].duration).toBe(0); // live, no lengthText
+   });
+   test('lengthToSeconds', () => {
+     expect(lengthToSeconds('5:43')).toBe(343);
+     expect(lengthToSeconds('1:59:47')).toBe(7187);
+     expect(lengthToSeconds('LIVE')).toBe(0);
+     expect(lengthToSeconds(undefined)).toBe(0);
+   });
+   test('unexpected shape throws (caller falls back to yt-dlp)', () => {
+     expect(() => parseSearch({})).toThrow();
+   });
+   test('search() throws on HTTP error', async () => {
+     const fetchImpl = async () => new Response('no', { status: 429 });
+     await expect(search('x', { fetchImpl })).rejects.toThrow('429');
+   });
+   ```
+   If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's
+   `lengthText` and use that value (the fixture is the source of truth).
+3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`.
+4. `server/server.js` — add import next to the other local imports:
+   `import * as innertube from './innertube.js';`
+5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];`
+   head with:
+   ```js
+   try {
+     let yt = [];
+     try {
+       yt = await innertube.search(q);
+     } catch (e) {
+       console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
+     }
+     if (!yt.length) {
+       const out = await runYtdlpResilient([
+         `ytsearch${SEARCH_LIMIT}:${q}`,
+         '--dump-json', '--flat-playlist',
+         '--no-warnings', '--ignore-errors',
+       ]);
+       yt = parseCards(out);
+     }
+     const results = [...mine, ...yt];
+   ```
+   Everything after (`searchCache.set`, return, catch) stays as is.
+6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')`
+   skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented
+   line `# SEARCH_INNERTUBE: "0"   # force yt-dlp search` next to the other commented env vars.
+
+## Out of scope / do NOT touch
+
+- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging.
+- Card shape, `searchCache`, `app.js`.
+
+## Verification
+
+```bash
+cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
+bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
+[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
+time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
+kill $SRV; true
+grep -c "innertube failed" /tmp/ytp006.log
+```
+
+Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns
+`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container
+has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).
+
+## Report format (executor: follow exactly)
+
+Output ONLY the following, no other prose:
+
+1. `git diff` (unified) of all changes.
+2. Raw output of the Verification commands.
+3. `Findings:` — max 10 lines.
+
+Do not commit. Do not push. Do not touch files outside the Steps.
+
+## Execution log
+
+- Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 0.
+- Orchestrator re-ran Verification: innertube tests 4 pass; `bun run test` all 6 server files 0 fail; `SERVER_OK`; live `/api/search` 1.05 s and 0.74 s with 0 InnerTube failures logged; new files byte-identical to the plan's code.
+- Orchestrator note: the executor's pasted diff had fabricated blob hashes and a typo (`@hono/node-node-server`) that does NOT exist in the real `server/package.json`; the on-disk files were checked directly.
+- Executor Findings (verbatim): Plan completed successfully. All steps implemented: innertube.js module created with parseSearch and search functions; innertube.test.js with 4 passing tests verifying fixture parsing, length conversion, error handling; server.js imports innertube and updated /api/search route to try InnerTube first (6s timeout) with fallback to yt-dlp when zero results or any error; kill-switch (SEARCH_INNERTUBE=0) added to server logic and documented in docker-compose.yml; all existing tests continue passing (62 total passes across all test files); search endpoint responds in 0.964s with correct results from InnerTube without fallback (0 failures logged).