ytplayer/plans/done/006-innertube-search-48066b.md

id, title, created, depends_on, est_files

id	title	created	depends_on	est_files
006-innertube-search-48066b	Answer searches from YouTube InnerTube directly with yt-dlp fallback	2026-09-29	001-perf-timing-marks-105acc	4

006 — InnerTube search with yt-dlp fallback

Objective

/api/search takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process (it already uses --flat-playlist). One HTTPS POST to YouTube's InnerTube search API returns the same data in a few hundred ms. After this plan /api/search tries InnerTube first (6 s timeout), maps results to the existing card shape, and falls back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos. Response JSON shape is unchanged ({ ok, results }), so app.js and the Tauri bridge contract are untouched.

Context the executor must NOT rediscover

A trimmed real response is committed at server/fixtures/innertube-search.json (3 real videos, one shelfRenderer to ignore, one live item without lengthText, and a trailing continuationItemRenderer). The paths (verified 2026-09-29):

contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
  .itemSectionRenderer.contents[].videoRenderer:
     videoId                                   -> id
     title.runs[0].text                        -> title
     ownerText.runs[0].text                    -> channel
     ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId          -> channelId (UC…)
     ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl  -> '/@handle' or '/channel/UC…'
     lengthText.simpleText  "5:43" | "1:59:47" | absent (live)             -> duration seconds (0 if absent)

Request that works (no key needed):

POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
Content-Type: application/json
{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}

The first page has ~15–20 videos (yt-dlp returned SEARCH_LIMIT = 25); that is acceptable. Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first cold request got HTTP 403 from InnerTube and fell back to yt-dlp — expected, that is what the fallback is for. Do not "fix" the 403 by adding cookies/keys.

Existing card shape — server/server.js:293-305 slimEntry:

return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };

channelUrl in yt-dlp output is a full URL like https://www.youtube.com/channel/UC… or https://www.youtube.com/@handle.

Current route server/server.js:365-395 (inside it):

  let mine = [];
  try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
  try {
    const out = await runYtdlpResilient([
      `ytsearch${SEARCH_LIMIT}:${q}`,
      '--dump-json', '--flat-playlist',
      '--no-warnings', '--ignore-errors',
    ]);
    const results = [...mine, ...parseCards(out)];

Server tests use bun:test (see server/notes.test.js); server/package.json "test" script runs each file separately joined by &&.

Steps

1. Create server/innertube.js:

   /* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
    * parseSearch() is pure (tested against fixtures/innertube-search.json);
    * search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
   const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
   
   export function lengthToSeconds(s) {
     if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
     return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
   }
   
   export function parseSearch(json) {
     const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
       ?.sectionListRenderer?.contents;
     if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
     const out = [];
     for (const s of sections) {
       for (const it of s?.itemSectionRenderer?.contents || []) {
         const v = it && it.videoRenderer;
         if (!v || typeof v.videoId !== 'string') continue;
         const owner = v.ownerText?.runs?.[0] || {};
         const be = owner.navigationEndpoint?.browseEndpoint || {};
         const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
         out.push({
           id: v.videoId,
           title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
           channel: owner.text || '',
           channelId: be.browseId || '',
           channelUrl: path ? `https://www.youtube.com${path}` : '',
           duration: lengthToSeconds(v.lengthText?.simpleText),
           thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
         });
       }
     }
     return out;
   }
   
   export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
     const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
       method: 'POST',
       headers: { 'Content-Type': 'application/json' },
       body: JSON.stringify({ context: { client: CLIENT }, query: q }),
       signal: AbortSignal.timeout(timeoutMs),
     });
     if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
     return parseSearch(await res.json());
   }
   

2. Create server/innertube.test.js:

   import { test, expect } from 'bun:test';
   import { readFileSync } from 'node:fs';
   import { parseSearch, lengthToSeconds, search } from './innertube.js';
   const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
   
   test('parses video renderers into slim cards', () => {
     const r = parseSearch(fixture);
     expect(r.length).toBe(4);
     expect(r[0]).toEqual({
       id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
       channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
       duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
     });
     expect(r[1].duration).toBe(7187);
     expect(r[3].duration).toBe(0); // live, no lengthText
   });
   test('lengthToSeconds', () => {
     expect(lengthToSeconds('5:43')).toBe(343);
     expect(lengthToSeconds('1:59:47')).toBe(7187);
     expect(lengthToSeconds('LIVE')).toBe(0);
     expect(lengthToSeconds(undefined)).toBe(0);
   });
   test('unexpected shape throws (caller falls back to yt-dlp)', () => {
     expect(() => parseSearch({})).toThrow();
   });
   test('search() throws on HTTP error', async () => {
     const fetchImpl = async () => new Response('no', { status: 429 });
     await expect(search('x', { fetchImpl })).rejects.toThrow('429');
   });
   

   If r[1].duration in the fixture differs from 7187, compute it from the fixture's lengthText and use that value (the fixture is the source of truth).
3. server/package.json "test" script — append && bun test ./innertube.test.js.
4. server/server.js — add import next to the other local imports: import * as innertube from './innertube.js';
5. server/server.js /api/search — replace the try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)]; head with:

   try {
     let yt = [];
     try {
       yt = await innertube.search(q);
     } catch (e) {
       console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
     }
     if (!yt.length) {
       const out = await runYtdlpResilient([
         `ytsearch${SEARCH_LIMIT}:${q}`,
         '--dump-json', '--flat-playlist',
         '--no-warnings', '--ignore-errors',
       ]);
       yt = parseCards(out);
     }
     const results = [...mine, ...yt];
   

   Everything after (searchCache.set, return, catch) stays as is.
6. Add an env kill-switch: at the top of the search block, if (process.env.SEARCH_INNERTUBE === '0') skip the innertube call (leave yt = []). Document it in docker-compose.yml as a commented line # SEARCH_INNERTUBE: "0" # force yt-dlp search next to the other commented env vars.

Out of scope / do NOT touch

• /api/channel, /api/playlist/expand (still yt-dlp). No continuation paging.
• Card shape, searchCache, app.js.

Verification

cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
kill $SRV; true
grep -c "innertube failed" /tmp/ytp006.log

Expected: innertube tests 4 pass 0 fail; all files 0 fail; the search returns {"ok":true,"results":[{"id":… well under 2 s when the network allows (if the container has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).

Report format (executor: follow exactly)

Output ONLY the following, no other prose:

1. git diff (unified) of all changes.
2. Raw output of the Verification commands.
3. Findings: — max 10 lines.

Do not commit. Do not push. Do not touch files outside the Steps.

Execution log

• Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 0.
• Orchestrator re-ran Verification: innertube tests 4 pass; bun run test all 6 server files 0 fail; SERVER_OK; live /api/search 1.05 s and 0.74 s with 0 InnerTube failures logged; new files byte-identical to the plan's code.
• Orchestrator note: the executor's pasted diff had fabricated blob hashes and a typo (@hono/node-node-server) that does NOT exist in the real server/package.json; the on-disk files were checked directly.
• Executor Findings (verbatim): Plan completed successfully. All steps implemented: innertube.js module created with parseSearch and search functions; innertube.test.js with 4 passing tests verifying fixture parsing, length conversion, error handling; server.js imports innertube and updated /api/search route to try InnerTube first (6s timeout) with fallback to yt-dlp when zero results or any error; kill-switch (SEARCH_INNERTUBE=0) added to server logic and documented in docker-compose.yml; all existing tests continue passing (62 total passes across all test files); search endpoint responds in 0.964s with correct results from InnerTube without fallback (0 failures logged).