220 lines
10 KiB
Markdown
220 lines
10 KiB
Markdown
---
|
||
id: 006-innertube-search-48066b
|
||
title: Answer searches from YouTube InnerTube directly with yt-dlp fallback
|
||
created: 2026-09-29
|
||
depends_on: [001-perf-timing-marks-105acc]
|
||
est_files: 4
|
||
---
|
||
|
||
# 006 — InnerTube search with yt-dlp fallback
|
||
|
||
## Objective
|
||
|
||
`/api/search` takes 4–5 s on prod; ~3.5 s of it is starting a yt-dlp process
|
||
(it already uses `--flat-playlist`). One HTTPS POST to YouTube's InnerTube search
|
||
API returns the same data in a few hundred ms. After this plan `/api/search` tries
|
||
InnerTube first (6 s timeout), maps results to the existing card shape, and falls
|
||
back to the current yt-dlp path on ANY error or when InnerTube returns 0 videos.
|
||
Response JSON shape is unchanged (`{ ok, results }`), so app.js and the Tauri
|
||
bridge contract are untouched.
|
||
|
||
## Context the executor must NOT rediscover
|
||
|
||
A trimmed real response is committed at `server/fixtures/innertube-search.json`
|
||
(3 real videos, one `shelfRenderer` to ignore, one live item without `lengthText`,
|
||
and a trailing `continuationItemRenderer`). The paths (verified 2026-09-29):
|
||
|
||
```
|
||
contents.twoColumnSearchResultsRenderer.primaryContents.sectionListRenderer.contents[]
|
||
.itemSectionRenderer.contents[].videoRenderer:
|
||
videoId -> id
|
||
title.runs[0].text -> title
|
||
ownerText.runs[0].text -> channel
|
||
ownerText.runs[0].navigationEndpoint.browseEndpoint.browseId -> channelId (UC…)
|
||
ownerText.runs[0].navigationEndpoint.browseEndpoint.canonicalBaseUrl -> '/@handle' or '/channel/UC…'
|
||
lengthText.simpleText "5:43" | "1:59:47" | absent (live) -> duration seconds (0 if absent)
|
||
```
|
||
|
||
Request that works (no key needed):
|
||
```
|
||
POST https://www.youtube.com/youtubei/v1/search?prettyPrint=false
|
||
Content-Type: application/json
|
||
{"context":{"client":{"clientName":"WEB","clientVersion":"2.20250101.00.00","hl":"en","gl":"US"}},"query":"<q>"}
|
||
```
|
||
The first page has ~15–20 videos (yt-dlp returned `SEARCH_LIMIT` = 25); that is acceptable.
|
||
Dry run of this plan (2026-09-29): searches answered in 0.68–0.83 s end to end; the very first
|
||
cold request got `HTTP 403` from InnerTube and fell back to yt-dlp — expected, that is what the
|
||
fallback is for. Do not "fix" the 403 by adding cookies/keys.
|
||
|
||
Existing card shape — `server/server.js:293-305` `slimEntry`:
|
||
```js
|
||
return { id, title, channel, channelId, channelUrl, duration, thumbnail: `https://i.ytimg.com/vi/${id}/mqdefault.jpg` };
|
||
```
|
||
`channelUrl` in yt-dlp output is a full URL like `https://www.youtube.com/channel/UC…` or `https://www.youtube.com/@handle`.
|
||
|
||
Current route `server/server.js:365-395` (inside it):
|
||
```js
|
||
let mine = [];
|
||
try { mine = (await notesDb.listUploads({ q, limit: 20 })).map(uploads.card); } catch { /* library optional */ }
|
||
try {
|
||
const out = await runYtdlpResilient([
|
||
`ytsearch${SEARCH_LIMIT}:${q}`,
|
||
'--dump-json', '--flat-playlist',
|
||
'--no-warnings', '--ignore-errors',
|
||
]);
|
||
const results = [...mine, ...parseCards(out)];
|
||
```
|
||
|
||
Server tests use `bun:test` (see `server/notes.test.js`); `server/package.json` "test" script
|
||
runs each file separately joined by `&&`.
|
||
|
||
## Steps
|
||
|
||
1. Create `server/innertube.js`:
|
||
```js
|
||
/* innertube.js — YouTube search via the InnerTube JSON API (no yt-dlp spawn).
|
||
* parseSearch() is pure (tested against fixtures/innertube-search.json);
|
||
* search() does the HTTP call. Callers MUST fall back to yt-dlp on any throw. */
|
||
const CLIENT = { clientName: 'WEB', clientVersion: '2.20250101.00.00', hl: 'en', gl: 'US' };
|
||
|
||
export function lengthToSeconds(s) {
|
||
if (typeof s !== 'string' || !/^\d+(:\d{1,2}){0,2}$/.test(s.trim())) return 0;
|
||
return s.trim().split(':').map(Number).reduce((acc, n) => acc * 60 + n, 0);
|
||
}
|
||
|
||
export function parseSearch(json) {
|
||
const sections = json?.contents?.twoColumnSearchResultsRenderer?.primaryContents
|
||
?.sectionListRenderer?.contents;
|
||
if (!Array.isArray(sections)) throw new Error('innertube: unexpected response shape');
|
||
const out = [];
|
||
for (const s of sections) {
|
||
for (const it of s?.itemSectionRenderer?.contents || []) {
|
||
const v = it && it.videoRenderer;
|
||
if (!v || typeof v.videoId !== 'string') continue;
|
||
const owner = v.ownerText?.runs?.[0] || {};
|
||
const be = owner.navigationEndpoint?.browseEndpoint || {};
|
||
const path = be.canonicalBaseUrl || (be.browseId ? `/channel/${be.browseId}` : '');
|
||
out.push({
|
||
id: v.videoId,
|
||
title: v.title?.runs?.map((r) => r.text).join('') || '(untitled)',
|
||
channel: owner.text || '',
|
||
channelId: be.browseId || '',
|
||
channelUrl: path ? `https://www.youtube.com${path}` : '',
|
||
duration: lengthToSeconds(v.lengthText?.simpleText),
|
||
thumbnail: `https://i.ytimg.com/vi/${v.videoId}/mqdefault.jpg`,
|
||
});
|
||
}
|
||
}
|
||
return out;
|
||
}
|
||
|
||
export async function search(q, { fetchImpl = fetch, timeoutMs = 6000 } = {}) {
|
||
const res = await fetchImpl('https://www.youtube.com/youtubei/v1/search?prettyPrint=false', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify({ context: { client: CLIENT }, query: q }),
|
||
signal: AbortSignal.timeout(timeoutMs),
|
||
});
|
||
if (!res.ok) throw new Error(`innertube: HTTP ${res.status}`);
|
||
return parseSearch(await res.json());
|
||
}
|
||
```
|
||
2. Create `server/innertube.test.js`:
|
||
```js
|
||
import { test, expect } from 'bun:test';
|
||
import { readFileSync } from 'node:fs';
|
||
import { parseSearch, lengthToSeconds, search } from './innertube.js';
|
||
const fixture = JSON.parse(readFileSync(new URL('./fixtures/innertube-search.json', import.meta.url)));
|
||
|
||
test('parses video renderers into slim cards', () => {
|
||
const r = parseSearch(fixture);
|
||
expect(r.length).toBe(4);
|
||
expect(r[0]).toEqual({
|
||
id: 'nQWFzMvCfLE', title: 'What A Beautiful Name - Hillsong Worship', channel: 'Hillsong Worship',
|
||
channelId: 'UC4q12NoPNySbVqwpw4iO5Vg', channelUrl: 'https://www.youtube.com/channel/UC4q12NoPNySbVqwpw4iO5Vg',
|
||
duration: 343, thumbnail: 'https://i.ytimg.com/vi/nQWFzMvCfLE/mqdefault.jpg',
|
||
});
|
||
expect(r[1].duration).toBe(7187);
|
||
expect(r[3].duration).toBe(0); // live, no lengthText
|
||
});
|
||
test('lengthToSeconds', () => {
|
||
expect(lengthToSeconds('5:43')).toBe(343);
|
||
expect(lengthToSeconds('1:59:47')).toBe(7187);
|
||
expect(lengthToSeconds('LIVE')).toBe(0);
|
||
expect(lengthToSeconds(undefined)).toBe(0);
|
||
});
|
||
test('unexpected shape throws (caller falls back to yt-dlp)', () => {
|
||
expect(() => parseSearch({})).toThrow();
|
||
});
|
||
test('search() throws on HTTP error', async () => {
|
||
const fetchImpl = async () => new Response('no', { status: 429 });
|
||
await expect(search('x', { fetchImpl })).rejects.toThrow('429');
|
||
});
|
||
```
|
||
If `r[1].duration` in the fixture differs from 7187, compute it from the fixture's
|
||
`lengthText` and use that value (the fixture is the source of truth).
|
||
3. `server/package.json` "test" script — append ` && bun test ./innertube.test.js`.
|
||
4. `server/server.js` — add import next to the other local imports:
|
||
`import * as innertube from './innertube.js';`
|
||
5. `server/server.js` `/api/search` — replace the `try { const out = await runYtdlpResilient([...]); const results = [...mine, ...parseCards(out)];`
|
||
head with:
|
||
```js
|
||
try {
|
||
let yt = [];
|
||
try {
|
||
yt = await innertube.search(q);
|
||
} catch (e) {
|
||
console.warn(`[search] innertube failed, using yt-dlp: ${e.message}`);
|
||
}
|
||
if (!yt.length) {
|
||
const out = await runYtdlpResilient([
|
||
`ytsearch${SEARCH_LIMIT}:${q}`,
|
||
'--dump-json', '--flat-playlist',
|
||
'--no-warnings', '--ignore-errors',
|
||
]);
|
||
yt = parseCards(out);
|
||
}
|
||
const results = [...mine, ...yt];
|
||
```
|
||
Everything after (`searchCache.set`, return, catch) stays as is.
|
||
6. Add an env kill-switch: at the top of the search block, `if (process.env.SEARCH_INNERTUBE === '0')`
|
||
skip the innertube call (leave `yt = []`). Document it in `docker-compose.yml` as a commented
|
||
line `# SEARCH_INNERTUBE: "0" # force yt-dlp search` next to the other commented env vars.
|
||
|
||
## Out of scope / do NOT touch
|
||
|
||
- `/api/channel`, `/api/playlist/expand` (still yt-dlp). No continuation paging.
|
||
- Card shape, `searchCache`, `app.js`.
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
cd /home/user/ytplayer/server && bun test ./innertube.test.js 2>&1 | tail -4
|
||
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail)"
|
||
[ -e public ] || ln -s ../frontend public; PORT=3996 bun server.js >/tmp/ytp006.log 2>&1 & SRV=$!; sleep 4
|
||
time curl -s 'http://localhost:3996/api/search?q=hillsong%20worship' | head -c 300; echo
|
||
kill $SRV; true
|
||
grep -c "innertube failed" /tmp/ytp006.log
|
||
```
|
||
|
||
Expected: innertube tests `4 pass 0 fail`; all files 0 fail; the search returns
|
||
`{"ok":true,"results":[{"id":…` well under 2 s when the network allows (if the container
|
||
has no internet, it falls back and the log grep prints ≥1 — report that, it is not a failure).
|
||
|
||
## Report format (executor: follow exactly)
|
||
|
||
Output ONLY the following, no other prose:
|
||
|
||
1. `git diff` (unified) of all changes.
|
||
2. Raw output of the Verification commands.
|
||
3. `Findings:` — max 10 lines.
|
||
|
||
Do not commit. Do not push. Do not touch files outside the Steps.
|
||
|
||
## Execution log
|
||
|
||
- Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 0.
|
||
- Orchestrator re-ran Verification: innertube tests 4 pass; `bun run test` all 6 server files 0 fail; `SERVER_OK`; live `/api/search` 1.05 s and 0.74 s with 0 InnerTube failures logged; new files byte-identical to the plan's code.
|
||
- Orchestrator note: the executor's pasted diff had fabricated blob hashes and a typo (`@hono/node-node-server`) that does NOT exist in the real `server/package.json`; the on-disk files were checked directly.
|
||
- Executor Findings (verbatim): Plan completed successfully. All steps implemented: innertube.js module created with parseSearch and search functions; innertube.test.js with 4 passing tests verifying fixture parsing, length conversion, error handling; server.js imports innertube and updated /api/search route to try InnerTube first (6s timeout) with fallback to yt-dlp when zero results or any error; kill-switch (SEARCH_INNERTUBE=0) added to server logic and documented in docker-compose.yml; all existing tests continue passing (62 total passes across all test files); search endpoint responds in 0.964s with correct results from InnerTube without fallback (0 failures logged).
|