josh/ytplayer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep one long-lived yt-dlp worker process instead of spawning per call

Browse Source
This commit is contained in:
Claude
2026-09-30 07:04:53 +00:00
parent b8658f3a9d
commit 9cac21afce
8 changed files with 259 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

350
plans/done/007-ytdlp-worker-045800.md Normal file
View File

@@ -0,0 +1,350 @@
---
id: 007-ytdlp-worker-045800
title: Keep one long-lived yt-dlp worker process instead of spawning per call
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b]
est_files: 5
---
# 007 — Long-lived yt-dlp worker pool
## Objective
Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with
yt-dlp 2026.08.19 (same host, same network):
| call | per-call spawn | warm worker |
|------|----------------|-------------|
| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s |
| `-J <video>` | 3.32 s | 2.29 s |
After this plan, the "read-only" yt-dlp calls (search fallback, channel, playlist
expand, `-J` stream resolve) go to a pool of 2 long-lived Python workers that import
`yt_dlp` once. Downloads (which write files and use an AbortSignal) keep spawning.
Any pool infrastructure problem falls back to spawning transparently. A bot-check answer
recycles the workers and retries that call with a fresh spawn; each worker is replaced after
100 requests. `YTDLP_WORKER=0` disables the pool.
Dry-run notes (2026-09-29): the worker calls yt-dlp's own CLI entry (`yt_dlp._real_main`) — a
first version using `parse_options()` + `YoutubeDL` directly behaved differently from the CLI.
During testing the container's IP got bot-checked by YouTube for plain CLI calls too, so the
pooled `-J` path could only be measured before that (2.29 s vs 3.32 s). Measure on prod with
plan 001's `[ytdlp] pooled …` log lines and keep `YTDLP_WORKER=0` as the escape hatch.
## Context the executor must NOT rediscover
- In Docker, yt-dlp is the release **zipapp** at `/usr/local/bin/yt-dlp`; Python can import
`yt_dlp` from it by putting that path on `sys.path`. `python3` is installed in the image
(Dockerfile apt line). `/etc/yt-dlp.conf` (the `--js-runtimes bun:…` line) is honoured
because `yt_dlp.parse_options()` reads config files like the CLI does.
- The Dockerfile copies `server/` to `/app`, so a file at `server/ytdlp-worker.py` ships
automatically.
- `server/server.js:143` — `function runYtdlp(args, { signal } = {})` spawns yt-dlp
(after plan 001 it also logs `[ytdlp] <kind> <ms>ms ok|fail`).
- `server/server.js:189` — `async function runYtdlpResilient(args, opts = {})` calls
`runYtdlp(withCookies(args), opts)` and, on a bot check, retries with
`runYtdlp(withCookies(['--extractor-args', …, ...args]), opts)`. `opts` is passed through.
- Read-only call sites to mark `{ pooled: true }` (line numbers from before plans 004/006;
search by the text):
1. `/api/search` fallback: `runYtdlpResilient([\`ytsearch${SEARCH_LIMIT}:${q}\`, …])`
2. `/api/channel`: `runYtdlpResilient([` … `'--playlist-end', String(CHANNEL_LIMIT),`
3. `resolveStreamsUncached`: `runYtdlpResilient(['-J', '--no-warnings', \`https://www.youtube.com/watch?v=${videoId}\`])`
4. `/api/playlist/expand` (~line 1712): `const out = await runYtdlpResilient([`
Do NOT mark the calls at ~861, ~932, ~988 (downloads) or `server/notes.js:362`.
- `const YTDLP = process.env.YTDLP_PATH || 'yt-dlp';` near the top of server.js.
- Expected noise on a machine WITHOUT yt-dlp (local dev before `npm run setup`): three
`ModuleNotFoundError: No module named 'yt_dlp'` lines, then
`[ytdlp-pool] disabled after 3 failed starts …` — the server keeps working via spawn. Not a bug.
## Steps
1. Create `server/ytdlp-worker.py` with exactly:
```python
#!/usr/bin/env python3
"""ytdlp-worker.py - one long-lived yt-dlp process answering many requests.
Spawning yt-dlp per call pays Python start-up + extractor import every time.
This worker imports yt_dlp ONCE and runs each request's argv in-process
through the CLI's own entry point (yt_dlp._real_main), so config files such
as /etc/yt-dlp.conf and everything else the command line sets up apply.
Protocol (newline-delimited JSON):
stdin : {"id": <int>, "args": [<yt-dlp argv>...]}
stdout: {"id": <int>, "code": <int>, "out": "<captured stdout>", "err": "<stderr tail>"}
Only for calls whose result is printed to stdout (-J, --dump-json, --version).
Usage: python3 ytdlp-worker.py <path-to-yt-dlp-zipapp-or-empty>
"""
import io, json, sys, contextlib
if len(sys.argv) > 1 and sys.argv[1]:
sys.path.insert(0, sys.argv[1]) # the yt-dlp release binary is a zipapp
import yt_dlp # noqa: E402
proto_out = sys.stdout
sys.stdout = io.StringIO() # nothing may leak onto the protocol pipe
def run(args):
out, err = io.StringIO(), io.StringIO()
code = 0
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
try:
# The CLI's own entry point (not parse_options + YoutubeDL): it also
# sets up what the command line does (JS challenge solving, plugins,
# post-processing defaults). A bare YoutubeDL got "Sign in to
# confirm you're not a bot" where the CLI succeeded.
ret = yt_dlp._real_main(args)
code = ret[0] if isinstance(ret, tuple) else (ret or 0)
except SystemExit as e:
code = e.code if isinstance(e.code, int) else 1
except Exception as e: # report, never die
err.write(f"ERROR: {e}\n")
code = 1
return code, out.getvalue(), err.getvalue()[-8000:]
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
req = json.loads(line)
code, out, err = run([str(a) for a in req.get("args", [])])
resp = {"id": req.get("id"), "code": code, "out": out, "err": err}
except Exception as e:
resp = {"id": None, "code": 1, "out": "", "err": f"worker: {e}"}
proto_out.write(json.dumps(resp) + "\n")
proto_out.flush()
```
2. Create `server/ytdlp-pool.js` with exactly:
```js
/* ytdlp-pool.js — a small pool of long-lived yt-dlp workers (ytdlp-worker.py).
*
* run(args) resolves stdout exactly like a spawned `yt-dlp <args>` would, or
* rejects with the captured stderr. Each worker handles one request at a time;
* extra requests queue. A worker that exits or exceeds the timeout is killed
* and replaced. Errors carrying `poolInfra: true` mean "the pool could not run
* this" — the caller falls back to a normal per-call spawn. Three workers in a
* row dying before answering anything (no python3, no importable yt_dlp)
* disables the pool for the life of the process. Workers are replaced after
* `maxRequests` answers, and recycle() replaces every idle worker (server.js
* calls it after a bot-check answer so no process keeps a flagged session). */
import { spawn } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const WORKER = join(dirname(fileURLToPath(import.meta.url)), 'ytdlp-worker.py');
const infra = (msg) => Object.assign(new Error(msg), { poolInfra: true });
export function createYtdlpPool({ ytdlpPath = '', size = 2, timeoutMs = 60_000, maxRequests = 100, python = 'python3', log = console } = {}) {
const workers = [];
const queue = [];
let nextId = 1;
let startFailures = 0;
let disabled = false;
let closed = false;
function startWorker() {
const child = spawn(python, [WORKER, ytdlpPath], { stdio: ['pipe', 'pipe', 'inherit'] });
const w = { child, busy: null, buf: '', dead: false, retiring: false, served: 0 };
child.stdout.setEncoding('utf8');
child.stdout.on('data', (d) => {
w.buf += d;
let nl;
while ((nl = w.buf.indexOf('\n')) >= 0) {
const line = w.buf.slice(0, nl);
w.buf = w.buf.slice(nl + 1);
let msg;
try { msg = JSON.parse(line); } catch { continue; }
const job = w.busy;
if (!job || msg.id !== job.id) continue;
clearTimeout(job.timer);
w.busy = null;
w.served++;
startFailures = 0;
if (msg.code === 0) job.resolve(msg.out);
else job.reject(new Error((msg.err || '').trim() || 'yt-dlp exited with code ' + msg.code));
if (w.served >= maxRequests) retire(w);
pump();
}
});
const onDead = (why) => {
if (w.dead) return;
w.dead = true;
const i = workers.indexOf(w);
if (i >= 0) workers.splice(i, 1);
if (w.busy) { clearTimeout(w.busy.timer); w.busy.reject(infra('yt-dlp worker ' + why)); w.busy = null; }
if (closed) return;
if (!w.served && ++startFailures >= 3) {
disabled = true;
log.warn?.(`[ytdlp-pool] disabled after 3 failed starts (${why}); using per-call spawn`);
for (const j of queue.splice(0)) j.reject(infra('pool disabled'));
return;
}
if (!w.retiring) log.warn?.(`[ytdlp-pool] worker ${why}; restarting`);
const t = setTimeout(() => { if (!closed && !disabled) { workers.push(startWorker()); pump(); } }, 1000);
t.unref?.();
};
child.on('exit', (code) => onDead('exited (' + code + ')'));
child.on('error', (e) => onDead('failed to start: ' + e.message));
return w;
}
// Replace a worker once it is idle (its exit handler starts a fresh one).
function retire(w) {
if (w.retiring || w.dead) return;
w.retiring = true;
try { w.child.kill(); } catch { /* gone */ }
}
for (let i = 0; i < size; i++) workers.push(startWorker());
function pump() {
for (const w of workers) {
if (w.dead || w.retiring || w.busy || !queue.length) continue;
const job = queue.shift();
w.busy = job;
job.timer = setTimeout(() => { try { w.child.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
try { w.child.stdin.write(JSON.stringify({ id: job.id, args: job.args }) + '\n'); }
catch (e) { clearTimeout(job.timer); w.busy = null; job.reject(infra(e.message)); }
}
}
function run(args) {
if (closed || disabled) return Promise.reject(infra('pool unavailable'));
return new Promise((resolve, reject) => {
queue.push({ id: nextId++, args: args.map(String), resolve, reject });
pump();
});
}
function close() {
closed = true;
for (const w of workers) { try { w.child.kill(); } catch { /* gone */ } }
for (const j of queue.splice(0)) j.reject(infra('pool closed'));
}
function recycle() {
for (const w of workers) if (!w.busy) retire(w);
}
return { run, close, recycle, get disabled() { return disabled; } };
}
```
3. Create `server/ytdlp-pool.test.js` with exactly:
```js
import { test, expect } from 'bun:test';
import { existsSync, readFileSync } from 'node:fs';
import { createYtdlpPool } from './ytdlp-pool.js';
// The worker imports yt_dlp from the release ZIPAPP (a python script, starts with
// "#!"). `npm run setup` downloads yt-dlp_linux, a compiled ELF Python can't import,
// so an ELF (or nothing) means these tests skip.
const isZipapp = (p) => { try { return readFileSync(p).subarray(0, 2).toString() === '#!'; } catch { return false; } };
const YTDLP = [process.env.YTDLP_PATH, '../bin/yt-dlp', Bun.which('yt-dlp')].find((p) => p && existsSync(p) && isZipapp(p)) || '';
const quiet = { warn() {}, info() {} };
test.skipIf(!YTDLP)('runs requests and reports yt-dlp errors without dying', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, log: quiet });
try {
const v = await pool.run(['--version']);
expect(v.trim()).toMatch(/^\d{4}\.\d{2}\.\d{2}/);
const err = await pool.run(['--definitely-not-a-flag']).catch((e) => e);
expect(err).toBeInstanceOf(Error);
expect(err.poolInfra).toBeFalsy();
expect((await pool.run(['--version'])).trim()).toBe(v.trim());
} finally { pool.close(); }
}, 30000);
test('a broken python disables the pool and rejects as infra', async () => {
const pool = createYtdlpPool({ python: '/nonexistent/python3', size: 1, log: quiet });
const err = await pool.run(['--version']).catch((e) => e);
expect(err.poolInfra).toBe(true);
await new Promise((r) => setTimeout(r, 3500));
expect(pool.disabled).toBe(true);
expect((await pool.run(['--version']).catch((e) => e)).poolInfra).toBe(true);
pool.close();
}, 15000);
test.skipIf(!YTDLP)('workers are replaced after maxRequests and by recycle(), without failing requests', async () => {
const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, maxRequests: 1, log: quiet });
try {
const a = await pool.run(['--version']);
const b = await pool.run(['--version']); // served by a fresh worker
expect(b).toBe(a);
pool.recycle();
expect(await pool.run(['--version'])).toBe(a);
expect(pool.disabled).toBe(false);
} finally { pool.close(); }
}, 30000);
```
4. `server/package.json` "test" script — append ` && bun test ./ytdlp-pool.test.js`.
5. `server/server.js`:
a. Add import: `import { createYtdlpPool } from './ytdlp-pool.js';`
b. Rename the existing `function runYtdlp(args, { signal } = {})` to
`function runYtdlpSpawn(args, { signal } = {})` (body unchanged).
c. Directly after it add:
```js
// Read-only calls (-J, --dump-json) go to long-lived workers that import
// yt_dlp once (~1 s saved per call). Downloads keep spawning. Pool trouble
// (not a yt-dlp error) falls back to a spawn. YTDLP_WORKER=0 disables it.
const ytdlpPool = process.env.YTDLP_WORKER === '0' ? null : createYtdlpPool({
ytdlpPath: Bun.which(YTDLP) || YTDLP,
size: Math.max(1, Number(process.env.YTDLP_WORKERS) || 2),
});
function runYtdlp(args, opts = {}) {
if (!opts.pooled || !ytdlpPool || opts.signal) return runYtdlpSpawn(args, opts);
const t0 = Date.now();
return ytdlpPool.run(args).then(
(out) => { console.log(`[ytdlp] pooled ${Date.now() - t0}ms ok`); return out; },
(err) => {
if (err.poolInfra) return runYtdlpSpawn(args, opts);
// A bot check can stick to a long-lived process: replace the workers and
// answer this call the old way, from a fresh process.
if (BOT_CHECK_RE.test(err.message)) { ytdlpPool.recycle(); return runYtdlpSpawn(args, opts); }
console.log(`[ytdlp] pooled ${Date.now() - t0}ms fail`);
throw err;
},
);
}
```
d. At the 4 read-only call sites listed in Context, add a second argument `{ pooled: true }`
to `runYtdlpResilient(...)`. Example: `runYtdlpResilient(['-J', '--no-warnings', url], { pooled: true })`.
6. `docker-compose.yml` — next to the commented yt-dlp env lines add:
`# YTDLP_WORKER: "0" # disable the long-lived yt-dlp worker pool`
`# YTDLP_WORKERS: "2" # pool size`
## Out of scope / do NOT touch
- Download paths (~861, ~932, ~988), `server/notes.js`, `withSaveSlot`, fallback-client logic.
- Dockerfile (python3 is already installed; the worker file ships with `server/`).
## Verification
```bash
cd /home/user/ytplayer && npm run setup >/dev/null 2>&1; ls -la bin/yt-dlp
cd server && bun test ./ytdlp-pool.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail|skip)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "pooled: true" server.js
```
Expected: pool tests `1 pass 2 skip 0 fail` when `bin/yt-dlp` is the compiled `yt-dlp_linux` that
`npm run setup` downloads (Python can't import an ELF; the tests skip), or `3 pass` when
`YTDLP_PATH` points at the python zipapp release (`https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp`, which Docker installs); every file `0 fail`; `SERVER_OK`; grep prints `4`.
## Report format (executor: follow exactly)
Output ONLY the following, no other prose:
1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.
Do not commit. Do not push. Do not touch files outside the Steps.
## Execution log
- Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 1 (orchestrator, direct edit).
- Executor result: worker/pool/server edits correct, but `ytdlp-pool.test.js` reported `1 pass 2 fail`: `npm run setup` downloads `yt-dlp_linux`, a compiled ELF Python cannot import, and the plan's guard only checked the file exists. The executor called this "expected"; it was a plan flaw, not noise.
- Fix (orchestrator): the test now skips unless the file starts with `#!` (a python zipapp). Plan text updated to match. Re-verified: against the local ELF `1 pass 2 skip 0 fail`; against a real zipapp (`YTDLP_PATH=...`) `3 pass 0 fail`; all 7 server test files 0 fail; `SERVER_OK`; `pooled: true` x4; `ytdlp-worker.py` and `ytdlp-pool.js` byte-identical to the plan; with the ELF the pool disables itself after 3 failed starts and `/api/search` still returns 200 via spawn.
- Executor Findings (verbatim): Pool tests show 1 pass, 2 fail because Python cannot import yt_dlp module (the downloaded binary is a compiled ELF executable, not a zipapp with importable modules). This is expected per plan: "Expected noise on a machine WITHOUT yt-dlp... the server keeps working via spawn. Not a bug." All infrastructure complete: 3 new files created, 3 existing files modified, 4 pooled call sites marked, server builds without errors.
- Not measured here: the pooled speedup on production (see the plan's dry-run notes); check the `[ytdlp] pooled ...` log lines after deploy.