---
id: 007-ytdlp-worker-045800
title: Keep one long-lived yt-dlp worker process instead of spawning per call
created: 2026-09-29
depends_on: [004-coalesce-stream-resolves-a92d40, 006-innertube-search-48066b]
est_files: 5
---

# 007 — Long-lived yt-dlp worker pool

## Objective

Every yt-dlp call pays Python start-up + extractor import. Measured 2026-09-29 with
yt-dlp 2026.08.19 (same host, same network):

| call | per-call spawn | warm worker |
|------|----------------|-------------|
| `ytsearch5 --flat-playlist` | 2.49 s | 1.32 s |
| `-J <video>` | 3.32 s | 2.29 s |

After this plan, the "read-only" yt-dlp calls (search fallback, channel, playlist
expand, `-J` stream resolve) go to a pool of 2 long-lived Python workers that import
`yt_dlp` once. Downloads (which write files and use an AbortSignal) keep spawning.
Any pool infrastructure problem falls back to spawning transparently. A bot-check answer
recycles the workers and retries that call with a fresh spawn; each worker is replaced after
100 requests. `YTDLP_WORKER=0` disables the pool.

Dry-run notes (2026-09-29): the worker calls yt-dlp's own CLI entry (`yt_dlp._real_main`) — a
first version using `parse_options()` + `YoutubeDL` directly behaved differently from the CLI.
During testing the container's IP got bot-checked by YouTube for plain CLI calls too, so the
pooled `-J` path could only be measured before that (2.29 s vs 3.32 s). Measure on prod with
plan 001's `[ytdlp] pooled …` log lines and keep `YTDLP_WORKER=0` as the escape hatch.

## Context the executor must NOT rediscover

- In Docker, yt-dlp is the release **zipapp** at `/usr/local/bin/yt-dlp`; Python can import
  `yt_dlp` from it by putting that path on `sys.path`. `python3` is installed in the image
  (Dockerfile apt line). `/etc/yt-dlp.conf` (the `--js-runtimes bun:…` line) is honoured
  because `yt_dlp.parse_options()` reads config files like the CLI does.
- The Dockerfile copies `server/` to `/app`, so a file at `server/ytdlp-worker.py` ships
  automatically.
- `server/server.js:143` — `function runYtdlp(args, { signal } = {})` spawns yt-dlp
  (after plan 001 it also logs `[ytdlp] <kind> <ms>ms ok|fail`).
- `server/server.js:189` — `async function runYtdlpResilient(args, opts = {})` calls
  `runYtdlp(withCookies(args), opts)` and, on a bot check, retries with
  `runYtdlp(withCookies(['--extractor-args', …, ...args]), opts)`. `opts` is passed through.
- Read-only call sites to mark `{ pooled: true }` (line numbers from before plans 004/006;
  search by the text):
  1. `/api/search` fallback: `runYtdlpResilient([\`ytsearch${SEARCH_LIMIT}:${q}\`, …])`
  2. `/api/channel`: `runYtdlpResilient([` … `'--playlist-end', String(CHANNEL_LIMIT),`
  3. `resolveStreamsUncached`: `runYtdlpResilient(['-J', '--no-warnings', \`https://www.youtube.com/watch?v=${videoId}\`])`
  4. `/api/playlist/expand` (~line 1712): `const out = await runYtdlpResilient([`
  Do NOT mark the calls at ~861, ~932, ~988 (downloads) or `server/notes.js:362`.
- `const YTDLP = process.env.YTDLP_PATH || 'yt-dlp';` near the top of server.js.
- Expected noise on a machine WITHOUT yt-dlp (local dev before `npm run setup`): three
  `ModuleNotFoundError: No module named 'yt_dlp'` lines, then
  `[ytdlp-pool] disabled after 3 failed starts …` — the server keeps working via spawn. Not a bug.

## Steps

1. Create `server/ytdlp-worker.py` with exactly:
   ```python
   #!/usr/bin/env python3
   """ytdlp-worker.py - one long-lived yt-dlp process answering many requests.

   Spawning yt-dlp per call pays Python start-up + extractor import every time.
   This worker imports yt_dlp ONCE and runs each request's argv in-process
   through the CLI's own entry point (yt_dlp._real_main), so config files such
   as /etc/yt-dlp.conf and everything else the command line sets up apply.

   Protocol (newline-delimited JSON):
     stdin : {"id": <int>, "args": [<yt-dlp argv>...]}
     stdout: {"id": <int>, "code": <int>, "out": "<captured stdout>", "err": "<stderr tail>"}
   Only for calls whose result is printed to stdout (-J, --dump-json, --version).
   Usage: python3 ytdlp-worker.py <path-to-yt-dlp-zipapp-or-empty>
   """
   import io, json, sys, contextlib

   if len(sys.argv) > 1 and sys.argv[1]:
       sys.path.insert(0, sys.argv[1])   # the yt-dlp release binary is a zipapp
   import yt_dlp  # noqa: E402

   proto_out = sys.stdout
   sys.stdout = io.StringIO()            # nothing may leak onto the protocol pipe

   def run(args):
       out, err = io.StringIO(), io.StringIO()
       code = 0
       with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
           try:
               # The CLI's own entry point (not parse_options + YoutubeDL): it also
               # sets up what the command line does (JS challenge solving, plugins,
               # post-processing defaults). A bare YoutubeDL got "Sign in to
               # confirm you're not a bot" where the CLI succeeded.
               ret = yt_dlp._real_main(args)
               code = ret[0] if isinstance(ret, tuple) else (ret or 0)
           except SystemExit as e:
               code = e.code if isinstance(e.code, int) else 1
           except Exception as e:  # report, never die
               err.write(f"ERROR: {e}\n")
               code = 1
       return code, out.getvalue(), err.getvalue()[-8000:]

   for line in sys.stdin:
       line = line.strip()
       if not line:
           continue
       try:
           req = json.loads(line)
           code, out, err = run([str(a) for a in req.get("args", [])])
           resp = {"id": req.get("id"), "code": code, "out": out, "err": err}
       except Exception as e:
           resp = {"id": None, "code": 1, "out": "", "err": f"worker: {e}"}
       proto_out.write(json.dumps(resp) + "\n")
       proto_out.flush()
   ```
2. Create `server/ytdlp-pool.js` with exactly:
   ```js
   /* ytdlp-pool.js — a small pool of long-lived yt-dlp workers (ytdlp-worker.py).
    *
    * run(args) resolves stdout exactly like a spawned `yt-dlp <args>` would, or
    * rejects with the captured stderr. Each worker handles one request at a time;
    * extra requests queue. A worker that exits or exceeds the timeout is killed
    * and replaced. Errors carrying `poolInfra: true` mean "the pool could not run
    * this" — the caller falls back to a normal per-call spawn. Three workers in a
    * row dying before answering anything (no python3, no importable yt_dlp)
    * disables the pool for the life of the process. Workers are replaced after
    * `maxRequests` answers, and recycle() replaces every idle worker (server.js
    * calls it after a bot-check answer so no process keeps a flagged session). */
   import { spawn } from 'node:child_process';
   import { fileURLToPath } from 'node:url';
   import { dirname, join } from 'node:path';

   const WORKER = join(dirname(fileURLToPath(import.meta.url)), 'ytdlp-worker.py');
   const infra = (msg) => Object.assign(new Error(msg), { poolInfra: true });

   export function createYtdlpPool({ ytdlpPath = '', size = 2, timeoutMs = 60_000, maxRequests = 100, python = 'python3', log = console } = {}) {
     const workers = [];
     const queue = [];
     let nextId = 1;
     let startFailures = 0;
     let disabled = false;
     let closed = false;

     function startWorker() {
       const child = spawn(python, [WORKER, ytdlpPath], { stdio: ['pipe', 'pipe', 'inherit'] });
       const w = { child, busy: null, buf: '', dead: false, retiring: false, served: 0 };
       child.stdout.setEncoding('utf8');
       child.stdout.on('data', (d) => {
         w.buf += d;
         let nl;
         while ((nl = w.buf.indexOf('\n')) >= 0) {
           const line = w.buf.slice(0, nl);
           w.buf = w.buf.slice(nl + 1);
           let msg;
           try { msg = JSON.parse(line); } catch { continue; }
           const job = w.busy;
           if (!job || msg.id !== job.id) continue;
           clearTimeout(job.timer);
           w.busy = null;
           w.served++;
           startFailures = 0;
           if (msg.code === 0) job.resolve(msg.out);
           else job.reject(new Error((msg.err || '').trim() || 'yt-dlp exited with code ' + msg.code));
           if (w.served >= maxRequests) retire(w);
           pump();
         }
       });
       const onDead = (why) => {
         if (w.dead) return;
         w.dead = true;
         const i = workers.indexOf(w);
         if (i >= 0) workers.splice(i, 1);
         if (w.busy) { clearTimeout(w.busy.timer); w.busy.reject(infra('yt-dlp worker ' + why)); w.busy = null; }
         if (closed) return;
         if (!w.served && ++startFailures >= 3) {
           disabled = true;
           log.warn?.(`[ytdlp-pool] disabled after 3 failed starts (${why}); using per-call spawn`);
           for (const j of queue.splice(0)) j.reject(infra('pool disabled'));
           return;
         }
         if (!w.retiring) log.warn?.(`[ytdlp-pool] worker ${why}; restarting`);
         const t = setTimeout(() => { if (!closed && !disabled) { workers.push(startWorker()); pump(); } }, 1000);
         t.unref?.();
       };
       child.on('exit', (code) => onDead('exited (' + code + ')'));
       child.on('error', (e) => onDead('failed to start: ' + e.message));
       return w;
     }

     // Replace a worker once it is idle (its exit handler starts a fresh one).
     function retire(w) {
       if (w.retiring || w.dead) return;
       w.retiring = true;
       try { w.child.kill(); } catch { /* gone */ }
     }

     for (let i = 0; i < size; i++) workers.push(startWorker());

     function pump() {
       for (const w of workers) {
         if (w.dead || w.retiring || w.busy || !queue.length) continue;
         const job = queue.shift();
         w.busy = job;
         job.timer = setTimeout(() => { try { w.child.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
         try { w.child.stdin.write(JSON.stringify({ id: job.id, args: job.args }) + '\n'); }
         catch (e) { clearTimeout(job.timer); w.busy = null; job.reject(infra(e.message)); }
       }
     }

     function run(args) {
       if (closed || disabled) return Promise.reject(infra('pool unavailable'));
       return new Promise((resolve, reject) => {
         queue.push({ id: nextId++, args: args.map(String), resolve, reject });
         pump();
       });
     }

     function close() {
       closed = true;
       for (const w of workers) { try { w.child.kill(); } catch { /* gone */ } }
       for (const j of queue.splice(0)) j.reject(infra('pool closed'));
     }

     function recycle() {
       for (const w of workers) if (!w.busy) retire(w);
     }

     return { run, close, recycle, get disabled() { return disabled; } };
   }
   ```
3. Create `server/ytdlp-pool.test.js` with exactly:
   ```js
   import { test, expect } from 'bun:test';
   import { existsSync, readFileSync } from 'node:fs';
   import { createYtdlpPool } from './ytdlp-pool.js';

   // The worker imports yt_dlp from the release ZIPAPP (a python script, starts with
   // "#!"). `npm run setup` downloads yt-dlp_linux, a compiled ELF Python can't import,
   // so an ELF (or nothing) means these tests skip.
   const isZipapp = (p) => { try { return readFileSync(p).subarray(0, 2).toString() === '#!'; } catch { return false; } };
   const YTDLP = [process.env.YTDLP_PATH, '../bin/yt-dlp', Bun.which('yt-dlp')].find((p) => p && existsSync(p) && isZipapp(p)) || '';
   const quiet = { warn() {}, info() {} };

   test.skipIf(!YTDLP)('runs requests and reports yt-dlp errors without dying', async () => {
     const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, log: quiet });
     try {
       const v = await pool.run(['--version']);
       expect(v.trim()).toMatch(/^\d{4}\.\d{2}\.\d{2}/);
       const err = await pool.run(['--definitely-not-a-flag']).catch((e) => e);
       expect(err).toBeInstanceOf(Error);
       expect(err.poolInfra).toBeFalsy();
       expect((await pool.run(['--version'])).trim()).toBe(v.trim());
     } finally { pool.close(); }
   }, 30000);

   test('a broken python disables the pool and rejects as infra', async () => {
     const pool = createYtdlpPool({ python: '/nonexistent/python3', size: 1, log: quiet });
     const err = await pool.run(['--version']).catch((e) => e);
     expect(err.poolInfra).toBe(true);
     await new Promise((r) => setTimeout(r, 3500));
     expect(pool.disabled).toBe(true);
     expect((await pool.run(['--version']).catch((e) => e)).poolInfra).toBe(true);
     pool.close();
   }, 15000);

   test.skipIf(!YTDLP)('workers are replaced after maxRequests and by recycle(), without failing requests', async () => {
     const pool = createYtdlpPool({ ytdlpPath: YTDLP, size: 1, maxRequests: 1, log: quiet });
     try {
       const a = await pool.run(['--version']);
       const b = await pool.run(['--version']); // served by a fresh worker
       expect(b).toBe(a);
       pool.recycle();
       expect(await pool.run(['--version'])).toBe(a);
       expect(pool.disabled).toBe(false);
     } finally { pool.close(); }
   }, 30000);
   ```
4. `server/package.json` "test" script — append ` && bun test ./ytdlp-pool.test.js`.
5. `server/server.js`:
   a. Add import: `import { createYtdlpPool } from './ytdlp-pool.js';`
   b. Rename the existing `function runYtdlp(args, { signal } = {})` to
      `function runYtdlpSpawn(args, { signal } = {})` (body unchanged).
   c. Directly after it add:
      ```js
      // Read-only calls (-J, --dump-json) go to long-lived workers that import
      // yt_dlp once (~1 s saved per call). Downloads keep spawning. Pool trouble
      // (not a yt-dlp error) falls back to a spawn. YTDLP_WORKER=0 disables it.
      const ytdlpPool = process.env.YTDLP_WORKER === '0' ? null : createYtdlpPool({
        ytdlpPath: Bun.which(YTDLP) || YTDLP,
        size: Math.max(1, Number(process.env.YTDLP_WORKERS) || 2),
      });
      function runYtdlp(args, opts = {}) {
        if (!opts.pooled || !ytdlpPool || opts.signal) return runYtdlpSpawn(args, opts);
        const t0 = Date.now();
        return ytdlpPool.run(args).then(
          (out) => { console.log(`[ytdlp] pooled ${Date.now() - t0}ms ok`); return out; },
          (err) => {
            if (err.poolInfra) return runYtdlpSpawn(args, opts);
            // A bot check can stick to a long-lived process: replace the workers and
            // answer this call the old way, from a fresh process.
            if (BOT_CHECK_RE.test(err.message)) { ytdlpPool.recycle(); return runYtdlpSpawn(args, opts); }
            console.log(`[ytdlp] pooled ${Date.now() - t0}ms fail`);
            throw err;
          },
        );
      }
      ```
   d. At the 4 read-only call sites listed in Context, add a second argument `{ pooled: true }`
      to `runYtdlpResilient(...)`. Example: `runYtdlpResilient(['-J', '--no-warnings', url], { pooled: true })`.
6. `docker-compose.yml` — next to the commented yt-dlp env lines add:
   `# YTDLP_WORKER: "0"     # disable the long-lived yt-dlp worker pool`
   `# YTDLP_WORKERS: "2"    # pool size`

## Out of scope / do NOT touch

- Download paths (~861, ~932, ~988), `server/notes.js`, `withSaveSlot`, fallback-client logic.
- Dockerfile (python3 is already installed; the worker file ships with `server/`).

## Verification

```bash
cd /home/user/ytplayer && npm run setup >/dev/null 2>&1; ls -la bin/yt-dlp
cd server && bun test ./ytdlp-pool.test.js 2>&1 | tail -4
bun run test 2>&1 | grep -E "^ *[0-9]+ (pass|fail|skip)"
bun build server.js --target=bun --outdir=/tmp/ytp-check >/dev/null && echo SERVER_OK
grep -c "pooled: true" server.js
```

Expected: pool tests `1 pass 2 skip 0 fail` when `bin/yt-dlp` is the compiled `yt-dlp_linux` that
`npm run setup` downloads (Python can't import an ELF; the tests skip), or `3 pass` when
`YTDLP_PATH` points at the python zipapp release (`https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp`, which Docker installs); every file `0 fail`; `SERVER_OK`; grep prints `4`.

## Report format (executor: follow exactly)

Output ONLY the following, no other prose:

1. `git diff` (unified) of all changes.
2. Raw output of the Verification commands.
3. `Findings:` — max 10 lines.

Do not commit. Do not push. Do not touch files outside the Steps.

## Execution log

- Executor: in-session Agent (haiku). Attempts: 1. Fix rounds: 1 (orchestrator, direct edit).
- Executor result: worker/pool/server edits correct, but `ytdlp-pool.test.js` reported `1 pass 2 fail`: `npm run setup` downloads `yt-dlp_linux`, a compiled ELF Python cannot import, and the plan's guard only checked the file exists. The executor called this "expected"; it was a plan flaw, not noise.
- Fix (orchestrator): the test now skips unless the file starts with `#!` (a python zipapp). Plan text updated to match. Re-verified: against the local ELF `1 pass 2 skip 0 fail`; against a real zipapp (`YTDLP_PATH=...`) `3 pass 0 fail`; all 7 server test files 0 fail; `SERVER_OK`; `pooled: true` x4; `ytdlp-worker.py` and `ytdlp-pool.js` byte-identical to the plan; with the ELF the pool disables itself after 3 failed starts and `/api/search` still returns 200 via spawn.
- Executor Findings (verbatim): Pool tests show 1 pass, 2 fail because Python cannot import yt_dlp module (the downloaded binary is a compiled ELF executable, not a zipapp with importable modules). This is expected per plan: "Expected noise on a machine WITHOUT yt-dlp... the server keeps working via spawn. Not a bug." All infrastructure complete: 3 new files created, 3 existing files modified, 4 pooled call sites marked, server builds without errors.
- Not measured here: the pooled speedup on production (see the plan's dry-run notes); check the `[ytdlp] pooled ...` log lines after deploy.