Explain save failures and keep download diagnostics available

docs/youtube-save-unavailable.md

@@ -0,0 +1,109 @@
+# Rock Medley save investigation
+
+Video: `wZzRoXymOUU` — **Rock Medley**, Petra - Topic, 605 seconds.
+Investigation date: 2026-10-03. Main was pulled with `--ff-only` before changes.
+No production shell, credentials, cookies, vault, deployment or proxy was used.
+
+## Evidence
+
+YouTube's public oEmbed endpoint returns HTTP 200, the correct title/artist and
+thumbnail. This confirms the ID, but not playback permission. The watch page's
+initial player response on this development machine returns:
+
+```json
+{"status":"UNPLAYABLE","reason":"Video unavailable"}
+```
+
+The server's metadata invocation (`-J --no-warnings URL`) and the exact cache
+save invocation fail with `ERROR: [youtube] wZzRoXymOUU: Video unavailable`.
+Providing Bun as the JS runtime mirrors the Docker image's `/etc/yt-dlp.conf`.
+The error occurs during player extraction, before selecting/downloading a
+format. The server uses a ladder of selectors, not a hard-coded format ID.
+
+| Local extractor | Default result |
+| --- | --- |
+| PATH yt-dlp 2026.07.04 | Video unavailable |
+| Repository binary 2026.08.19 | Video unavailable |
+| Downloaded nightly 2026.09.27.232945 | Video unavailable |
+
+With 2026.08.19, explicit `web_embedded`, `web_safari`, `tv_simply`, `android_vr`,
+`mweb` and `web_music` clients all fail for this song too. A control video
+(`0gfX0dFLaBc`) succeeds from the same machine/runtime. This is video-specific
+upstream playability refusal, not a demonstrated stale-extractor, missing-format
+or broken-JS-runtime problem. It is **not exclusively reproducible on prod**.
+The generic response does not establish region, IP, account or licensing as the
+specific cause. Cookies or PO tokens were not requested or tested.
+
+Upstream release notes: [2026.08.19](https://github.com/yt-dlp/yt-dlp/releases/tag/2026.08.19).
+[yt-dlp documentation](https://github.com/yt-dlp/yt-dlp#dependencies) describes
+runtime dependencies; no dependency update resolved this local reproduction.
+The Dockerfile is unchanged because an update alone is not a proven remedy.
+Its existing release-download layer is cached by Docker, so a future intentional
+extractor update must rebuild that layer, rather than assume new app code updates
+it automatically.
+
+## Changes
+
+1. Shared extraction now tries a bounded client ladder for generic unavailable
+   and missing-format errors, as well as bot checks. This covers both metadata
+   probes and actual saves. Explicit private/member/age/country/removal errors,
+   local disk errors, cancellation and explicit client overrides do not trigger
+   that ladder. The final diagnostic preserves the original failure and lists
+   attempted clients; it does not mislabel the song as deleted.
+2. Explicit Retry POSTs `/api/download/:id/retry` before preparing a device save.
+   It clears stale preparation/stream hints and calls the normal cache with
+   `force:true` to bypass a **failed row's** cooldown. It does not delete ready
+   media or duplicate running jobs. USB-volume, free-space/budget, duration and
+   validation rules still belong to the existing cache. Repeated failed retries
+   are limited to one per song per 15 seconds. Normal automatic resumes do not
+   force a retry. Manual Retry follows the same new client ladder.
+3. Downloads, Settings → Downloads & storage, and save-failure toasts use plain
+   language. Expandable “Technical details” retains the raw tool error as text,
+   never HTML. Controls remain usable in both themes at 390/1440 px.
+
+## Remaining reviewer check
+
+**These changes do not demonstrate a successful download of Rock Medley.**
+The local default and all tested clients still receive YouTube's refusal.
+To establish the production-specific reason, the reviewer should compare:
+
+- Actual yt-dlp version and the invocation below inside the deployed container.
+- Logged-out YouTube playback on the homelab's outbound network versus the
+  working phone network; repeat with the phone on that same network if possible.
+- If only authenticated playback works, confirm whether the video needs account
+  access before considering the existing optional read-only cookies mechanism.
+  Do not assume a PO token is required without extractor diagnostics saying so.
+- If the failure tracks a network/region, use a permitted outbound network where
+  the video is available, or a permitted alternate source/upload/device copy.
+  Merely changing a region hint does not prove actual playback eligibility.
+
+No network, cookies, token, Docker or deployment workaround was guessed or
+silently enabled. After any confirmed environment correction, Retry now starts
+a genuine fresh attempt instead of just returning the old backoff failure.
+
+```sh
+# Metadata probe, same as the server (Docker config supplies the Bun runtime)
+yt-dlp -J --no-warnings 'https://www.youtube.com/watch?v=wZzRoXymOUU'
+
+# Cache save, same selector/merge/rate args as the server's defaults
+yt-dlp 'https://www.youtube.com/watch?v=wZzRoXymOUU' --no-warnings --no-playlist \
+  -f 'bv*[height<=720][vcodec^=avc1]+ba[ext=m4a]/bv*[height<=720][vcodec^=avc1]+ba/b[ext=mp4][vcodec^=avc1]/bv*[height<=720]+ba/b[ext=mp4]/b' \
+  --merge-output-format mp4 --limit-rate 2M -o /tmp/rock-medley-check.mp4
+```
+
+Tests: `node --test frontend/*.test.js`; app syntax check; unbundled server build;
+`bun test server/ytdlp-resilience.test.js server/download-retry.test.js
+server/media-cache.test.js`; and
+`npx playwright test --config playwright.download-retry.config.js`.
+The existing pool tests additionally need an importable yt-dlp zipapp:
+`YTDLP_PATH=/path/to/zipapp bun test server/ytdlp-pool.test.js`. The installed
+PATH executable here is a Python entry-point script, not an importable zipapp;
+providing the downloaded nightly zipapp makes all pool tests pass without any
+production code or environment changes.
+
+Final validation: **118 frontend unit tests, 44 server tests and 13 browser cases
+pass**, plus the app syntax check and server build. Browser cases cover the real
+Retry controls in both Downloads and Settings, normal resumes, friendly failures,
+44 px details controls, safe raw text and overflow at phone/desktop widths.
+A real iPhone Safari/PWA save should still be checked after deployment, especially
+pause/resume of a partial copy and retry after a network change.

frontend/app.js

@@ -1842,7 +1842,7 @@ async function preload(video, { quiet = false, mux = false, retry = false } = {}
       warmThumb(thumbUrlFor(id, video));
       if (!quiet) toast(`Saved “${video.title}” ✓`);
     } else if (!quiet && !(res && res.paused)) {
-      toast('⚠ ' + ((res && res.error) || 'Could not save video'));
+      toast('⚠ ' + (window.DownloadErrors ? DownloadErrors.message(res?.error) : res?.error || 'Could not save video'));
     }
   } catch (e) {
     if (e.name !== 'AbortError') SaveQueue.state(id, 'failed', e.message || 'Save failed');

frontend/download-errors.js

@@ -0,0 +1,28 @@
+/* Plain-language save failures, with the original diagnostic kept on demand. */
+(function(root) {
+  'use strict';
+  function message(error) {
+    const raw = String(error?.message || error || '').trim();
+    if (!raw) return 'Could not save this video. Try Retry.';
+    if (/not available in your country|geo.?restricted/i.test(raw)) return 'This video is unavailable in the server’s region. It may still play on your device.';
+    if (/private video|members.only|join this channel|confirm your age|age.restricted/i.test(raw)) return 'YouTube requires access this server does not have to save this video.';
+    if (/removed by|video has been removed|copyright claim/i.test(raw)) return 'YouTube is no longer making this video available to the server.';
+    if (/sign in to confirm|not a bot/i.test(raw)) return 'YouTube temporarily blocked the server’s download request. Try Retry later.';
+    if (/video unavailable/i.test(raw)) return 'YouTube isn’t letting the server save this video. Try Retry; it may still play on YouTube.';
+    if (/requested format (?:is )?not available|no video formats found/i.test(raw)) return 'No downloadable version was available. Try Retry to check other download methods.';
+    if (/^ERROR:|^WARNING:|Traceback|yt-dlp exited|yt-dlp not found/i.test(raw)) return 'The server could not save this video. Try Retry.';
+    // Existing plain-language connection/quota/validation messages remain useful.
+    return raw.length <= 180 && !raw.includes('\n') ? raw : 'Could not save this video. Try Retry.';
+  }
+  function appendDetails(container, error) {
+    if (!error) return;
+    const doc = container.ownerDocument;
+    const details = doc.createElement('details'); details.className = 'download-error-details';
+    const summary = doc.createElement('summary'); summary.textContent = 'Technical details';
+    const pre = doc.createElement('pre'); pre.textContent = String(error?.message || error);
+    details.append(summary, pre); container.append(details);
+    return details;
+  }
+  const api = { message, appendDetails };
+  if (typeof module !== 'undefined') module.exports = api; else root.DownloadErrors = api;
+})(typeof window !== 'undefined' ? window : globalThis);

frontend/download-errors.test.js

@@ -0,0 +1,24 @@
+const { test } = require('node:test');
+const assert = require('node:assert/strict');
+const { message } = require('./download-errors.js');
+const { formatJobStatus } = require('./downloads-page.js');
+test('unavailable songs get useful copy without claiming that they were removed', () => {
+  const raw = 'ERROR: [youtube] wZzRoXymOUU: Video unavailable\nYouTube clients tried: default, web_embedded, web_safari';
+  assert.equal(message(raw), 'YouTube isn’t letting the server save this video. Try Retry; it may still play on YouTube.');
+  assert.equal(formatJobStatus({ status: 'failed', error: raw }), 'Failed: ' + message(raw));
+});
+test('known restrictions and bot failures have distinct explanations', () => {
+  assert.match(message('Video unavailable. not available in your country'), /server’s region/);
+  assert.match(message('Video unavailable. Private video'), /requires access/);
+  assert.match(message('Video unavailable. Confirm your age'), /requires access/);
+  assert.match(message('Video has been removed'), /no longer/);
+  assert.match(message('Sign in to confirm you are not a bot'), /temporarily blocked/);
+  assert.match(message('Requested format is not available'), /other download methods/);
+});
+test('missing and raw tool failures get a short fallback; existing useful messages survive', () => {
+  assert.equal(message(null), 'Could not save this video. Try Retry.');
+  assert.equal(message('ERROR: mysterious extractor failure'), 'The server could not save this video. Try Retry.');
+  assert.equal(message('Connection lost'), 'Connection lost');
+  assert.equal(message('Please wait a few seconds before trying again.'), 'Please wait a few seconds before trying again.');
+  assert.equal(message('a'.repeat(200)), 'Could not save this video. Try Retry.');
+});

frontend/downloads-page.js

@@ -1,6 +1,7 @@
 /* Downloads page UI module. Grouped jobs, storage info, confirmed cancellation. */
 (function (root) {
   'use strict';
+  const errors = typeof module !== 'undefined' ? require('./download-errors.js') : root.DownloadErrors;
 
   const bytes = n => {
     if (typeof n !== 'number' || isNaN(n) || n <= 0) return '0 MB';
@@ -18,7 +19,7 @@
 
   function formatJobStatus(job) {
     if (job.status === 'failed') {
-      return job.error ? `Failed: ${job.error}` : 'Save failed';
+      return job.error ? `Failed: ${errors.message(job.error)}` : 'Save failed';
     }
     if (job.status === 'paused') {
       return 'Paused — resume when you’re ready';
@@ -215,6 +216,8 @@
           </div>
         `;
 
+        if (isFailed && job.error) errors.appendDetails(row.querySelector('.card-info'), job.error);
+
         const acts = doc.createElement('div');
         acts.className = 'dl-actions';
 

frontend/downloads.js

@@ -1,6 +1,7 @@
 /* Download controls and device storage accounting. */
 (function(root) {
   'use strict';
+  const errors = typeof module !== 'undefined' ? require('./download-errors.js') : root.DownloadErrors;
   const bytes = n => n >= 1073741824 ? `${(n / 1073741824).toFixed(1)} GB` : `${(n / 1048576).toFixed(1)} MB`;
   function breakdown(files, playlists) {
     const kinds = { video: 0, audio: 0, eq: 0 }, byId = new Map();
@@ -24,13 +25,15 @@
         for (const job of items) {
           const row = doc.createElement('div'); row.className = 'download-job';
           const title = doc.createElement('b'); title.textContent = job.title || job.id;
-          const status = doc.createElement('small'); status.textContent = job.error || job.status;
+          const status = doc.createElement('small'); status.textContent = job.error ? errors.message(job.error) : job.status;
           const progress = doc.createElement('progress'); progress.max = job.progress?.total || 1; if (job.progress?.total) progress.value = job.progress.received || 0; progress.setAttribute('aria-label', `Saving ${job.title || job.id}`);
           const actions = doc.createElement('div');
           for (const action of [...(['active', 'queued'].includes(job.status) ? ['pause'] : [job.status === 'failed' ? 'retry' : 'resume']), 'cancel']) {
             const button = doc.createElement('button'); button.type = 'button'; button.textContent = action[0].toUpperCase() + action.slice(1); button.setAttribute('aria-label', `${button.textContent} ${job.title || job.id}`); button.onclick = () => { adapter.action(job.id, action); signature = ''; paint(); }; actions.append(button);
           }
-          row.append(title, status, progress, actions); jobs.append(row);
+          row.append(title, status, progress, actions);
+          if (job.error) errors.appendDetails(row, job.error);
+          jobs.append(row);
         }
       }
       async function disk() {

frontend/index.html

@@ -660,6 +660,7 @@
     <script src="async-guard.js"></script>
     <script src="sw-update.js"></script>
     <script src="settings-sections.js"></script>
+    <script src="download-errors.js"></script>
     <script src="downloads.js"></script>
     <script src="downloads-page.js"></script>
     <script src="saved-page.js"></script>

frontend/offline-pages.css

@@ -21,3 +21,13 @@ html[data-layout] .dl-row .card-info { grid-column:2; grid-row:1; min-width:0; }
 .dl-row .dl-act.danger { color:var(--accent); background:transparent; }
 .dl-row .dl-act:disabled { opacity:.5; cursor:wait; }
 @media(min-width:900px) { html[data-layout] .card.dl-row { grid-template-columns:88px minmax(0,1fr); } html[data-layout] .dl-row .thumb { width:88px; } }
+
+/* Save failures remain readable; raw extractor output is opt-in, never HTML. */
+.dl-failed .card-channel { white-space:normal; line-height:1.5; }
+.download-error-details { min-width:0; color:var(--text-2); font:12px/1.5 var(--ui); }
+.download-error-details summary { min-height:44px; padding:12px 0; cursor:pointer; }
+.download-error-details summary:focus-visible { outline:2px solid var(--accent); outline-offset:2px; border-radius:4px; }
+.download-error-details pre { white-space:pre-wrap; overflow-wrap:anywhere; max-height:240px; overflow:auto; margin:0 0 8px; font:12px/1.5 var(--mono); }
+.download-job { grid-template-columns:minmax(0,1fr); }
+.download-job > div { flex-wrap:wrap; }
+.download-job progress { min-width:0; }

frontend/sw.js

@@ -84,6 +84,7 @@ const SHELL = [
   '/p2p-transfer.js',
   '/p2p-recv-worker.js',
   '/settings-sections.js',
+  '/download-errors.js',
   '/downloads.js',
   '/downloads-page.js',
   '/offline-pages.css',

tests/download-retry.spec.js

@@ -44,3 +44,29 @@ test('normal resumptions do not force the server to restart a cache job', async
   expect(retries).toBe(0);
   expect(await page.evaluate(id => cachedIds.has(id), id)).toBe(true);
 });
+
+for (const layout of ['classic', 'glass-stage']) for (const width of [390, 1440]) for (const screen of ['downloads', 'settings']) {
+  test(`${layout} ${width} ${screen}: friendly failure retains safe technical details`, async ({ page }) => {
+    await page.setViewportSize({ width, height: 844 });
+    await openClassic(page, { settings: { autoPreload: false } });
+    const raw = `ERROR: [youtube] ${id}: Video unavailable\nYouTube clients tried: default, web_embedded, web_safari\n<script>window.unsafeDetail=true</script>`;
+    await page.evaluate(({ layout, screen, id, raw }) => {
+      data.settings.layout = layout; applyAppearance();
+      SaveQueue.add({ id, title: 'Rock Medley', channel: 'Petra - Topic' });
+      SaveQueue.state(id, 'failed', raw);
+      view = { type: screen }; render();
+    }, { layout, screen, id, raw });
+    if (screen === 'settings') await page.locator('[data-section="downloads-storage"]').click();
+    const row = page.locator(screen === 'settings' ? '.download-job' : '.dl-failed').first();
+    await expect(row).toContainText('YouTube isn’t letting the server save this video.');
+    await expect(row.locator('pre')).toBeHidden();
+    const summary = row.locator('summary');
+    expect((await summary.boundingBox()).height).toBeGreaterThanOrEqual(44);
+    await summary.click();
+    await expect(row.locator('pre')).toHaveText(raw);
+    expect(await page.evaluate(() => window.unsafeDetail)).toBeUndefined();
+    const bounds = await row.evaluate(el => ({ width: el.clientWidth, content: el.scrollWidth, children: [...el.querySelectorAll('*')].filter(child => child.scrollWidth > child.clientWidth + 1).map(child => [child.tagName, child.className, child.clientWidth, child.scrollWidth]) }));
+    expect(bounds.content <= bounds.width + 1, JSON.stringify(bounds)).toBe(true);
+    await expect(row.getByRole('button', { name: 'Retry Rock Medley', exact: true })).toBeVisible();
+  });
+}