diff --git a/docs/youtube-save-unavailable.md b/docs/youtube-save-unavailable.md new file mode 100644 index 0000000..d46f389 --- /dev/null +++ b/docs/youtube-save-unavailable.md @@ -0,0 +1,109 @@ +# Rock Medley save investigation + +Video: `wZzRoXymOUU` — **Rock Medley**, Petra - Topic, 605 seconds. +Investigation date: 2026-10-03. Main was pulled with `--ff-only` before changes. +No production shell, credentials, cookies, vault, deployment or proxy was used. + +## Evidence + +YouTube's public oEmbed endpoint returns HTTP 200, the correct title/artist and +thumbnail. This confirms the ID, but not playback permission. The watch page's +initial player response on this development machine returns: + +```json +{"status":"UNPLAYABLE","reason":"Video unavailable"} +``` + +The server's metadata invocation (`-J --no-warnings URL`) and the exact cache +save invocation fail with `ERROR: [youtube] wZzRoXymOUU: Video unavailable`. +Providing Bun as the JS runtime mirrors the Docker image's `/etc/yt-dlp.conf`. +The error occurs during player extraction, before selecting/downloading a +format. The server uses a ladder of selectors, not a hard-coded format ID. + +| Local extractor | Default result | +| --- | --- | +| PATH yt-dlp 2026.07.04 | Video unavailable | +| Repository binary 2026.08.19 | Video unavailable | +| Downloaded nightly 2026.09.27.232945 | Video unavailable | + +With 2026.08.19, explicit `web_embedded`, `web_safari`, `tv_simply`, `android_vr`, +`mweb` and `web_music` clients all fail for this song too. A control video +(`0gfX0dFLaBc`) succeeds from the same machine/runtime. This is video-specific +upstream playability refusal, not a demonstrated stale-extractor, missing-format +or broken-JS-runtime problem. It is **not exclusively reproducible on prod**. +The generic response does not establish region, IP, account or licensing as the +specific cause. Cookies or PO tokens were not requested or tested. + +Upstream release notes: [2026.08.19](https://github.com/yt-dlp/yt-dlp/releases/tag/2026.08.19). +[yt-dlp documentation](https://github.com/yt-dlp/yt-dlp#dependencies) describes +runtime dependencies; no dependency update resolved this local reproduction. +The Dockerfile is unchanged because an update alone is not a proven remedy. +Its existing release-download layer is cached by Docker, so a future intentional +extractor update must rebuild that layer, rather than assume new app code updates +it automatically. + +## Changes + +1. Shared extraction now tries a bounded client ladder for generic unavailable + and missing-format errors, as well as bot checks. This covers both metadata + probes and actual saves. Explicit private/member/age/country/removal errors, + local disk errors, cancellation and explicit client overrides do not trigger + that ladder. The final diagnostic preserves the original failure and lists + attempted clients; it does not mislabel the song as deleted. +2. Explicit Retry POSTs `/api/download/:id/retry` before preparing a device save. + It clears stale preparation/stream hints and calls the normal cache with + `force:true` to bypass a **failed row's** cooldown. It does not delete ready + media or duplicate running jobs. USB-volume, free-space/budget, duration and + validation rules still belong to the existing cache. Repeated failed retries + are limited to one per song per 15 seconds. Normal automatic resumes do not + force a retry. Manual Retry follows the same new client ladder. +3. Downloads, Settings → Downloads & storage, and save-failure toasts use plain + language. Expandable “Technical details” retains the raw tool error as text, + never HTML. Controls remain usable in both themes at 390/1440 px. + +## Remaining reviewer check + +**These changes do not demonstrate a successful download of Rock Medley.** +The local default and all tested clients still receive YouTube's refusal. +To establish the production-specific reason, the reviewer should compare: + +- Actual yt-dlp version and the invocation below inside the deployed container. +- Logged-out YouTube playback on the homelab's outbound network versus the + working phone network; repeat with the phone on that same network if possible. +- If only authenticated playback works, confirm whether the video needs account + access before considering the existing optional read-only cookies mechanism. + Do not assume a PO token is required without extractor diagnostics saying so. +- If the failure tracks a network/region, use a permitted outbound network where + the video is available, or a permitted alternate source/upload/device copy. + Merely changing a region hint does not prove actual playback eligibility. + +No network, cookies, token, Docker or deployment workaround was guessed or +silently enabled. After any confirmed environment correction, Retry now starts +a genuine fresh attempt instead of just returning the old backoff failure. + +```sh +# Metadata probe, same as the server (Docker config supplies the Bun runtime) +yt-dlp -J --no-warnings 'https://www.youtube.com/watch?v=wZzRoXymOUU' + +# Cache save, same selector/merge/rate args as the server's defaults +yt-dlp 'https://www.youtube.com/watch?v=wZzRoXymOUU' --no-warnings --no-playlist \ + -f 'bv*[height<=720][vcodec^=avc1]+ba[ext=m4a]/bv*[height<=720][vcodec^=avc1]+ba/b[ext=mp4][vcodec^=avc1]/bv*[height<=720]+ba/b[ext=mp4]/b' \ + --merge-output-format mp4 --limit-rate 2M -o /tmp/rock-medley-check.mp4 +``` + +Tests: `node --test frontend/*.test.js`; app syntax check; unbundled server build; +`bun test server/ytdlp-resilience.test.js server/download-retry.test.js +server/media-cache.test.js`; and +`npx playwright test --config playwright.download-retry.config.js`. +The existing pool tests additionally need an importable yt-dlp zipapp: +`YTDLP_PATH=/path/to/zipapp bun test server/ytdlp-pool.test.js`. The installed +PATH executable here is a Python entry-point script, not an importable zipapp; +providing the downloaded nightly zipapp makes all pool tests pass without any +production code or environment changes. + +Final validation: **118 frontend unit tests, 44 server tests and 13 browser cases +pass**, plus the app syntax check and server build. Browser cases cover the real +Retry controls in both Downloads and Settings, normal resumes, friendly failures, +44 px details controls, safe raw text and overflow at phone/desktop widths. +A real iPhone Safari/PWA save should still be checked after deployment, especially +pause/resume of a partial copy and retry after a network change. diff --git a/frontend/app.js b/frontend/app.js index 96b8edf..6bf57ca 100755 --- a/frontend/app.js +++ b/frontend/app.js @@ -1842,7 +1842,7 @@ async function preload(video, { quiet = false, mux = false, retry = false } = {} warmThumb(thumbUrlFor(id, video)); if (!quiet) toast(`Saved “${video.title}” ✓`); } else if (!quiet && !(res && res.paused)) { - toast('⚠ ' + ((res && res.error) || 'Could not save video')); + toast('⚠ ' + (window.DownloadErrors ? DownloadErrors.message(res?.error) : res?.error || 'Could not save video')); } } catch (e) { if (e.name !== 'AbortError') SaveQueue.state(id, 'failed', e.message || 'Save failed'); diff --git a/frontend/download-errors.js b/frontend/download-errors.js new file mode 100644 index 0000000..6022446 --- /dev/null +++ b/frontend/download-errors.js @@ -0,0 +1,28 @@ +/* Plain-language save failures, with the original diagnostic kept on demand. */ +(function(root) { + 'use strict'; + function message(error) { + const raw = String(error?.message || error || '').trim(); + if (!raw) return 'Could not save this video. Try Retry.'; + if (/not available in your country|geo.?restricted/i.test(raw)) return 'This video is unavailable in the server’s region. It may still play on your device.'; + if (/private video|members.only|join this channel|confirm your age|age.restricted/i.test(raw)) return 'YouTube requires access this server does not have to save this video.'; + if (/removed by|video has been removed|copyright claim/i.test(raw)) return 'YouTube is no longer making this video available to the server.'; + if (/sign in to confirm|not a bot/i.test(raw)) return 'YouTube temporarily blocked the server’s download request. Try Retry later.'; + if (/video unavailable/i.test(raw)) return 'YouTube isn’t letting the server save this video. Try Retry; it may still play on YouTube.'; + if (/requested format (?:is )?not available|no video formats found/i.test(raw)) return 'No downloadable version was available. Try Retry to check other download methods.'; + if (/^ERROR:|^WARNING:|Traceback|yt-dlp exited|yt-dlp not found/i.test(raw)) return 'The server could not save this video. Try Retry.'; + // Existing plain-language connection/quota/validation messages remain useful. + return raw.length <= 180 && !raw.includes('\n') ? raw : 'Could not save this video. Try Retry.'; + } + function appendDetails(container, error) { + if (!error) return; + const doc = container.ownerDocument; + const details = doc.createElement('details'); details.className = 'download-error-details'; + const summary = doc.createElement('summary'); summary.textContent = 'Technical details'; + const pre = doc.createElement('pre'); pre.textContent = String(error?.message || error); + details.append(summary, pre); container.append(details); + return details; + } + const api = { message, appendDetails }; + if (typeof module !== 'undefined') module.exports = api; else root.DownloadErrors = api; +})(typeof window !== 'undefined' ? window : globalThis); diff --git a/frontend/download-errors.test.js b/frontend/download-errors.test.js new file mode 100644 index 0000000..cbd9f00 --- /dev/null +++ b/frontend/download-errors.test.js @@ -0,0 +1,24 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { message } = require('./download-errors.js'); +const { formatJobStatus } = require('./downloads-page.js'); +test('unavailable songs get useful copy without claiming that they were removed', () => { + const raw = 'ERROR: [youtube] wZzRoXymOUU: Video unavailable\nYouTube clients tried: default, web_embedded, web_safari'; + assert.equal(message(raw), 'YouTube isn’t letting the server save this video. Try Retry; it may still play on YouTube.'); + assert.equal(formatJobStatus({ status: 'failed', error: raw }), 'Failed: ' + message(raw)); +}); +test('known restrictions and bot failures have distinct explanations', () => { + assert.match(message('Video unavailable. not available in your country'), /server’s region/); + assert.match(message('Video unavailable. Private video'), /requires access/); + assert.match(message('Video unavailable. Confirm your age'), /requires access/); + assert.match(message('Video has been removed'), /no longer/); + assert.match(message('Sign in to confirm you are not a bot'), /temporarily blocked/); + assert.match(message('Requested format is not available'), /other download methods/); +}); +test('missing and raw tool failures get a short fallback; existing useful messages survive', () => { + assert.equal(message(null), 'Could not save this video. Try Retry.'); + assert.equal(message('ERROR: mysterious extractor failure'), 'The server could not save this video. Try Retry.'); + assert.equal(message('Connection lost'), 'Connection lost'); + assert.equal(message('Please wait a few seconds before trying again.'), 'Please wait a few seconds before trying again.'); + assert.equal(message('a'.repeat(200)), 'Could not save this video. Try Retry.'); +}); diff --git a/frontend/downloads-page.js b/frontend/downloads-page.js index 8d45e80..279efa5 100644 --- a/frontend/downloads-page.js +++ b/frontend/downloads-page.js @@ -1,6 +1,7 @@ /* Downloads page UI module. Grouped jobs, storage info, confirmed cancellation. */ (function (root) { 'use strict'; + const errors = typeof module !== 'undefined' ? require('./download-errors.js') : root.DownloadErrors; const bytes = n => { if (typeof n !== 'number' || isNaN(n) || n <= 0) return '0 MB'; @@ -18,7 +19,7 @@ function formatJobStatus(job) { if (job.status === 'failed') { - return job.error ? `Failed: ${job.error}` : 'Save failed'; + return job.error ? `Failed: ${errors.message(job.error)}` : 'Save failed'; } if (job.status === 'paused') { return 'Paused — resume when you’re ready'; @@ -215,6 +216,8 @@ `; + if (isFailed && job.error) errors.appendDetails(row.querySelector('.card-info'), job.error); + const acts = doc.createElement('div'); acts.className = 'dl-actions'; diff --git a/frontend/downloads.js b/frontend/downloads.js index dc5c4e2..0ba29b9 100644 --- a/frontend/downloads.js +++ b/frontend/downloads.js @@ -1,6 +1,7 @@ /* Download controls and device storage accounting. */ (function(root) { 'use strict'; + const errors = typeof module !== 'undefined' ? require('./download-errors.js') : root.DownloadErrors; const bytes = n => n >= 1073741824 ? `${(n / 1073741824).toFixed(1)} GB` : `${(n / 1048576).toFixed(1)} MB`; function breakdown(files, playlists) { const kinds = { video: 0, audio: 0, eq: 0 }, byId = new Map(); @@ -24,13 +25,15 @@ for (const job of items) { const row = doc.createElement('div'); row.className = 'download-job'; const title = doc.createElement('b'); title.textContent = job.title || job.id; - const status = doc.createElement('small'); status.textContent = job.error || job.status; + const status = doc.createElement('small'); status.textContent = job.error ? errors.message(job.error) : job.status; const progress = doc.createElement('progress'); progress.max = job.progress?.total || 1; if (job.progress?.total) progress.value = job.progress.received || 0; progress.setAttribute('aria-label', `Saving ${job.title || job.id}`); const actions = doc.createElement('div'); for (const action of [...(['active', 'queued'].includes(job.status) ? ['pause'] : [job.status === 'failed' ? 'retry' : 'resume']), 'cancel']) { const button = doc.createElement('button'); button.type = 'button'; button.textContent = action[0].toUpperCase() + action.slice(1); button.setAttribute('aria-label', `${button.textContent} ${job.title || job.id}`); button.onclick = () => { adapter.action(job.id, action); signature = ''; paint(); }; actions.append(button); } - row.append(title, status, progress, actions); jobs.append(row); + row.append(title, status, progress, actions); + if (job.error) errors.appendDetails(row, job.error); + jobs.append(row); } } async function disk() { diff --git a/frontend/index.html b/frontend/index.html index 6767cbc..faee07f 100755 --- a/frontend/index.html +++ b/frontend/index.html @@ -660,6 +660,7 @@ + diff --git a/frontend/offline-pages.css b/frontend/offline-pages.css index b701f4e..e04ddea 100644 --- a/frontend/offline-pages.css +++ b/frontend/offline-pages.css @@ -21,3 +21,13 @@ html[data-layout] .dl-row .card-info { grid-column:2; grid-row:1; min-width:0; } .dl-row .dl-act.danger { color:var(--accent); background:transparent; } .dl-row .dl-act:disabled { opacity:.5; cursor:wait; } @media(min-width:900px) { html[data-layout] .card.dl-row { grid-template-columns:88px minmax(0,1fr); } html[data-layout] .dl-row .thumb { width:88px; } } + +/* Save failures remain readable; raw extractor output is opt-in, never HTML. */ +.dl-failed .card-channel { white-space:normal; line-height:1.5; } +.download-error-details { min-width:0; color:var(--text-2); font:12px/1.5 var(--ui); } +.download-error-details summary { min-height:44px; padding:12px 0; cursor:pointer; } +.download-error-details summary:focus-visible { outline:2px solid var(--accent); outline-offset:2px; border-radius:4px; } +.download-error-details pre { white-space:pre-wrap; overflow-wrap:anywhere; max-height:240px; overflow:auto; margin:0 0 8px; font:12px/1.5 var(--mono); } +.download-job { grid-template-columns:minmax(0,1fr); } +.download-job > div { flex-wrap:wrap; } +.download-job progress { min-width:0; } diff --git a/frontend/sw.js b/frontend/sw.js index 3b6526b..01e2297 100644 --- a/frontend/sw.js +++ b/frontend/sw.js @@ -84,6 +84,7 @@ const SHELL = [ '/p2p-transfer.js', '/p2p-recv-worker.js', '/settings-sections.js', + '/download-errors.js', '/downloads.js', '/downloads-page.js', '/offline-pages.css', diff --git a/tests/download-retry.spec.js b/tests/download-retry.spec.js index 64e4eaf..034afe3 100644 --- a/tests/download-retry.spec.js +++ b/tests/download-retry.spec.js @@ -44,3 +44,29 @@ test('normal resumptions do not force the server to restart a cache job', async expect(retries).toBe(0); expect(await page.evaluate(id => cachedIds.has(id), id)).toBe(true); }); + +for (const layout of ['classic', 'glass-stage']) for (const width of [390, 1440]) for (const screen of ['downloads', 'settings']) { + test(`${layout} ${width} ${screen}: friendly failure retains safe technical details`, async ({ page }) => { + await page.setViewportSize({ width, height: 844 }); + await openClassic(page, { settings: { autoPreload: false } }); + const raw = `ERROR: [youtube] ${id}: Video unavailable\nYouTube clients tried: default, web_embedded, web_safari\n`; + await page.evaluate(({ layout, screen, id, raw }) => { + data.settings.layout = layout; applyAppearance(); + SaveQueue.add({ id, title: 'Rock Medley', channel: 'Petra - Topic' }); + SaveQueue.state(id, 'failed', raw); + view = { type: screen }; render(); + }, { layout, screen, id, raw }); + if (screen === 'settings') await page.locator('[data-section="downloads-storage"]').click(); + const row = page.locator(screen === 'settings' ? '.download-job' : '.dl-failed').first(); + await expect(row).toContainText('YouTube isn’t letting the server save this video.'); + await expect(row.locator('pre')).toBeHidden(); + const summary = row.locator('summary'); + expect((await summary.boundingBox()).height).toBeGreaterThanOrEqual(44); + await summary.click(); + await expect(row.locator('pre')).toHaveText(raw); + expect(await page.evaluate(() => window.unsafeDetail)).toBeUndefined(); + const bounds = await row.evaluate(el => ({ width: el.clientWidth, content: el.scrollWidth, children: [...el.querySelectorAll('*')].filter(child => child.scrollWidth > child.clientWidth + 1).map(child => [child.tagName, child.className, child.clientWidth, child.scrollWidth]) })); + expect(bounds.content <= bounds.width + 1, JSON.stringify(bounds)).toBe(true); + await expect(row.getByRole('button', { name: 'Retry Rock Medley', exact: true })).toBeVisible(); + }); +}