Keep iPhone audio playing through background handoffs

docs/fullscreen-background-audio.md

@@ -0,0 +1,88 @@
+# Fullscreen orientation and iPhone audio continuity
+
+## Task A — done
+
+Settings → Listening → Video & fullscreen offers Auto (follow device), Landscape,
+and Portrait. The choice persists on this device and can be cycled from the
+fullscreen controls. Supported browsers request fullscreen and then lock the
+selected orientation; Auto leaves orientation unlocked. Exit releases our lock.
+
+When orientation locking or element fullscreen is unavailable, the player uses a
+viewport overlay. A portrait viewport with Landscape selected rotates the existing
+stage and its controls 90 degrees; Portrait in a landscape viewport rotates it
+in the opposite direction. Safe-area padding follows the rotated axes. Touch
+seeking, double-tap seeking, volume/brightness swipes and pinch movement use
+inverse coordinates. The video stays in its existing DOM position to avoid
+interrupting WebKit playback. Controls and browser fullscreen exit restore the
+inline surface, including when a native orientation lock was rejected.
+
+Safari's custom overlay fills the available dynamic viewport; it cannot force
+Safari to hide browser chrome or change the phone's physical orientation. An
+installed PWA has no browser toolbar occupying that viewport.
+
+Files: `frontend/fullscreen-orientation.js`, `.css`, `.test.js`, small hooks in
+`app.js`, `index.html`, `sw.js`, `tests/fullscreen-orientation.spec.js` and
+`playwright.fullscreen.config.js`.
+
+## Task B — done; physical iPhone verification remains
+
+The previous progressive handoff loaded the audio source and sought it only when
+the page was hidden, then paused the video before transferring master ownership.
+That required a cold audio decoder start and allowed pause-handler re-entry.
+Foreground recovery also sought the video, whose shared seeking handler then
+sought the still-playing audio. Repeated play/playing/focus/watchdog handlers
+could independently issue the same audio play/seek requests.
+
+On iPhone, progressive playback now uses the existing dual playback path from the
+start: a muted video and an audible audio element. Saved files use the same local
+source; streams prefer their audio-only URL, falling back to the muxed source.
+EQ-render playback already used this path and remains unchanged. Locking promotes
+the already-playing audio to master without loading, seeking or restarting it.
+Unlocking aligns only the silent video; its internal seek does not propagate to
+the audible audio. Pending play requests are shared, small drift does not cause
+seeks, and the watchdog never seeks an element to itself. Hidden focus events do
+not restore the video prematurely. The audio clock still owns background progress
+and ended events, preserving auto-advance and lock-screen playback intent.
+
+This uses an audio decoder alongside video while visible. Audio-only playback
+and tracks that start while already backgrounded retain their existing paths.
+Browsers other than iPhone retain their foreground progressive path. Their cold
+handoff fallback now sets master ownership before media events and waits for audio
+to play before pausing video; returning keeps that audio playing while video resumes.
+
+Files: `frontend/audio-continuity.js`, `.test.js`, player/event hooks in `app.js`,
+`index.html`, `sw.js`, `tests/audio-continuity.spec.js`, and
+`playwright.audio-continuity.config.js`.
+
+## Validation and hardware review
+
+- 132 frontend unit tests pass, including orientation mapping, source selection,
+  small-drift policy, concurrent play deduplication, autoplay rejection retry, and
+  stale pending requests after pause/source reset.
+- `node --check frontend/app.js` and `bun build --no-bundle server/server.js` pass.
+- Eight fullscreen browser cases pass in Chromium/WebKit at 390 × 844, covering
+  Classic and Glass Stage overlays, visible controls, quick toggle, exit/restore,
+  lock/unlock calls and browser exit after lock rejection.
+- Four audio browser cases use real local PCM media in Chromium/WebKit, with an
+  iPhone user agent, covering saved and streamed progressive attachments. Repeated
+  visibility/pagehide/focus/pageshow events keep the same advancing audio element
+  with zero audio loads, pauses, seeks or additional play calls.
+- All 155 server tests pass with `YTDLP_PATH=/tmp/ytplayer-test-yt-dlp` (the existing
+  local test ZIPAPP). The initial default run failed because the system Python
+  environment lacked `yt_dlp`; no server changes were necessary.
+
+Review on an actual iPhone in Safari and installed PWA: saved MP4, streamed songs,
+EQ renders, multiple lock/unlock and app switches, auto-advance while locked,
+Media Session controls, seeking and PiP. Test Landscape/Portrait with safe areas,
+rotation and touch swipes in both themes. On Android/PWA, verify an actual native
+orientation lock and release; browser tests verify API calls with a controlled
+orientation implementation.
+
+iOS may suspend JavaScript, pause media or interrupt the audio session during lock,
+app switching, route changes or incoming calls. A web app cannot guarantee a zero
+OS-level gap. These tests demonstrate removal of application-induced source
+reloads and audio seeks; desktop WebKit cannot simulate physical iOS suspension.
+Tracks first started in the background still reattach their picture on return;
+that existing transition also needs hardware review.
+
+No server imports were added, and nothing was pushed or deployed.

frontend/app.js

@@ -2495,6 +2495,19 @@ const Player = {
     }
   },
 
+  // Use one uninterrupted audio decoder on iPhone from the first play. Locking
+  // then promotes that existing clock; it never loads/seeks a new sound source.
+  prepareBackgroundAudio() {
+    if (!IS_IOS || this.mode !== 'progressive') return;
+    const source = AudioContinuity.progressiveSource(current, els.video.getAttribute('src'));
+    if (!source) return;
+    this.mode = 'dual';
+    this.secondary = els.audio;
+    els.video.muted = true;
+    els.audio.src = source;
+    els.audio.load();
+  },
+
   attach(quality) {
     const V = els.video, A = els.audio;
     // An uploaded audio file has no picture track: play it like audio-only and
@@ -2508,6 +2521,7 @@ const Player = {
     const audioOnly = data.settings.audioOnly || (current && current.kind === 'audio') || bgStart;
 
     // Reset
+    audioContinuity.reset(V); audioContinuity.reset(A);
     V.pause(); A.pause();
     this.stopDrift();
     this._bgHandoff = false;
@@ -2551,6 +2565,7 @@ const Player = {
         A.removeAttribute('src'); A.load();
         els.art.classList.add('hidden');
       }
+      this.prepareBackgroundAudio();
       this.applyVolume();
       this.applySpeed();
       this.master.load();
@@ -2596,6 +2611,7 @@ const Player = {
       return;
     }
 
+    this.prepareBackgroundAudio();
     this.applyVolume();
     this.applySpeed();
     this.master.load();
@@ -2639,15 +2655,14 @@ const Player = {
 
   play() {
     this._wantsPlaying = true;
-    this.master.play().catch(() => {});
-    if (this.secondary) {
-      this.secondary.currentTime = this.master.currentTime;
-      this.secondary.play().catch(() => {});
-    }
+    audioContinuity.play(this.master);
+    audioContinuity.resume(this.secondary, this.master);
   },
   pause() {
     this._wantsPlaying = false;
     this.clearBufferGrace();
+    audioContinuity.reset(this.master);
+    if (this.secondary) audioContinuity.reset(this.secondary);
     this.master.pause();
     if (this.secondary) this.secondary.pause();
   },
@@ -2743,8 +2758,7 @@ const Player = {
       // The element that's actually producing sound in the current mode.
       const sounder = this.soundEl;
       if (sounder && sounder.paused) {
-        sounder.currentTime = this.master.currentTime;
-        sounder.play().catch(() => {});
+        audioContinuity.resume(sounder, this.master);
       }
     }, 1000);
   },
@@ -2785,25 +2799,26 @@ const Player = {
     if (!src) return;   // nothing to continue with — leave playback alone
     const V = els.video, A = els.audio;
     try {
-      A.src = src;
-      A.currentTime = V.currentTime || 0;
+      this._bgHandoff = true;
+      this.master = A;
+      if (A.getAttribute('src') !== src) A.src = src;
+      if (AudioContinuity.needsAlignment(A.currentTime, V.currentTime)) A.currentTime = V.currentTime || 0;
       A.volume = V.volume;
       A.muted = V.muted;
-      const p = A.play();
-      if (p && p.catch) p.catch(() => {});
-      V.pause();
-      // Making <audio> the master keeps soundEl, the drift watchdog and the
-      // progress UI pointed at whatever is actually producing sound.
-      this.master = A;
-      this._bgHandoff = true;
+      audioContinuity.play(A).then(() => {
+        if (this._bgHandoff && this.master === A && !A.paused) V.pause();
+      });
+      // Master ownership is set before play/pause can dispatch another handler.
     } catch {
       this._bgHandoff = false;
+      this.master = V;
     }
   },
 
   // Undo handoffToBackgroundAudio, putting the video back in charge at the
   // position the audio reached while the app was away.
   restoreFromBackgroundAudio() {
+    if (document.visibilityState !== 'visible') return;
     // A track that STARTED in the background is audio-only: re-attach its
     // picture at the position the audio reached.
     if (this._bgAudioMode && current && !data.settings.audioOnly) {
@@ -2823,8 +2838,8 @@ const Player = {
       this.master = V;
       this.secondary = A;
       try {
-        V.currentTime = t;
-        if (this._wantsPlaying) { const p = V.play(); if (p && p.catch) p.catch(() => {}); }
+        if (AudioContinuity.needsAlignment(V.currentTime, t, 0.12)) { this._internalSeek = true; V.currentTime = t; }
+        if (this._wantsPlaying) audioContinuity.play(V);
       } catch { /* the watchdog picks up whatever is left */ }
       return;
     }
@@ -2832,14 +2847,11 @@ const Player = {
     this._bgHandoff = false;
     this.master = V;
     try {
-      V.currentTime = t;
-      A.pause();
-      A.removeAttribute('src');
-      A.load();
-      if (this._wantsPlaying) {
-        const p = V.play();
-        if (p && p.catch) p.catch(() => {});
-      }
+      this.mode = 'dual';
+      this.secondary = A;
+      V.muted = true;
+      if (AudioContinuity.needsAlignment(V.currentTime, t, 0.12)) { this._internalSeek = true; V.currentTime = t; }
+      if (this._wantsPlaying) audioContinuity.play(V);
     } catch { /* the watchdog picks up whatever is left */ }
   },
 
@@ -2851,12 +2863,10 @@ const Player = {
     if (!this._wantsPlaying) return;
     const sounder = this.soundEl;
     if (sounder && sounder.paused) {
-      sounder.currentTime = this.master.currentTime;
-      sounder.play().catch(() => {});
+      audioContinuity.resume(sounder, this.master);
     }
     if (this.mode === 'dual' && this.secondary && this.secondary.paused && !this.master.paused) {
-      this.secondary.currentTime = this.master.currentTime;
-      this.secondary.play().catch(() => {});
+      audioContinuity.resume(this.secondary, this.master);
     }
   },
 };
@@ -2933,7 +2943,7 @@ function wirePlayerEvents() {
   const masterIs = (el) => Player.master === el;
 
   function bind(el) {
-    el.addEventListener('play', () => { if (masterIs(el) && Player.secondary && Player.secondary.paused) { Player.secondary.currentTime = el.currentTime; Player.secondary.play().catch(() => {}); } updatePlayBtn(); });
+    el.addEventListener('play', () => { if (masterIs(el)) audioContinuity.resume(Player.secondary, el); updatePlayBtn(); });
     el.addEventListener('pause', () => {
       // iOS paused the master video because the app went to the background
       // (sometimes before visibilitychange arrives): hand the sound over now.
@@ -2983,8 +2993,7 @@ function wirePlayerEvents() {
       if (Player.secondary && !el.paused && Player.secondary.paused) {
         // Resume-only path (audio silently paused by iOS or a real stall):
         // seeking a *paused* audio element is inaudible, so sync then play.
-        Player.secondary.currentTime = el.currentTime;
-        Player.secondary.play().catch(() => {});
+        audioContinuity.resume(Player.secondary, el);
       } else if (Player.secondary && !el.paused) {
         // Audio kept playing through the video stall — realign the video.
         Player.correctDrift();

frontend/audio-continuity.js

@@ -0,0 +1,30 @@
+(function (root) {
+  'use strict';
+  function progressiveSource(current, videoSource) {
+    return current?.eqAudioUrl || current?.localUrl || current?.audioUrl || videoSource || '';
+  }
+  function needsAlignment(time, clock, threshold = 0.3) {
+    return Number.isFinite(time) && Number.isFinite(clock) && Math.abs(time - clock) >= threshold;
+  }
+  function create() {
+    const pending = new WeakMap();
+    function play(element) {
+      if (pending.has(element)) return pending.get(element);
+      if (!element.paused) return Promise.resolve();
+      let request;
+      try { request = Promise.resolve(element.play()); } catch (error) { request = Promise.reject(error); }
+      // Preserve the intent to play after an OS interruption; the watchdog retries.
+      request = request.catch(() => {}).finally(() => { if (pending.get(element) === request) pending.delete(element); });
+      pending.set(element, request); return request;
+    }
+    function resume(element, clock) {
+      if (!element || !element.paused || pending.has(element)) return;
+      if (clock && element !== clock && needsAlignment(element.currentTime, clock.currentTime)) element.currentTime = clock.currentTime;
+      return play(element);
+    }
+    return { play, resume, reset: element => pending.delete(element) };
+  }
+  const api = { progressiveSource, needsAlignment, create };
+  if (typeof module !== 'undefined' && module.exports) module.exports = api;
+  else { root.AudioContinuity = api; root.audioContinuity = create(); }
+})(globalThis);

frontend/audio-continuity.test.js

@@ -0,0 +1,43 @@
+const { test } = require('node:test');
+const assert = require('node:assert/strict');
+const { progressiveSource, needsAlignment, create } = require('./audio-continuity');
+test('iPhone progressive audio uses saved/EQ sources before remote fallbacks', () => {
+  assert.equal(progressiveSource({ localUrl: 'blob:saved', audioUrl: 'remote' }, 'mux'), 'blob:saved');
+  assert.equal(progressiveSource({ eqAudioUrl: 'blob:eq', localUrl: 'blob:saved' }), 'blob:eq');
+  assert.equal(progressiveSource({ audioUrl: 'audio' }, 'mux'), 'audio');
+  assert.equal(progressiveSource({}, 'mux'), 'mux');
+});
+test('small drift and invalid clocks do not trigger an audible seek', () => {
+  assert.equal(needsAlignment(50.05, 50), false);
+  assert.equal(needsAlignment(49, 50), true);
+  assert.equal(needsAlignment(NaN, 50), false);
+});
+test('simultaneous play, visibility and watchdog resumes share one pending play', async () => {
+  let finish, calls = 0, seeks = 0;
+  const audio = { paused: true, get currentTime() { return 10; }, set currentTime(value) { seeks++; }, play() { calls++; return new Promise(resolve => { finish = resolve; }); } };
+  const controller = create();
+  controller.resume(audio, { currentTime: 10.05 });
+  controller.resume(audio, { currentTime: 12 });
+  const pending = controller.play(audio);
+  assert.equal(calls, 1); assert.equal(seeks, 0);
+  finish(); await pending;
+  controller.resume(audio, audio); // never seek an element to itself
+  assert.equal(seeks, 0); assert.equal(calls, 2); finish();
+});
+test('rejected autoplay can retry and only paused, misaligned secondaries seek', async () => {
+  const controller = create(); let calls = 0;
+  const audio = { paused: true, currentTime: 0, play() { calls++; return Promise.reject(new Error('OS suspension')); } };
+  await controller.play(audio); await controller.resume(audio, { currentTime: 4 });
+  assert.equal(calls, 2); assert.equal(audio.currentTime, 4);
+  audio.paused = false; controller.resume(audio, { currentTime: 10 });
+  assert.equal(audio.currentTime, 4); assert.equal(calls, 2);
+});
+test('a new play after pause/source reset is not blocked by a stale promise', async () => {
+  const controller = create(), finishes = []; let calls = 0;
+  const audio = { paused: true, play() { calls++; return new Promise(resolve => finishes.push(resolve)); } };
+  const old = controller.play(audio); controller.reset(audio);
+  const fresh = controller.play(audio); assert.equal(calls, 2);
+  finishes[0](); await old;
+  assert.equal(controller.play(audio), fresh); // old completion cannot discard the new request
+  finishes[1](); await fresh;
+});

frontend/index.html

@@ -662,6 +662,7 @@
     <script src="p2p-transfer.js"></script>
     <script src="async-guard.js"></script>
     <script src="sw-update.js"></script>
+    <script src="audio-continuity.js"></script>
     <script src="fullscreen-orientation.js"></script>
     <script src="settings-sections.js"></script>
     <script src="direct-protocol.js"></script>

frontend/sw.js

@@ -91,6 +91,7 @@ const SHELL = [
   '/p2p-transfer.js',
   '/p2p-recv-worker.js',
   '/settings-sections.js',
+  '/audio-continuity.js',
   '/fullscreen-orientation.js',
   '/fullscreen-orientation.css',
   '/download-errors.js',

playwright.audio-continuity.config.js

@@ -0,0 +1,2 @@
+const { defineConfig } = require('@playwright/test');
+module.exports = defineConfig({ testDir: './tests', testMatch: /audio-continuity\.spec\.js/, timeout: 30000, workers: 1, projects: [{ name: 'chromium', use: { browserName: 'chromium' } }, { name: 'webkit', use: { browserName: 'webkit' } }], use: { baseURL: 'http://localhost:8098', viewport: { width: 390, height: 844 }, serviceWorkers: 'block' }, webServer: { command: 'npx serve frontend -l 8098 --no-clipboard', port: 8098 } });

tests/audio-continuity.spec.js

@@ -0,0 +1,46 @@
+const { test, expect } = require('@playwright/test');
+const { openClassic, showPlayingFixture } = require('./helpers/classic-fixture');
+// A small local PCM fixture exercises actual media elements without YouTube.
+function wav() {
+  const rate = 8000, samples = rate * 12, bytes = Buffer.alloc(44 + samples * 2);
+  bytes.write('RIFF'); bytes.writeUInt32LE(bytes.length - 8, 4); bytes.write('WAVEfmt ', 8);
+  bytes.writeUInt32LE(16, 16); bytes.writeUInt16LE(1, 20); bytes.writeUInt16LE(1, 22);
+  bytes.writeUInt32LE(rate, 24); bytes.writeUInt32LE(rate * 2, 28); bytes.writeUInt16LE(2, 32); bytes.writeUInt16LE(16, 34);
+  bytes.write('data', 36); bytes.writeUInt32LE(samples * 2, 40);
+  for (let i = 0; i < samples; i++) bytes.writeInt16LE(Math.sin(i * 2 * Math.PI * 220 / rate) * 2000, 44 + i * 2);
+  return bytes;
+}
+for (const source of ['saved', 'streamed']) test(`${source}: iPhone keeps the same playing audio across repeated lock/unlock events`, async ({ page }) => {
+  await page.addInitScript(() => Object.defineProperty(navigator, 'userAgent', { value: 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15' }));
+  await page.route('**/continuity.wav', route => route.fulfill({ contentType: 'audio/wav', body: wav() }));
+  await openClassic(page, { populated: true }); await showPlayingFixture(page);
+  await page.evaluate(source => {
+    current.localUrl = source === 'saved' ? '/continuity.wav' : '';
+    current.audioUrl = '/continuity.wav';
+    Player.attach(source === 'streamed' ? { url: '/continuity.wav', hasAudio: true } : null);
+  }, source);
+  await expect.poll(() => page.evaluate(() => els.audio.readyState)).toBeGreaterThanOrEqual(2);
+  await page.evaluate(() => Player.play());
+  await expect.poll(() => page.evaluate(() => !els.audio.paused && els.audio.currentTime > 0)).toBe(true);
+  const result = await page.evaluate(async () => {
+    const A = els.audio, V = els.video;
+    const initial = A.currentTime, src = A.src;
+    let playCalls = 0, audioSeeks = 0, loads = 0, pauses = 0;
+    const play = A.play.bind(A), load = A.load.bind(A), pause = A.pause.bind(A);
+    A.play = () => { playCalls++; return play(); }; A.load = () => { loads++; load(); }; A.pause = () => { pauses++; pause(); };
+    A.addEventListener('seeking', () => audioSeeks++);
+    Object.defineProperty(document, 'visibilityState', { configurable: true, value: 'hidden' });
+    document.dispatchEvent(new Event('visibilitychange')); window.dispatchEvent(new Event('pagehide'));
+    window.dispatchEvent(new Event('focus')); // focus arriving while hidden must not restore
+    const promoted = Player.master === A && Player._bgHandoff === 'dual';
+    V.pause();
+    await new Promise(resolve => setTimeout(resolve, 350));
+    const advanced = A.currentTime > initial;
+    Object.defineProperty(document, 'visibilityState', { configurable: true, value: 'visible' });
+    document.dispatchEvent(new Event('visibilitychange')); window.dispatchEvent(new Event('pageshow')); window.dispatchEvent(new Event('focus'));
+    await new Promise(resolve => setTimeout(resolve, 200));
+    const answer = { mode: Player.mode, promoted, advanced, sameSource: A.src === src, playing: !A.paused, playCalls, audioSeeks, loads, pauses, restored: Player.master === V && Player.secondary === A };
+    Player.pause(); Player.stopDrift(); return answer;
+  });
+  expect(result).toEqual({ mode: 'dual', promoted: true, advanced: true, sameSource: true, playing: true, playCalls: 0, audioSeeks: 0, loads: 0, pauses: 0, restored: true });
+});