ytplayer/docs/feature-ideas.md

ytplayer — Feature Ideas & Enhancement Roadmap

Status: Research & Proposed Roadmap
Target: Web PWA, Tauri (Windows), zero-native (macOS / Linux)
Date: October 2026

---

1. Context & Architecture Baseline

ytplayer is an ad-free, privacy-preserving media engine with a dual identity:

1. Ad-Free Media Player & Offline PWA: Built with vanilla JS on the frontend and Bun + Hono on the backend (libsql SQLite database). It uses yt-dlp and InnerTube for direct stream extraction without Google accounts, and caches media locally (OPFS in browser, single-file H.264/HEVC faststart on server, WebRTC P2P mesh between devices).
2. Live Worship & Stage Performance Tool: Equipped with Service Mode (wake-locked setlist view, auto-scaling lyric fonts, per-song arrangement notes), synchronized LRC lyrics with section/cue tags, and a Presenter screen (/?present=<code>) with QR pairing.

This document synthesizes high-value feature opportunities gathered from research into modern YouTube frontends (FreeTube, ReVanced, SmartTube, Namida), modern music players (Plexamp, Spotube), and live stage tools (Planning Center Music Stand, OnSong, MultiTracks Playback).

---

2. Feature Proposals by Domain

Domain 1: Smart YouTube & Playback Experience

1.1 SponsorBlock & Non-Music Segment Skipping

• The Concept: Automatically skip or visually mark sponsored segments, self-promos, interaction reminders, and non-music sections.
• Why it matters: Music videos frequently feature skits, prolonged silent pauses, or spoken intros. Skipping music_offtopic ensures the music starts immediately.
• Technical Path:
  • Free, unauthenticated REST API: GET https://sponsor.ajay.app/api/skipSegments?videoID=<id>&categories=["sponsor","intro","outro","music_offtopic","selfpromo"].
  • Highlight segments as color-coded spans over the seek bar (#timeline / Glass Stage scrubber).
  • During the playback tick in frontend/app.js, if currentTime enters a skippable segment and auto-skip is enabled in Settings, seek to the segment end and show an "Unskip" toast.
  • Server media cache can optionally apply --sponsorblock-mark during ffmpeg intake.

1.2 Automatic YouTube Chapter Extraction & Interactive Scrubber

• The Concept: Automatically parse video chapters from YouTube metadata and display chapter splits on the progress bar.
• Why it matters: Currently, chapters are only created manually. Full albums, sermons, conference talks, and live sets already have timestamps in video descriptions.
• Technical Path:
  • In server/server.js (/api/streams), expose chapters: [{ start_time, end_time, title }] directly from InnerTube or yt-dlp.
  • Render tick marks / gaps on the progress scrubber for each chapter boundary.
  • Display the current chapter title next to the timestamp.
  • Add keyboard shortcuts: [ (previous chapter) and ] (next chapter).

1.3 Return YouTube Dislike (RYD) & Quality Rating

• The Concept: Show community like/dislike ratios and ratings for videos.
• Why it matters: Helps identify misleading arrangements, clickbait, poor audio rips, or broken videos before playback.
• Technical Path:
  • Query https://returnyoutubedislikeapi.com/votes?videoId=<id> (public, CORS-enabled).
  • Display a discreet like ratio bar or percentage badge on the video card and now-playing details.

1.4 On-Screen Subtitles & Closed Captions (CC) Overlay

• The Concept: Render synchronized captions directly over the video playback stage.
• Why it matters: ytplayer currently only uses captions for transcript search or importing into lyrics. Spoken sermons, tutorials, foreign tracks, and non-musical videos need readable on-screen subtitles.
• Technical Path:
  • Add a CC button on the player controls.
  • Fetch timed lines from /api/notes/:id/captions.
  • Render an HTML overlay synchronized to Player.master.currentTime with settings for font size, background opacity, and positioning.

1.5 Silence Skipping (Talk & Sermon Mode)

• The Concept: Automatically speed up or skip past silent gaps in spoken-word media.
• Why it matters: Long sermons, podcasts, and talks often have pauses; skipping silence saves 10–20% of listening time without altering speech tone.
• Technical Path:
  • Connect an AnalyserNode or AudioWorklet in the Web Audio graph.
  • If audio energy (RMS) stays below -45 dB for >800 ms, ramp playbackRate to 2.5× until speech resumes.

---

Domain 2: Audiophile & Music Player Polish

2.1 EBU R128 / LUFS Loudness Normalization

• The Concept: Eliminate jarring volume jumps between tracks in playlists and queues.
• Why it matters: YouTube master volumes vary widely (-24 LUFS to -8 LUFS). Normalization ensures consistent listening levels.
• Technical Path:
  • When generating waveform peaks in server/media-cache.js, run ffmpeg's ebur128 filter once to measure integrated loudness (LUFS).
  • Store lufs in the database and pass it via /api/streams and /api/media/:id.
  • In frontend/app.js, apply a Web Audio GainNode to normalize tracks to a standard -14 LUFS target.

2.2 Real-Time Pitch Shifting & Independent Tempo Adjustment

• The Concept: Transpose playback key (±6 semitones) without altering tempo, or slow down tempo without changing pitch.
• Why it matters: Singers and worship teams often need to adapt songs to a band's vocal key. Instrumentalists need to slow down fast sections to learn them.
• Technical Path:
  • Use an AudioWorklet running a Phase Vocoder or SoundTouchJS DSP processor.
  • Provide stepper controls for semitone transposition and tempo percentage (0.5×–1.5×).

2.3 Ambient Glow / Reactive Lighting Mode

• The Concept: A soft dynamic glow behind the video player that mirrors current video colors (similar to YouTube Ambient Mode or Apple Music dynamic art).
• Technical Path:
  • Render video frames onto an offscreen 16×16 <canvas> at ~10 FPS.
  • Display a blurred canvas (filter: blur(50px)) directly behind the player container with smooth CSS opacity.
  • Auto-disable when settings.perfMode is active or under prefers-reduced-motion.

2.4 Smart Autoplay Radio / Endless Mix

• The Concept: Keep playing related songs when the active playlist or queue concludes.
• Technical Path:
  • Fetch candidate related tracks via InnerTube next continuity tokens.
  • Re-rank candidates using the client's local StatsTrack (playCount, listening duration, and skips) to favor familiar favorites while avoiding immediate repeats.

2.5 Audio Stem Separation & Vocal Attenuator (Karaoke / Rehearsal Mode)

• The Concept: Mute or attenuate lead vocals in real time (🎤 Vocal: 100% ────● 0%), or isolate the vocal track for harmony practice.
• Why it matters: Gives singers clean backing tracks for rehearsal and live events even when official minus-one recordings do not exist.
• Technical Path:
  • Client-side: Run demucs-onnx with WebGPU acceleration to process saved OPFS tracks offline.
  • Server-side: Provide GET /api/media/:id/stems (caching .vocals.m4a and .instrumental.m4a alongside the existing .m4a audio sidecar in server/media-cache.js).
  • Frontend: Two synchronized audio elements or Web Audio GainNode faders for seamless real-time blending.

2.6 10-Band Graphic Equalizer & Spatial Hall Simulation

• The Concept: Upgrade the existing 5-band filter in EQ (frontend/app.js) to a full 10-band graphic equalizer with acoustic presets and spatial audio widening.
• Technical Path:
  • 10 BiquadFilterNode bands (31Hz, 62Hz, 125Hz, 250Hz, 500Hz, 1kHz, 2kHz, 4kHz, 8kHz, 16kHz).
  • Presets: Vocal Clarity (speech presence boost), Acoustic Warmth, Worship Stage, Bass Punch, and Flat.
  • Spatial stereo widener using a Haas-effect micro-delay and StereoPannerNode for an immersive headphone soundstage.

---

Domain 3: Live Worship, Band & Stage Performance Engine

3.1 Interactive ChordPro Sheets & Live Transposition

• The Concept: Display chords inline or above lyrics with 1-tap live transposition and capo calculation.
• Why it matters: Replaces the need for external music stand apps (Planning Center Music Stand, OnSong) on stage.
• Technical Path:
  • Extend frontend/lyrics-core.js to parse ChordPro directives ([G], [D/F#], {key: G}).
  • Add a Transpose controller (+1 / -1 semitone, Capo selector) that dynamically recalculates chord symbols in real time.
  • Add a view toggle: "Lyrics Only" vs. "Chords & Lyrics".

3.2 Bluetooth Foot Pedal / Page-Turner Support

• The Concept: Hands-free navigation for musicians on stage using standard Bluetooth foot pedals (AirTurn, PageFlip, Donner, Stomp).
• Why it matters: Musicians cannot remove their hands from instruments while playing.
• Technical Path:
  • Standard foot pedals map to keyboard keys: PageDown, PageUp, ArrowUp, ArrowDown.
  • Extend the global keydown listener in frontend/app.js to handle pedal events in Service Mode:
    • PageDown / PageUp: Scroll down / up by one lyric section.
    • Configurable actions: Next/Previous track, trigger A-B loop, or jump to Chorus.

3.3 Confidence Monitor & Stage Clock (Presenter Mode)

• The Concept: A high-visibility stage display mode for the Presenter screen (/?present=<code>).
• Why it matters: Singers and worship leaders need glanceable cues for upcoming lyrics, section names, and remaining time.
• Technical Path:
  • In Presenter (frontend/app.js), provide a Confidence Monitor layout:
    • Current Line: Extra-large, high-contrast text.
    • Next Line Preview: Smaller, dimmed text directly underneath.
    • Section Badge: Prominent label (e.g. # CHORUS 2).
    • Stage Clock & Timer: Current wall clock plus elapsed/remaining song timer.

3.4 Broadcast Lower-Third Overlay for OBS & Live Streams

• The Concept: A dedicated live stream output URL (/?present=<code>&mode=lowerthird) designed to be added as a transparent Browser Source in OBS, vMix, or ATEM.
• Why it matters: Enables churches and streamers to overlay synchronized lower-third lyrics onto their video feed without separate presentation software.
• Technical Path:
  • Transparent background (background: transparent;).
  • Auto-updating 2-line broadcast graphic with smooth slide/fade transitions and drop-shadows.
  • Driven in real-time by the host or phone remote via the existing WebSocket relay in server/remote.js.

3.5 Section Quick-Jump Pad

• The Concept: A grid of buttons representing song sections ([Intro], [Verse 1], [Chorus], [Bridge], [Outro]).
• Why it matters: Worship teams often repeat choruses or spontaneously transition between sections during live services.
• Technical Path:
  • Parse # Section headers from synced lyrics in frontend/lyrics-core.js.
  • Render an arrangement button strip in Service Mode.
  • Tapping a section immediately seeks playback to that section's start timestamp.

3.6 Hardware MIDI Controller Integration (Web MIDI API)

• The Concept: Direct plug-and-play support for hardware USB/Bluetooth MIDI pedalboards (Morningstar MC3/MC6, Looptimus, Behringer FCB1010) without native drivers.
• Technical Path:
  • Use the browser navigator.requestMIDIAccess() API.
  • Map MIDI Note-On and CC messages to player actions (Play/Pause, Next Track, Chorus Repeat, Tap Tempo, Mute).

3.7 Planning Center Services (PCO) & SongSelect Setlist Importer

• The Concept: One-click setlist and chord chart importing from Planning Center Services and CCLI SongSelect.
• Technical Path:
  • Import setlist text or CSV files from PCO.
  • Auto-match titles with YouTube media, attaching keys (@ Key E) and chord sheets directly to playlist entries.

3.8 Bilingual & Phonetic Lyrics (Romanization / Transliteration)

• The Concept: Simultaneous dual-language or phonetic pronunciation lyrics (e.g. Mandarin with Pinyin, Spanish/English).
• Technical Path:
  • Extend frontend/lyrics-core.js to parse secondary translation lines.
  • Display translations or romanized cues in smaller muted text under the primary lyric line.

3.9 Visual Metronome & Count-In Click

• The Concept: Visual flash or gentle audio click count-in before track playback begins.
• Why it matters: Enables the band to start playing on beat before the recorded track introduces audio.
• Technical Path:
  • When @ 70 BPM metadata is present in song notes, offer an optional 4-beat visual flash on the stage frame or Web Audio click before playback starts.

---

Domain 4: Interactive Piano Transcription, Sheet Music & Practice Engine

4.1 Automatic Audio-to-Piano Transcription (Polyphonic Audio to Score)

• The Concept: Transcribe piano melodies and chords from recorded audio into polyphonic note events (MIDI / MusicXML).
• Why it matters: Allows musicians, vocalists, and students to learn the exact piano parts played in any YouTube song or uploaded track.
• Technical Path:
  • Option A (Client-Side WASM / WebGPU): Run Spotify's Basic Pitch model (@spotify/basic-pitch) in the browser via TensorFlow.js or ONNX Runtime Web. It performs polyphonic pitch prediction directly on decoded audio frames without server compute.
  • Option B (Server-Side Worker, like lyrics-worker): Run ByteDance's piano_transcription_inference (High-Resolution Piano Transcription on PyTorch/ONNX) inside a container worker. Saves <id>.notes.json / <id>.mid to the cache directory.
  • Output data shape: Array of note events [{ pitch: 60, start: 1.25, end: 1.85, velocity: 84 }].

4.2 Sheet Music Generation & Real-Time Scrolling Score

• The Concept: Convert transcribed note events into standard grand staff sheet music (Treble and Bass clefs) rendered on screen.
• Technical Path:
  • Render sheet notation on an HTML5 <canvas> or <svg> using VexFlow or OpenSheetMusicDisplay (OSMD).
  • Display a vertical playhead cursor that scrolls synchronously with Player.master.currentTime.
  • Support instant transposition: adjusting the song key shifts the musical notation and key signature on the fly.

4.3 Interactive Virtual Piano & Falling Notes (Synthesia Style)

• The Concept: A responsive 88-key or 61-key virtual piano keyboard that visualizes which keys to press in sync with playback.
• Visual Modes:
  • Falling Notes Mode: Vertical roll above the keyboard where color-coded note bars fall downward toward the keys (Treble notes in Amber/Coral, Bass notes in Cyan/Blue). When a bar hits the key line, the corresponding piano key illuminates and depresses.
  • Sheet Music + Lit Keys Mode: Traditional score at the top, illuminated keyboard at the bottom highlighting active notes in real time.
• Technical Path:
  • Canvas-rendered piano keyboard with responsive key sizing that fits desktop, tablet, and mobile orientations.
  • Note rendering loop tied to requestAnimationFrame and driven by the current playback timestamp.

4.4 Queued Song & Section Learning Mode

• The Concept: A dedicated learning queue and practice loop system that guides users step-by-step through mastering a song.
• Features:
  • Queue Practice Sessions: Queue an entire song, specific sections (e.g. [Intro], [Verse 1], [Chorus]), or arbitrary A-B loop bars into the practice queue.
  • Tempo Slowdown (Pitch-Preserved): Slow down playback to 50%, 65%, 75%, or 90% speed without altering key.
  • Speed Ramp Auto-Trainer: Start a looped section at 60% speed; each time the user completes the loop, speed automatically increases by +5% until 100% full tempo is achieved.
  • Loop Pause Interval: A configurable 1–2 beat pause between repetitions to allow players to reset their hand position.

4.5 Interactive "Wait-for-Note" Practice via Hardware MIDI Input

• The Concept: Connect any USB or Bluetooth digital piano / MIDI keyboard to the computer or tablet. The player pauses and waits for the user to press the correct key(s) before advancing!
• Technical Path:
  • Listen to incoming MIDI noteon events via the Web MIDI API (navigator.requestMIDIAccess()).
  • Compare the user's played MIDI note number against active notes in the transcription dataset.
  • If the note is correct, turn the key green and resume playback to the next note event. If incorrect, turn the key red and provide visual guidance on the virtual keyboard.

4.6 Technical Feasibility Analysis

Component	Technical Feasibility	Resource Cost & Constraints	Recommendation
Audio-to-MIDI Transcription	High (Proven)	Basic Pitch: ~15 MB model file, runs on client WebGPU/CPU in ~0.3×–0.8× real time. ByteDance: higher accuracy, requires ~1.5 GB PyTorch model on server.	Use Basic Pitch (WASM/ONNX) on client for instant preview; offload to homelab server worker for full-song high-fidelity scores.
Piano Roll / Virtual Keyboard	High (Straightforward)	Pure HTML5 Canvas rendering. Minimal CPU overhead (~60 FPS with <5% CPU usage).	Built natively into frontend without heavy external dependencies.
Sheet Music Engraving	High (Proven)	VexFlow (~180 KB minified) renders standard grand staff notation cleanly from JSON notes.	Dynamic import (import()) of VexFlow only when the user opens Sheet View.
Interactive MIDI Feedback	High (Zero-latency)	Web MIDI API is supported natively in Chrome, Edge, and Opera (and via polyfill in Firefox/Safari). Latency is sub-5ms.	Native browser event listeners listening to MIDI input.

---

Domain 5: Collaboration, Windowing & Mobile Experience

5.1 Document Picture-in-Picture (Always-On-Top Lyrics & Chords)

• The Concept: An always-on-top floating desktop window containing interactive synchronized lyrics, chords, and transport controls.
• Technical Path:
  • Use the Chrome/Edge documentPictureInPicture.requestWindow({ width: 420, height: 500 }) API.
  • Moves the active #stageLyrics or ChordPro view into the floating window with full DOM interactivity.

5.2 Voice Search & Driving Car Mode

• The Concept: A hands-free, distraction-free interface for driving or kitchen listening.
• Features:
  • Voice Search: Microphone button on the search bar powered by webkitSpeechRecognition for real-time speech-to-query search.
  • Car Mode: One-tap toggle for a simplified layout with giant 100px touch buttons (Play/Pause, Next, Previous, 15s Rewind) and disabled scroll lists.

5.3 Collaborative Setlists & Playlists

• The Concept: Multiple devices editing a single playlist simultaneously with live sync.
• Technical Path:
  • Extend playlist sync in server/db.js with shared write keys.
  • Broadcast item reordering and song additions over the existing WebSocket hub in server/party.js.

5.4 Watch Party DJ Mode & Upvote Queue

• The Concept: Allow guests in a watch party (/ws/party) to submit song requests that the group can upvote.
• Technical Path:
  • Add a "Suggested Queue" tab in the watch party drawer.
  • Upvotes dynamically sort songs; when current playback finishes, the highest-voted song plays next.

---

Domain 6: Library, Offline & Sync Innovations

6.1 Visual Download Manager & Storage Dashboard

• The Concept: A unified management dashboard in Settings for offline media.
• Technical Path:
  • Show active download tasks with real-time progress bars.
  • List storage usage grouped by playlist, audio vs. video format, and OPFS vs. server cache.
  • Add controls: "Pause All", "Resume All", and "Retry Failed".

6.2 Smart Auto-Offline Favorites

• The Concept: Keep a user's most essential songs available offline automatically.
• Technical Path:
  • Setting: "Keep Top 25 Most-Played Songs Offline" or "Keep Active Setlist Offline".
  • Automated background routine checks local playCount and triggers cache downloads for missing items during idle time.

6.3 Self-Contained Offline Playlist Bundling (Zip Export)

• The Concept: Export a full offline bundle containing playlist JSON plus the saved audio/video files for complete air-gapped device migration.

---

3. Implementation Matrix & Recommended Phases

Phase	Feature	Effort	Prerequisites & Reused Modules
Phase 1: Quick Wins	SponsorBlock & Non-Music Skip	Low	Free SponsorBlock API, playback tick in frontend/app.js
	Automatic YouTube Chapter Ticks	Low	InnerTube metadata in server/server.js, timeline scrubber
	Bluetooth Foot Pedal Shortcuts	Low	Keydown event listener in frontend/app.js
	Section Quick-Jump Pad	Low	Lyric section parser in frontend/lyrics-core.js
	Voice Search (SpeechRecognition)	Low	Web Speech API, search input handler
Phase 2: Stage & Audio	EBU R128 LUFS Normalization	Medium	ffmpeg ebur128 in server/media-cache.js, Web Audio GainNode
	ChordPro Display & Transpose	Medium	frontend/lyrics-core.js, chord parser
	Confidence Monitor Stage View	Medium	Presenter & fitLyricLines in frontend/app.js
	OBS Lower-Third Overlay	Low-Med	server/remote.js WebSocket relay, transparent route
	On-Screen Subtitles (CC)	Medium	/api/notes/:id/captions, overlay renderer
Phase 3: Piano & Interactive Learning	Interactive Piano Roll & Key Visualizer	Medium	HTML5 Canvas keyboard renderer, note event synchronizer
	Audio-to-MIDI Transcription	Medium-High	Spotify Basic Pitch (WASM) / ByteDance transcription worker
	Sheet Music View (VexFlow)	Medium	VexFlow dynamic import, MusicXML / MIDI generator
	Wait-for-Note Practice (Web MIDI)	Medium	Web MIDI API (navigator.requestMIDIAccess), practice queue
Phase 4: Advanced Engine	Audio Stem Separation (Karaoke)	High	ONNX Runtime Web / server Demucs
	Pitch Shift & Tempo Worklet	High	AudioWorklet / SoundTouchJS DSP
	Document Picture-in-Picture	Low-Med	Chrome Document PiP API
	Collaborative Setlists	High	WebSocket hub in server/party.js