From 258aad83ddfcb0c29573c0a319599abf0dfccfba Mon Sep 17 00:00:00 2001 From: Jonathan Sykes Date: Sat, 3 Oct 2026 07:54:41 +0800 Subject: [PATCH] Add piano transcription, interactive key visualizer, and practice queue features --- docs/feature-ideas.md | 163 +++++++++++++++++++++++++++++++++++------- 1 file changed, 136 insertions(+), 27 deletions(-) diff --git a/docs/feature-ideas.md b/docs/feature-ideas.md index cdeed14..1eb6963 100644 --- a/docs/feature-ideas.md +++ b/docs/feature-ideas.md @@ -92,6 +92,21 @@ This document synthesizes high-value feature opportunities gathered from researc * Fetch candidate related tracks via InnerTube `next` continuity tokens. * Re-rank candidates using the client's local `StatsTrack` (`playCount`, listening duration, and skips) to favor familiar favorites while avoiding immediate repeats. +#### 2.5 Audio Stem Separation & Vocal Attenuator (Karaoke / Rehearsal Mode) +* **The Concept**: Mute or attenuate lead vocals in real time (`🎤 Vocal: 100% ────● 0%`), or isolate the vocal track for harmony practice. +* **Why it matters**: Gives singers clean backing tracks for rehearsal and live events even when official minus-one recordings do not exist. +* **Technical Path**: + * Client-side: Run `demucs-onnx` with WebGPU acceleration to process saved OPFS tracks offline. + * Server-side: Provide `GET /api/media/:id/stems` (caching `.vocals.m4a` and `.instrumental.m4a` alongside the existing `.m4a` audio sidecar in `server/media-cache.js`). + * Frontend: Two synchronized audio elements or Web Audio `GainNode` faders for seamless real-time blending. + +#### 2.6 10-Band Graphic Equalizer & Spatial Hall Simulation +* **The Concept**: Upgrade the existing 5-band filter in `EQ` (`frontend/app.js`) to a full 10-band graphic equalizer with acoustic presets and spatial audio widening. +* **Technical Path**: + * 10 `BiquadFilterNode` bands (31Hz, 62Hz, 125Hz, 250Hz, 500Hz, 1kHz, 2kHz, 4kHz, 8kHz, 16kHz). + * Presets: *Vocal Clarity (speech presence boost), Acoustic Warmth, Worship Stage, Bass Punch, and Flat*. + * Spatial stereo widener using a Haas-effect micro-delay and `StereoPannerNode` for an immersive headphone soundstage. + --- ### Domain 3: Live Worship, Band & Stage Performance Engine @@ -123,7 +138,15 @@ This document synthesizes high-value feature opportunities gathered from researc * **Section Badge**: Prominent label (e.g. `# CHORUS 2`). * **Stage Clock & Timer**: Current wall clock plus elapsed/remaining song timer. -#### 3.4 Section Quick-Jump Pad +#### 3.4 Broadcast Lower-Third Overlay for OBS & Live Streams +* **The Concept**: A dedicated live stream output URL (`/?present=&mode=lowerthird`) designed to be added as a transparent Browser Source in OBS, vMix, or ATEM. +* **Why it matters**: Enables churches and streamers to overlay synchronized lower-third lyrics onto their video feed without separate presentation software. +* **Technical Path**: + * Transparent background (`background: transparent;`). + * Auto-updating 2-line broadcast graphic with smooth slide/fade transitions and drop-shadows. + * Driven in real-time by the host or phone remote via the existing WebSocket relay in `server/remote.js`. + +#### 3.5 Section Quick-Jump Pad * **The Concept**: A grid of buttons representing song sections (`[Intro]`, `[Verse 1]`, `[Chorus]`, `[Bridge]`, `[Outro]`). * **Why it matters**: Worship teams often repeat choruses or spontaneously transition between sections during live services. * **Technical Path**: @@ -131,7 +154,25 @@ This document synthesizes high-value feature opportunities gathered from researc * Render an arrangement button strip in Service Mode. * Tapping a section immediately seeks playback to that section's start timestamp. -#### 3.5 Visual Metronome & Count-In Click +#### 3.6 Hardware MIDI Controller Integration (Web MIDI API) +* **The Concept**: Direct plug-and-play support for hardware USB/Bluetooth MIDI pedalboards (Morningstar MC3/MC6, Looptimus, Behringer FCB1010) without native drivers. +* **Technical Path**: + * Use the browser `navigator.requestMIDIAccess()` API. + * Map MIDI Note-On and CC messages to player actions (Play/Pause, Next Track, Chorus Repeat, Tap Tempo, Mute). + +#### 3.7 Planning Center Services (PCO) & SongSelect Setlist Importer +* **The Concept**: One-click setlist and chord chart importing from Planning Center Services and CCLI SongSelect. +* **Technical Path**: + * Import setlist text or CSV files from PCO. + * Auto-match titles with YouTube media, attaching keys (`@ Key E`) and chord sheets directly to playlist entries. + +#### 3.8 Bilingual & Phonetic Lyrics (Romanization / Transliteration) +* **The Concept**: Simultaneous dual-language or phonetic pronunciation lyrics (e.g. Mandarin with Pinyin, Spanish/English). +* **Technical Path**: + * Extend `frontend/lyrics-core.js` to parse secondary translation lines. + * Display translations or romanized cues in smaller muted text under the primary lyric line. + +#### 3.9 Visual Metronome & Count-In Click * **The Concept**: Visual flash or gentle audio click count-in before track playback begins. * **Why it matters**: Enables the band to start playing on beat before the recorded track introduces audio. * **Technical Path**: @@ -139,44 +180,106 @@ This document synthesizes high-value feature opportunities gathered from researc --- -### Domain 4: Library, Offline & Sync Innovations +### Domain 4: Interactive Piano Transcription, Sheet Music & Practice Engine -#### 4.1 Visual Download Manager & Storage Dashboard +#### 4.1 Automatic Audio-to-Piano Transcription (Polyphonic Audio to Score) +* **The Concept**: Transcribe piano melodies and chords from recorded audio into polyphonic note events (MIDI / MusicXML). +* **Why it matters**: Allows musicians, vocalists, and students to learn the exact piano parts played in any YouTube song or uploaded track. +* **Technical Path**: + * **Option A (Client-Side WASM / WebGPU)**: Run Spotify's **Basic Pitch** model (`@spotify/basic-pitch`) in the browser via TensorFlow.js or ONNX Runtime Web. It performs polyphonic pitch prediction directly on decoded audio frames without server compute. + * **Option B (Server-Side Worker, like `lyrics-worker`)**: Run ByteDance's `piano_transcription_inference` (High-Resolution Piano Transcription on PyTorch/ONNX) inside a container worker. Saves `.notes.json` / `.mid` to the cache directory. + * Output data shape: Array of note events `[{ pitch: 60, start: 1.25, end: 1.85, velocity: 84 }]`. + +#### 4.2 Sheet Music Generation & Real-Time Scrolling Score +* **The Concept**: Convert transcribed note events into standard grand staff sheet music (Treble and Bass clefs) rendered on screen. +* **Technical Path**: + * Render sheet notation on an HTML5 `` or `` using **VexFlow** or **OpenSheetMusicDisplay (OSMD)**. + * Display a vertical playhead cursor that scrolls synchronously with `Player.master.currentTime`. + * Support instant transposition: adjusting the song key shifts the musical notation and key signature on the fly. + +#### 4.3 Interactive Virtual Piano & Falling Notes (Synthesia Style) +* **The Concept**: A responsive 88-key or 61-key virtual piano keyboard that visualizes which keys to press in sync with playback. +* **Visual Modes**: + * **Falling Notes Mode**: Vertical roll above the keyboard where color-coded note bars fall downward toward the keys (Treble notes in Amber/Coral, Bass notes in Cyan/Blue). When a bar hits the key line, the corresponding piano key illuminates and depresses. + * **Sheet Music + Lit Keys Mode**: Traditional score at the top, illuminated keyboard at the bottom highlighting active notes in real time. +* **Technical Path**: + * Canvas-rendered piano keyboard with responsive key sizing that fits desktop, tablet, and mobile orientations. + * Note rendering loop tied to `requestAnimationFrame` and driven by the current playback timestamp. + +#### 4.4 Queued Song & Section Learning Mode +* **The Concept**: A dedicated learning queue and practice loop system that guides users step-by-step through mastering a song. +* **Features**: + * **Queue Practice Sessions**: Queue an entire song, specific sections (e.g. `[Intro]`, `[Verse 1]`, `[Chorus]`), or arbitrary A-B loop bars into the practice queue. + * **Tempo Slowdown (Pitch-Preserved)**: Slow down playback to 50%, 65%, 75%, or 90% speed without altering key. + * **Speed Ramp Auto-Trainer**: Start a looped section at 60% speed; each time the user completes the loop, speed automatically increases by +5% until 100% full tempo is achieved. + * **Loop Pause Interval**: A configurable 1–2 beat pause between repetitions to allow players to reset their hand position. + +#### 4.5 Interactive "Wait-for-Note" Practice via Hardware MIDI Input +* **The Concept**: Connect any USB or Bluetooth digital piano / MIDI keyboard to the computer or tablet. The player pauses and waits for the user to press the correct key(s) before advancing! +* **Technical Path**: + * Listen to incoming MIDI `noteon` events via the Web MIDI API (`navigator.requestMIDIAccess()`). + * Compare the user's played MIDI note number against active notes in the transcription dataset. + * If the note is correct, turn the key green and resume playback to the next note event. If incorrect, turn the key red and provide visual guidance on the virtual keyboard. + +#### 4.6 Technical Feasibility Analysis + +| Component | Technical Feasibility | Resource Cost & Constraints | Recommendation | +|:---|:---|:---|:---| +| **Audio-to-MIDI Transcription** | **High (Proven)** | Basic Pitch: ~15 MB model file, runs on client WebGPU/CPU in ~0.3×–0.8× real time. ByteDance: higher accuracy, requires ~1.5 GB PyTorch model on server. | Use **Basic Pitch (WASM/ONNX)** on client for instant preview; offload to homelab server worker for full-song high-fidelity scores. | +| **Piano Roll / Virtual Keyboard** | **High (Straightforward)** | Pure HTML5 Canvas rendering. Minimal CPU overhead (~60 FPS with <5% CPU usage). | Built natively into frontend without heavy external dependencies. | +| **Sheet Music Engraving** | **High (Proven)** | **VexFlow** (~180 KB minified) renders standard grand staff notation cleanly from JSON notes. | Dynamic import (`import()`) of VexFlow only when the user opens Sheet View. | +| **Interactive MIDI Feedback** | **High (Zero-latency)** | Web MIDI API is supported natively in Chrome, Edge, and Opera (and via polyfill in Firefox/Safari). Latency is sub-5ms. | Native browser event listeners listening to MIDI input. | + +--- + +### Domain 5: Collaboration, Windowing & Mobile Experience + +#### 5.1 Document Picture-in-Picture (Always-On-Top Lyrics & Chords) +* **The Concept**: An always-on-top floating desktop window containing interactive synchronized lyrics, chords, and transport controls. +* **Technical Path**: + * Use the Chrome/Edge `documentPictureInPicture.requestWindow({ width: 420, height: 500 })` API. + * Moves the active `#stageLyrics` or ChordPro view into the floating window with full DOM interactivity. + +#### 5.2 Voice Search & Driving Car Mode +* **The Concept**: A hands-free, distraction-free interface for driving or kitchen listening. +* **Features**: + * **Voice Search**: Microphone button on the search bar powered by `webkitSpeechRecognition` for real-time speech-to-query search. + * **Car Mode**: One-tap toggle for a simplified layout with giant 100px touch buttons (Play/Pause, Next, Previous, 15s Rewind) and disabled scroll lists. + +#### 5.3 Collaborative Setlists & Playlists +* **The Concept**: Multiple devices editing a single playlist simultaneously with live sync. +* **Technical Path**: + * Extend playlist sync in `server/db.js` with shared write keys. + * Broadcast item reordering and song additions over the existing WebSocket hub in `server/party.js`. + +#### 5.4 Watch Party DJ Mode & Upvote Queue +* **The Concept**: Allow guests in a watch party (`/ws/party`) to submit song requests that the group can upvote. +* **Technical Path**: + * Add a "Suggested Queue" tab in the watch party drawer. + * Upvotes dynamically sort songs; when current playback finishes, the highest-voted song plays next. + +--- + +### Domain 6: Library, Offline & Sync Innovations + +#### 6.1 Visual Download Manager & Storage Dashboard * **The Concept**: A unified management dashboard in Settings for offline media. -* **Why it matters**: Gives clear visibility into queued, active, and completed downloads with storage usage breakdowns. * **Technical Path**: * Show active download tasks with real-time progress bars. * List storage usage grouped by playlist, audio vs. video format, and OPFS vs. server cache. * Add controls: "Pause All", "Resume All", and "Retry Failed". -#### 4.2 Smart Auto-Offline Favorites +#### 6.2 Smart Auto-Offline Favorites * **The Concept**: Keep a user's most essential songs available offline automatically. * **Technical Path**: * Setting: *"Keep Top 25 Most-Played Songs Offline"* or *"Keep Active Setlist Offline"*. * Automated background routine checks local `playCount` and triggers cache downloads for missing items during idle time. -#### 4.3 Self-Contained Offline Playlist Bundling (Zip Export) +#### 6.3 Self-Contained Offline Playlist Bundling (Zip Export) * **The Concept**: Export a full offline bundle containing playlist JSON plus the saved audio/video files for complete air-gapped device migration. --- -### Domain 5: Collaboration & Watch Parties - -#### 5.1 Collaborative Setlists & Playlists -* **The Concept**: Multiple devices editing a single playlist simultaneously with live sync. -* **Why it matters**: Band leaders and team members can collaborate on Sunday setlists from different locations. -* **Technical Path**: - * Extend playlist sync in `server/db.js` with shared write keys. - * Broadcast item reordering and song additions over the existing WebSocket hub in `server/party.js`. - -#### 5.2 Watch Party DJ Mode & Upvote Queue -* **The Concept**: Allow guests in a watch party (`/ws/party`) to submit song requests that the group can upvote. -* **Technical Path**: - * Add a "Suggested Queue" tab in the watch party drawer. - * Upvotes dynamically sort songs; when current playback finishes, the highest-voted song plays next. - ---- - ## 3. Implementation Matrix & Recommended Phases | Phase | Feature | Effort | Prerequisites & Reused Modules | @@ -185,11 +288,17 @@ This document synthesizes high-value feature opportunities gathered from researc | | **Automatic YouTube Chapter Ticks** | Low | InnerTube metadata in `server/server.js`, timeline scrubber | | | **Bluetooth Foot Pedal Shortcuts** | Low | Keydown event listener in `frontend/app.js` | | | **Section Quick-Jump Pad** | Low | Lyric section parser in `frontend/lyrics-core.js` | +| | **Voice Search (SpeechRecognition)** | Low | Web Speech API, search input handler | | **Phase 2: Stage & Audio** | **EBU R128 LUFS Normalization** | Medium | ffmpeg `ebur128` in `server/media-cache.js`, Web Audio GainNode | | | **ChordPro Display & Transpose** | Medium | `frontend/lyrics-core.js`, chord parser | | | **Confidence Monitor Stage View** | Medium | `Presenter` & `fitLyricLines` in `frontend/app.js` | +| | **OBS Lower-Third Overlay** | Low-Med | `server/remote.js` WebSocket relay, transparent route | | | **On-Screen Subtitles (CC)** | Medium | `/api/notes/:id/captions`, overlay renderer | -| **Phase 3: Deep Capabilities** | **Pitch Shift & Tempo Worklet** | High | AudioWorklet / `SoundTouchJS` DSP | -| | **Ambient Glow Lighting** | Low-Med | Offscreen `` + CSS blur | -| | **Smart Autoplay Radio Mix** | Medium | InnerTube `next` API, `frontend/stats-core.js` | +| **Phase 3: Piano & Interactive Learning** | **Interactive Piano Roll & Key Visualizer** | Medium | HTML5 Canvas keyboard renderer, note event synchronizer | +| | **Audio-to-MIDI Transcription** | Medium-High | Spotify Basic Pitch (WASM) / ByteDance transcription worker | +| | **Sheet Music View (VexFlow)** | Medium | VexFlow dynamic import, MusicXML / MIDI generator | +| | **Wait-for-Note Practice (Web MIDI)**| Medium | Web MIDI API (`navigator.requestMIDIAccess`), practice queue | +| **Phase 4: Advanced Engine** | **Audio Stem Separation (Karaoke)** | High | ONNX Runtime Web / server Demucs | +| | **Pitch Shift & Tempo Worklet** | High | AudioWorklet / `SoundTouchJS` DSP | +| | **Document Picture-in-Picture** | Low-Med | Chrome Document PiP API | | | **Collaborative Setlists** | High | WebSocket hub in `server/party.js` |