Add admin analytics, metadata collection, and grouped lyric cues
Queue reviewable Whisper drafts from the song list and lyrics editor. Preserve line breaks within one timed cue across editing, saving, reporting, and service views. Add storage and listening analytics with a durable metadata collector, related-search depth, video limits, thumbnail storage, and a browsable metadata library.
This commit is contained in:
Jonathan Sykes
@@ -79,7 +79,7 @@ services:
|
||||
- "traefik.http.routers.ytplayer-https.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.ytplayer-svc.loadbalancer.server.port=3000"
|
||||
|
||||
# Transcribes saved songs that have no lyrics yet, one at a time, with
|
||||
# Handles requested Whisper drafts and transcribes saved songs without lyrics, with
|
||||
# faster-whisper on CPU (no API keys, no credits). Separate container so it
|
||||
# never competes with playback/downloads: capped CPU + memory, low priority,
|
||||
# and it fetches audio from the ytplayer service over the private network.
|
||||
|
||||
122
docs/admin-lyrics-and-analytics.md
Normal file
122
docs/admin-lyrics-and-analytics.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Lyrics and server analytics
|
||||
|
||||
## Whisper drafts
|
||||
|
||||
In `/admin#editor`, every song has a Whisper button alongside its Open button.
|
||||
The loaded song also has **Transcribe with Whisper**. Both enqueue the same
|
||||
server job; duplicate active requests reuse that job. The existing CPU
|
||||
faster-whisper container polls explicit requests every ten seconds and returns
|
||||
a timed draft even when that song already has lyrics. Manual requests always
|
||||
use Whisper, bypassing LRCLIB.
|
||||
|
||||
**Use transcript in editor** loads the draft for review. **Save lyrics** publishes
|
||||
it through the existing revision history and conflict checks. Until saved,
|
||||
shared lyrics remain unchanged. Applying a draft can be undone, including the
|
||||
original sync offset. Queued/running jobs survive restarts; worker leases expire
|
||||
after two minutes and heartbeat every 25 seconds. After three failed worker
|
||||
attempts the job reports a failure. Completed/failed drafts are retained for
|
||||
30 days; cleanup runs when workers report progress.
|
||||
|
||||
A YouTube video must already have a ready server media copy; uploads are also
|
||||
supported. The existing maximum of one hour and 40 MiB audio input applies.
|
||||
Songs without enough recognized vocals fail visibly. Transcription accuracy
|
||||
and word timing depend on the singing and audio mix; review before saving.
|
||||
|
||||
Deploy the updated server and rebuild the `lyrics-worker` image. Both need the
|
||||
same `LYRICS_WORKER_TOKEN` (at least 24 characters; compose passes it to the
|
||||
worker as `YTP_TOKEN`). A configured token permits queuing; the status endpoint
|
||||
also reports whether a worker has recently polled. No paid transcription API
|
||||
is involved.
|
||||
|
||||
Admin endpoints (admin cookie or API token):
|
||||
|
||||
- `POST /api/admin/transcriptions/:video` queues a draft.
|
||||
- `GET /api/admin/transcriptions/:video` returns the latest draft/progress.
|
||||
|
||||
Worker-only endpoints require the worker token:
|
||||
|
||||
- `POST /api/lyrics-worker/claim` claims one job.
|
||||
- `POST /api/lyrics-worker/jobs/:job` reports progress/results with its lease.
|
||||
|
||||
## Grouped lyric lines
|
||||
|
||||
One lyric cue can contain multiple visual lines with one timestamp. In the
|
||||
admin editor, **Shift+Enter** inserts a tight line break. On phones, select the
|
||||
cue and use **Line break** in its options. Ordinary Enter or **Below** adds a
|
||||
separate cue with its own timing. Both visual lines of a grouped cue highlight
|
||||
and seek together in service mode, with a smaller internal gap than the space
|
||||
between cues. The player panel, stage and presenter views also preserve breaks.
|
||||
|
||||
Text editing/export uses a continuation prefix to preserve the grouping:
|
||||
|
||||
```text
|
||||
[0:12.00] Because You are God
|
||||
| You can do anything
|
||||
[0:18.00] Another separate cue
|
||||
```
|
||||
|
||||
The continuation belongs to the preceding cue. JSON stores its text as
|
||||
`"Because You are God\nYou can do anything"`. Saving, revisions and text
|
||||
round trips retain the single timestamp. Reporting a wrong lyric also preserves the grouped text. Existing single-line lyrics work
|
||||
unchanged.
|
||||
|
||||
## Server analytics and metadata collector
|
||||
|
||||
Open **Stats** (`/admin#analytics`). It shows cached video and upload totals,
|
||||
recorded plays, discovery sources, metadata/thumbnail payload sizes, and
|
||||
available filesystem space for media, uploads and the database. Paths on the
|
||||
same device share their free space; their capacities must not be added.
|
||||
Media totals come from library records and exclude temporary files and
|
||||
filesystem overhead. Metadata bytes exclude indexes and SQLite overhead.
|
||||
|
||||
The collector accepts a search, a total unique-video limit (1–500), and depth:
|
||||
|
||||
- **0:** only the specified search.
|
||||
- **1:** also search channels and tags found in its results.
|
||||
- **2/3:** follow up to two/three related rounds.
|
||||
|
||||
The total limit applies across every round. The collector reserves result
|
||||
budget for deeper rounds, deduplicates video IDs and related queries, and stops
|
||||
after at most 24 searches. Duplicate, unavailable or empty results can produce
|
||||
fewer videos than requested. Up to three collections may be queued/running;
|
||||
one executes at a time per server. Atomic ownership leases prevent two
|
||||
servers from claiming the same job. Progress is saved after each discovery
|
||||
batch and video. An expired interrupted job resumes its pending items.
|
||||
**Stop collection** cancels future work; an extraction already in progress
|
||||
can finish its network request before the cancellation is observed.
|
||||
|
||||
Discovery cards enter the same catalog used for recommendations. Each video
|
||||
is then enriched through yt-dlp without downloading media. The server stores
|
||||
full descriptive fields (including descriptions, dates, engagement counts,
|
||||
language, tags, chapters, thumbnail variants and format specifications) in
|
||||
`video_details`. Expiring media URLs and request headers are excluded. Details
|
||||
are capped at 500 KB per video and follow catalog eviction. Thumbnail image
|
||||
bytes use the existing durable thumbnail queue, host restrictions and budget
|
||||
(default 512 MiB). Failures stay visible in job history; successful cards remain
|
||||
stored even when enrichment fails. Refresh analytics to update the aggregate
|
||||
numbers after collection; job progress polls automatically while this tab is
|
||||
open.
|
||||
|
||||
The metadata library supports title/channel/tag filtering, 50-row pages and a
|
||||
raw descriptive metadata view. Admin cookie or API token is required for:
|
||||
|
||||
- `GET /api/admin/analytics`
|
||||
- `GET /api/admin/metadata?q=...&offset=...`
|
||||
- `GET /api/admin/metadata/:id`
|
||||
- `GET /api/admin/collections`
|
||||
- `POST /api/admin/collections` with `{query,maxVideos,depth}`
|
||||
- `POST /api/admin/collections/:id/cancel`
|
||||
|
||||
## Verification
|
||||
|
||||
Run `bun run test` in `server`, `npm test` at the repo root,
|
||||
`python3 -m unittest discover -s scripts/lyrics -p test_auto_lyrics.py`, and
|
||||
`npx playwright test -c playwright.admin.config.js`. Admin browser fixtures
|
||||
cover Whisper review/save/undo, unsaved edits, failures, mobile actions,
|
||||
grouped cues and service highlighting, and analytics collector controls.
|
||||
|
||||
Design review fixed the primary action’s white-on-gradient contrast by using a
|
||||
solid purple fill, and fixed tablet navigation overflow. No new findings were
|
||||
suppressed. Existing admin play/tap gradient contrast and decorative glow
|
||||
findings remain outside these controls. Real viewport captures passed the
|
||||
mobile dashboard/collector and desktop visual checks.
|
||||
@@ -92,7 +92,7 @@
|
||||
}
|
||||
.btn:hover { background: rgba(255,255,255,.1); }
|
||||
.btn:active { transform: scale(.97); }
|
||||
.btn.pri { border: 0; background: var(--grad); color: #fff; box-shadow: 0 10px 26px -10px var(--a1); }
|
||||
.btn.pri { border: 0; background: #6143cf; color: #fff; box-shadow: none; }
|
||||
.btn.pri:hover { filter: brightness(1.1); }
|
||||
.btn.ok { border: 0; background: var(--ok); color: #06210f; }
|
||||
.btn.dng { color: var(--bad); border-color: rgba(255,107,107,.35); background: rgba(255,107,107,.07); }
|
||||
@@ -164,6 +164,13 @@
|
||||
.song-row small { color: var(--text-dim); font-size: 12px; }
|
||||
.song-row .st { flex: none; font: 700 11px var(--ui); padding: 3px 9px; border-radius: 99px; border: 1px solid var(--line-2); color: var(--text-dim); }
|
||||
.song-row .st.has { color: var(--ok); border-color: rgba(74,222,128,.4); }
|
||||
.song-open { display: flex; align-items: center; gap: 10px; min-width: 0; flex: 1; padding: 0; border: 0; background: none; color: inherit; text-align: left; }
|
||||
.whisper-request { min-height: 44px; flex: none; }
|
||||
.whisper-draft { margin-block: 14px; }
|
||||
.whisper-draft pre { white-space: pre-wrap; overflow-wrap: anywhere; max-height: 180px; overflow: auto; font: 400 14px/1.55 var(--ui); color: var(--text); }
|
||||
#whisperStatus { color: var(--text-dim); font-size: 14px; margin-block: 10px; }
|
||||
.song { flex-wrap: wrap; }
|
||||
@media (max-width: 480px) { .song-row { flex-wrap: wrap; } .song-open { flex-basis: 100%; min-height: 44px; } }
|
||||
.chips { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 12px; }
|
||||
.chip { height: 34px; padding: 0 13px; border-radius: 99px; border: 1px solid var(--line-2); background: rgba(255,255,255,.04); font-size: 13px; font-weight: 600; color: var(--text-2); max-width: 100%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.chip:hover { color: var(--text); border-color: var(--a1); }
|
||||
@@ -211,7 +218,7 @@
|
||||
}
|
||||
.tc:hover { border-color: var(--a2); color: var(--text); }
|
||||
.ln.untimed .tc { color: var(--warn); border-style: dashed; border-color: rgba(255,180,84,.55); }
|
||||
.ln-text { flex: 1 1 auto; min-width: 0; height: 38px; padding: 0 8px; border: 0; border-radius: 9px; background: transparent; font-size: 16px; }
|
||||
.ln-text { flex: 1 1 auto; min-width: 0; min-height: 38px; height: auto; padding: 7px 8px; resize: vertical; overflow: hidden; line-height: 1.3; border: 0; border-radius: 9px; background: transparent; font-size: 16px; }
|
||||
.ln.section .ln-text { font-weight: 800; text-transform: uppercase; letter-spacing: .06em; font-size: 13px; color: var(--a2); }
|
||||
.ln.cue .ln-text { font-style: italic; color: var(--warn); }
|
||||
.ln.now { background: rgba(124,92,255,.16); border-color: rgba(124,92,255,.55); box-shadow: 0 0 0 1px rgba(124,92,255,.25), 0 10px 30px -16px var(--a1); }
|
||||
@@ -257,6 +264,26 @@
|
||||
.tap small { display: block; font-weight: 600; opacity: .8; font-size: 11px; letter-spacing: 0; max-width: 84px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.tap:active { transform: scale(.93); }
|
||||
|
||||
.analytics-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 12px; margin: 20px 0; }
|
||||
.metric { padding: 18px; background: var(--bg-2); border: 1px solid var(--line-2); border-radius: 14px; }
|
||||
.metric b { display: block; font-size: 26px; margin: 4px 0; }
|
||||
.metric small, .metric span { color: var(--text-2); }
|
||||
.analytics-form { display: flex; flex-wrap: wrap; align-items: end; gap: 12px; }
|
||||
.analytics-form label { display: grid; gap: 5px; min-width: 0; }
|
||||
.analytics-form .query { flex: 1 1 250px; }
|
||||
.analytics-form .field { width: 100%; min-width: 0; }
|
||||
.analytics-form button { min-height: 44px; }
|
||||
.analytics-table { overflow-x: auto; margin-top: 12px; }
|
||||
.analytics-table table { width: 100%; }
|
||||
.collection-job { padding: 14px 0; border-bottom: 1px solid var(--line-2); overflow-wrap: anywhere; }
|
||||
.collection-job p { margin: 4px 0; color: var(--text-2); }
|
||||
#metadataDetail { white-space: pre-wrap; overflow-wrap: anywhere; max-height: 400px; overflow: auto; font-size: 13px; }
|
||||
@media (max-width: 480px) { .analytics-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } .metric { padding: 12px; } .metric b { font-size: 22px; } }
|
||||
@media (min-width: 761px) and (max-width: 1100px) {
|
||||
.top { flex-wrap: wrap; }
|
||||
.tabs { order: 2; width: 100%; margin-left: 0; overflow-x: auto; }
|
||||
.tabs .tab { flex: none; }
|
||||
}
|
||||
.nav { display: none; }
|
||||
@media (max-width: 760px) {
|
||||
.tabs, .top .ghost.desk { display: none; }
|
||||
@@ -347,6 +374,7 @@
|
||||
<div class="ring" id="ring"><span id="ringTxt">0/0</span></div>
|
||||
<div style="min-width:0"><h1 id="songTitle"></h1><small id="songMeta"></small></div>
|
||||
<span class="grow"></span>
|
||||
<button class="btn sm whisper-request" id="whisperBtn" type="button">Transcribe with Whisper</button>
|
||||
<button class="ghost" id="closeSong" type="button">Change song</button>
|
||||
</div>
|
||||
|
||||
@@ -386,8 +414,15 @@
|
||||
</div>
|
||||
|
||||
<div id="msg" class="msg"></div>
|
||||
<p id="whisperStatus" role="status" aria-live="polite"></p>
|
||||
<div id="whisperResult" class="card whisper-draft hidden">
|
||||
<h2 style="font-size:18px;margin:0 0 8px">Whisper transcript ready</h2>
|
||||
<p class="muted">Saved lyrics stay unchanged until you review and save this draft.</p>
|
||||
<pre id="whisperPreview"></pre>
|
||||
<button id="whisperUse" class="btn" type="button">Use transcript in editor</button>
|
||||
</div>
|
||||
<div id="lines" class="lines"></div>
|
||||
<textarea id="raw" class="hidden" spellcheck="false"></textarea>
|
||||
<textarea id="raw" aria-label="Lyrics as text; prefix continuation lines with vertical bar and space" class="hidden" spellcheck="false"></textarea>
|
||||
<div class="row" style="margin-top:14px">
|
||||
<button class="btn sm" id="addLine" type="button">＋ Add line</button>
|
||||
<span class="muted" style="font-size:12.5px">Shortcuts: <code>Space</code> stamp (Tap-sync) · <code>S</code> stamp selected · <code>K</code> play · <code>↑↓</code> select · <code>⌘/Ctrl S</code> save</span>
|
||||
@@ -480,6 +515,35 @@
|
||||
</section>
|
||||
|
||||
<!-- ================= ACCESS ================= -->
|
||||
<section id="p-analytics" class="panel">
|
||||
<h1 class="page">Server analytics</h1>
|
||||
<p class="lead">Video storage, listening activity and the metadata library.</p>
|
||||
<button class="btn" type="button" id="analyticsRefresh">Refresh analytics</button>
|
||||
<p id="analyticsStatus" role="status" aria-live="polite"></p>
|
||||
<div class="analytics-grid" id="analyticsMetrics"></div>
|
||||
<div class="card"><h2>Remaining disk space</h2><div id="analyticsVolumes"></div><p class="muted">Volumes on the same device share the reported free space. Stored media totals exclude temporary files and filesystem overhead.</p></div>
|
||||
<div class="card"><h2>Metadata collector</h2>
|
||||
<p class="muted">Search and save descriptive video metadata and thumbnails. Video and audio files are not downloaded.</p>
|
||||
<form id="collectorForm" class="analytics-form">
|
||||
<label class="query">Search<input id="collectorQuery" class="field" type="search" maxlength="200" required placeholder="Cathedral of Praise Worship" /></label>
|
||||
<label>Total video limit<input id="collectorLimit" class="field" type="number" min="1" max="500" value="30" required /></label>
|
||||
<label>Related-search depth<select id="collectorDepth" class="field"><option value="0">0 · Search only</option><option value="1">1 · Channels and topics</option><option value="2">2 · Two related rounds</option><option value="3">3 · Three related rounds</option></select></label>
|
||||
<button class="btn pri" type="submit" id="collectorStart">Collect metadata</button>
|
||||
</form>
|
||||
<p class="muted">The limit applies across every round. Depth follows channels and tags found in video metadata, with at most 24 searches. A collection can return fewer videos when results repeat or sources are unavailable. Thumbnails follow the server’s storage budget.</p>
|
||||
<p id="collectorStatus" role="status" aria-live="polite"></p><div id="collectionJobs"></div>
|
||||
</div>
|
||||
<div class="card"><h2>Metadata library</h2>
|
||||
<form id="metadataFilterForm" class="analytics-form"><label class="query">Filter saved metadata<input id="metadataFilter" class="field" type="search" maxlength="200" placeholder="Title, channel or tag" /></label><button class="btn" type="submit">Filter</button></form>
|
||||
<p id="metadataCount" class="muted"></p>
|
||||
<div class="analytics-table"><table id="metadataTable"></table></div>
|
||||
<div class="row"><button id="metadataPrev" class="btn" type="button">Previous</button><button id="metadataNext" class="btn" type="button">Next</button></div>
|
||||
<details id="metadataDetails" class="hidden"><summary>Video metadata</summary><pre id="metadataDetail"></pre></details>
|
||||
</div>
|
||||
<div class="card"><h2>Discovery sources</h2><div class="analytics-table"><table id="analyticsSources"></table></div></div>
|
||||
<div class="card"><h2>Most played</h2><div id="analyticsTop"></div></div>
|
||||
</section>
|
||||
|
||||
<section id="p-access" class="panel">
|
||||
<h1 class="page">API access</h1>
|
||||
<p class="lead">Tokens let scripts write shared lyrics and chapters. A token is shown once — store it in your secrets vault.</p>
|
||||
@@ -505,6 +569,7 @@
|
||||
<nav class="nav" id="tabsBottom" role="tablist"></nav>
|
||||
</div>
|
||||
|
||||
<script src="lyrics-core.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
'use strict';
|
||||
@@ -537,9 +602,10 @@
|
||||
library: '<path d="M9 18V6l10-2v12"/><circle cx="6.5" cy="18" r="2.5"/><circle cx="16.5" cy="16" r="2.5"/>',
|
||||
peers: '<circle cx="6" cy="12" r="2.5"/><circle cx="18" cy="6" r="2.5"/><circle cx="18" cy="18" r="2.5"/><path d="M8.3 10.9l7.4-3.8M8.3 13.1l7.4 3.8"/>',
|
||||
history: '<path d="M3 12a9 9 0 109-9 9 9 0 00-6.4 2.6L3 8"/><path d="M3 3v5h5M12 7v5l3 2"/>',
|
||||
analytics: '<path d="M4 20h16M7 16V9M12 16V4M17 16v-5"/>',
|
||||
access: '<circle cx="8" cy="15" r="4"/><path d="M11 12l9-9M16 7l3 3M14 9l2 2"/>',
|
||||
};
|
||||
const TABS = [['editor', 'Editor'], ['reports', 'Reports'], ['library', 'Library'], ['peers', 'P2P'], ['history', 'History'], ['access', 'API']];
|
||||
const TABS = [['editor', 'Editor'], ['reports', 'Reports'], ['library', 'Library'], ['peers', 'P2P'], ['history', 'History'], ['analytics', 'Stats'], ['access', 'API']];
|
||||
let tab = 'editor';
|
||||
function buildTabs() {
|
||||
const html = TABS.map(([k, l]) => `<button class="tab" role="tab" data-tab="${k}" aria-selected="false"><svg viewBox="0 0 24 24">${ICONS[k]}</svg><span>${l}</span>${k === 'reports' ? '<span class="badge hidden" data-badge>0</span>' : ''}</button>`).join('');
|
||||
@@ -558,6 +624,7 @@
|
||||
if (name === 'peers') loadP2p();
|
||||
if (name === 'library') loadUploads();
|
||||
if (name === 'access') loadTokens();
|
||||
if (name === 'analytics') loadAnalytics();
|
||||
window.scrollTo({ top: 0 });
|
||||
}
|
||||
function setBadge(n) {
|
||||
@@ -596,6 +663,67 @@
|
||||
});
|
||||
$('logoutBtn').addEventListener('click', async () => { await api('/api/admin/logout', { method: 'POST' }); showLogin(); });
|
||||
|
||||
/* ---------------- server analytics / metadata collection ---------------- */
|
||||
let analyticsTimer = null, metadataOffset = 0;
|
||||
const bytes = value => { let size = Number(value) || 0, unit = 0; const units = ['B','KB','MB','GB','TB']; while (size >= 1024 && unit < 4) { size /= 1024; unit++; } return `${size.toFixed(unit ? 1 : 0)} ${units[unit]}`; };
|
||||
async function loadAnalytics() {
|
||||
$('analyticsStatus').textContent = 'Loading server analytics…';
|
||||
try {
|
||||
const j = await api('/api/admin/analytics');
|
||||
if (!j.ok) throw Error(j.error || 'Analytics unavailable.');
|
||||
const ready = j.media.find(row => row.status === 'ready') || {}, uploadBytes = j.uploads.reduce((total, row) => total + Number(row.bytes), 0);
|
||||
const metrics = [
|
||||
['Stored media', bytes(Number(ready.bytes || 0) + uploadBytes), `${ready.count || 0} cached videos · ${j.uploads.reduce((sum, row) => sum + Number(row.count), 0)} uploads`],
|
||||
['Metadata videos', Number(j.catalog.videos).toLocaleString(), `${j.details.videos} with detailed extractor metadata`],
|
||||
['Saved thumbnails', Number(j.thumbnails.saved || 0).toLocaleString(), `${bytes(j.thumbnails.bytes)} · ${j.thumbnails.pending || 0} awaiting download`],
|
||||
['Recorded plays', Number(j.listening.plays).toLocaleString(), `${j.listening.videos} different videos`],
|
||||
['Cached video size', bytes(ready.bytes), `${Math.round(Number(ready.seconds || 0) / 3600)} hours of audio/video`],
|
||||
['Upload size', bytes(uploadBytes), 'Server media library'],
|
||||
['Metadata size', bytes(Number(j.catalog.bytes) + Number(j.details.bytes)), 'JSON payloads; database indexes excluded'],
|
||||
['Cache queue', j.media.filter(row => row.status !== 'ready').reduce((sum, row) => sum + Number(row.count), 0), j.media.filter(row => row.status !== 'ready').map(row => `${row.status}: ${row.count}`).join(' · ') || 'No pending downloads'],
|
||||
];
|
||||
$('analyticsMetrics').innerHTML = metrics.map(([label, value, detail]) => `<div class="metric"><span>${esc(label)}</span><b>${esc(value)}</b><small>${esc(detail)}</small></div>`).join('');
|
||||
$('analyticsVolumes').innerHTML = j.volumes.map(v => `<p><b>${esc(v.label)}</b> · ${v.unavailable ? 'Storage unavailable' : `${bytes(v.free)} available of ${bytes(v.total)} · device ${esc(v.device)}`}</p>`).join('');
|
||||
$('analyticsSources').innerHTML = '<tr><th>Source</th><th>Videos</th><th>Discoveries</th></tr>' + j.sources.map(row => `<tr><td>${esc(row.source)}</td><td>${esc(row.videos)}</td><td>${esc(row.discoveries)}</td></tr>`).join('');
|
||||
$('analyticsTop').innerHTML = j.topPlayed.length ? j.topPlayed.map(row => `<p>${esc(row.title)} · <b>${esc(row.plays)} plays</b></p>`).join('') : '<p class="muted">No listening activity recorded yet.</p>';
|
||||
$('analyticsStatus').textContent = 'Updated ' + new Date().toLocaleTimeString();
|
||||
await Promise.all([loadCollections(), loadMetadata()]);
|
||||
} catch (error) { $('analyticsStatus').textContent = error.message || 'Could not load server analytics.'; }
|
||||
}
|
||||
async function loadCollections() {
|
||||
try {
|
||||
const j = await api('/api/admin/collections'); if (!j.ok) return;
|
||||
$('collectionJobs').innerHTML = j.jobs.length ? j.jobs.map(job => `<div class="collection-job"><b>${esc(job.query)}</b> · ${esc(job.status)}<p>${job.collected}/${job.maxVideos} videos collected · ${job.enriched} enriched · ${job.failed} failed · ${job.searches} searches · depth ${job.depth}</p>${job.current ? `<p>Working on ${esc(job.current)}</p>` : ''}${job.error ? `<p>${esc(job.error)}</p>` : ''}${job.errors?.length ? `<details><summary>Extraction errors</summary>${job.errors.map(e => `<p>${esc(e.id)}: ${esc(e.error)}</p>`).join('')}</details>` : ''}${['queued','running'].includes(job.status) ? `<button class="btn" type="button" data-cancel-collection="${esc(job.id)}">Stop collection</button>` : ''}</div>`).join('') : '<p class="muted">No collections yet.</p>';
|
||||
clearTimeout(analyticsTimer); analyticsTimer = null;
|
||||
if (tab === 'analytics' && j.jobs.some(job => ['queued','running'].includes(job.status))) analyticsTimer = setTimeout(loadCollections, 4000);
|
||||
} catch { $('collectorStatus').textContent = 'Connection interrupted. Collection continues on the server.'; }
|
||||
}
|
||||
async function loadMetadata() {
|
||||
const j = await api(`/api/admin/metadata?q=${encodeURIComponent($('metadataFilter').value)}&offset=${metadataOffset}`); if (!j.ok) return;
|
||||
$('metadataCount').textContent = `${j.total} saved videos · ${j.total ? metadataOffset + 1 : 0}–${metadataOffset + j.videos.length}`;
|
||||
$('metadataTable').innerHTML = '<tr><th>Video</th><th>Metadata</th><th>Thumbnail</th></tr>' + j.videos.map(v => `<tr><td><button class="btn sm" type="button" data-metadata-id="${esc(v.id)}">${esc(v.title)}</button><br><small>${esc(v.channel)} · ${Number(v.duration) > 0 ? Math.round(Number(v.duration)) + 's' : 'Duration unavailable'}</small></td><td>${v.enriched ? 'Detailed' : 'Search card'}</td><td>${v.thumbnailBytes ? bytes(v.thumbnailBytes) : 'Pending / unavailable'}</td></tr>`).join('');
|
||||
$('metadataPrev').disabled = metadataOffset === 0; $('metadataNext').disabled = metadataOffset + 50 >= j.total;
|
||||
}
|
||||
$('analyticsRefresh').addEventListener('click', loadAnalytics);
|
||||
$('collectorForm').addEventListener('submit', async e => {
|
||||
e.preventDefault(); $('collectorStart').disabled = true; $('collectorStatus').textContent = 'Queuing collection…';
|
||||
try {
|
||||
const j = await api('/api/admin/collections', { method: 'POST', body: { query: $('collectorQuery').value, maxVideos: Number($('collectorLimit').value), depth: Number($('collectorDepth').value) } });
|
||||
$('collectorStatus').textContent = j.ok ? 'Collection queued. You can leave this page while it runs.' : j.error || 'Could not start collection.';
|
||||
if (j.ok) await loadCollections();
|
||||
} catch { $('collectorStatus').textContent = 'Could not reach the server.'; }
|
||||
finally { $('collectorStart').disabled = false; }
|
||||
});
|
||||
$('collectionJobs').addEventListener('click', async e => { const btn = e.target.closest('[data-cancel-collection]'); if (!btn) return; await api('/api/admin/collections/' + btn.dataset.cancelCollection + '/cancel', { method: 'POST', body: {} }); await loadCollections(); });
|
||||
$('metadataFilterForm').addEventListener('submit', e => { e.preventDefault(); metadataOffset = 0; loadMetadata(); });
|
||||
$('metadataPrev').addEventListener('click', () => { metadataOffset = Math.max(0, metadataOffset - 50); loadMetadata(); });
|
||||
$('metadataNext').addEventListener('click', () => { metadataOffset += 50; loadMetadata(); });
|
||||
$('metadataTable').addEventListener('click', async e => {
|
||||
const btn = e.target.closest('[data-metadata-id]'); if (!btn) return;
|
||||
const j = await api('/api/admin/metadata/' + btn.dataset.metadataId); if (!j.ok) return;
|
||||
$('metadataDetail').textContent = JSON.stringify(j.metadata, null, 2); $('metadataDetails').classList.remove('hidden'); $('metadataDetails').open = true;
|
||||
});
|
||||
|
||||
/* ---------------- tokens / inject ---------------- */
|
||||
async function loadTokens() {
|
||||
const j = await api('/api/admin/tokens');
|
||||
@@ -776,13 +904,18 @@
|
||||
function paintSongs() {
|
||||
const q = $('songFilter').value.trim().toLowerCase();
|
||||
const rows = SONGS.filter((r) => !q || (r.title + ' ' + (r.sub || '')).toLowerCase().includes(q));
|
||||
$('songList').innerHTML = rows.length ? rows.map((r) => `<button type="button" class="song-row" data-open="${esc(r.id)}">
|
||||
$('songList').innerHTML = rows.length ? rows.map((r) => `<div class="song-row"><button type="button" class="song-open" data-open="${esc(r.id)}">
|
||||
<span style="min-width:0;flex:1"><b>${esc(r.title)}</b><small>${esc(r.sub || '')}</small></span>
|
||||
<span class="st ${r.lines > 0 ? 'has' : ''}">${r.lines > 0 ? `${r.lines} lines` : r.lines === 0 ? 'no lyrics' : 'open'}</span></button>`).join('')
|
||||
<span class="st ${r.lines > 0 ? 'has' : ''}">${r.lines > 0 ? `${r.lines} lines` : r.lines === 0 ? 'no lyrics' : 'open'}</span></button>
|
||||
<button type="button" class="btn sm whisper-request" data-whisper="${esc(r.id)}" aria-label="Transcribe ${esc(r.title)} with Whisper">Whisper</button></div>`).join('')
|
||||
: '<div class="muted" style="padding:10px">Nothing matches — paste a link above to open any YouTube song.</div>';
|
||||
paintWhisper();
|
||||
}
|
||||
$('songFilter').addEventListener('input', paintSongs);
|
||||
$('songList').addEventListener('click', (e) => { const b = e.target.closest('[data-open]'); if (b) leOpen(b.dataset.open); });
|
||||
$('songList').addEventListener('click', (e) => {
|
||||
const whisper = e.target.closest('[data-whisper]'); if (whisper) { startWhisper(whisper.dataset.whisper); return; }
|
||||
const b = e.target.closest('[data-open]'); if (b) leOpen(b.dataset.open);
|
||||
});
|
||||
$('recentChips').addEventListener('click', (e) => { const b = e.target.closest('[data-open]'); if (b) leOpen(b.dataset.open); });
|
||||
|
||||
/* ---------------- p2p ---------------- */
|
||||
@@ -907,6 +1040,67 @@
|
||||
$('save').disabled = !on;
|
||||
$('discard').disabled = !on;
|
||||
}
|
||||
|
||||
const whisperJobs = new Map(), whisperStarting = new Set();
|
||||
let whisperTimer = null;
|
||||
const activeWhisper = job => job && (job.status === 'queued' || job.status === 'running');
|
||||
function paintWhisper() {
|
||||
document.querySelectorAll('[data-whisper]').forEach(btn => {
|
||||
const job = whisperJobs.get(btn.dataset.whisper);
|
||||
btn.disabled = whisperStarting.has(btn.dataset.whisper) || activeWhisper(job);
|
||||
btn.textContent = job?.status === 'running' ? 'Transcribing…' : job?.status === 'queued' ? 'Queued' : 'Whisper';
|
||||
});
|
||||
const job = whisperJobs.get(LE.id);
|
||||
$('whisperBtn').disabled = whisperStarting.has(LE.id) || activeWhisper(job);
|
||||
$('whisperBtn').textContent = activeWhisper(job) ? (job.status === 'queued' ? 'Whisper queued' : 'Whisper transcribing…') : 'Transcribe with Whisper';
|
||||
const labels = { 'loading-model': 'Whisper is loading its model…', 'downloading-audio': 'Whisper is reading the song’s audio…', transcribing: 'Whisper is transcribing the vocals. This can take several minutes.' };
|
||||
$('whisperStatus').textContent = job?.status === 'queued' ? 'Queued for Whisper. Waiting for the lyrics worker.'
|
||||
: job?.status === 'running' ? labels[job.stage] || 'Whisper is working…'
|
||||
: job?.status === 'failed' ? 'Whisper could not finish: ' + (job.error || 'Try again.') : '';
|
||||
const ready = job?.status === 'complete' && job.result?.lines?.length;
|
||||
$('whisperResult').classList.toggle('hidden', !ready);
|
||||
$('whisperPreview').textContent = ready ? job.result.lines.map(l => l.text).join('\n') : '';
|
||||
}
|
||||
async function refreshWhisper(id) {
|
||||
try {
|
||||
const result = await api('/api/admin/transcriptions/' + encodeURIComponent(id));
|
||||
if (result.ok) whisperJobs.set(id, result.job);
|
||||
paintWhisper(); scheduleWhisper();
|
||||
} catch { /* a reconnect or another poll can recover the durable job */ }
|
||||
}
|
||||
function scheduleWhisper() {
|
||||
if (whisperTimer || ![...whisperJobs.values()].some(activeWhisper)) return;
|
||||
whisperTimer = setTimeout(async () => {
|
||||
whisperTimer = null;
|
||||
for (const [id, job] of whisperJobs) if (activeWhisper(job)) await refreshWhisper(id);
|
||||
scheduleWhisper();
|
||||
}, 5000);
|
||||
}
|
||||
async function startWhisper(id) {
|
||||
if (!id || whisperStarting.has(id) || activeWhisper(whisperJobs.get(id))) return;
|
||||
whisperStarting.add(id); paintWhisper();
|
||||
try {
|
||||
if (LE.id !== id || !LE.loaded) { const opened = await leOpen(id); if (opened === false) return; }
|
||||
const result = await api('/api/admin/transcriptions/' + encodeURIComponent(id), { method: 'POST', body: {} });
|
||||
if (!result.ok) { say(result.error || 'Whisper could not start.', 'err'); return; }
|
||||
whisperJobs.set(id, result.job); paintWhisper(); scheduleWhisper();
|
||||
} catch { say('Could not reach Whisper. Try again when connected.', 'err'); }
|
||||
finally { whisperStarting.delete(id); paintWhisper(); }
|
||||
}
|
||||
$('whisperBtn').addEventListener('click', () => startWhisper(LE.id));
|
||||
$('whisperUse').addEventListener('click', () => {
|
||||
const job = whisperJobs.get(LE.id); if (job?.status !== 'complete' || !job.result?.lines?.length) return;
|
||||
if (LE.dirty && !confirm('Replace your unsaved editor text with the Whisper draft? Undo can restore it.')) return;
|
||||
edit(() => {
|
||||
LE.lines = job.result.lines.map(l => ({ ...l })); LE.offset = 0;
|
||||
LE.tags = [...LE.tags.filter(t => !/^(from |auto-transcribed)/i.test(t)), ...job.result.tags];
|
||||
LE.raw = false; $('raw').classList.add('hidden'); $('lines').classList.remove('hidden');
|
||||
$('rawBtn').textContent = '✎ Edit as text'; $('tags').value = LE.tags.join(', ');
|
||||
});
|
||||
render(); draw(); syncBars();
|
||||
$('whisperResult').classList.add('hidden');
|
||||
say('Whisper draft loaded. Check the words and timing, then Save to publish; Undo restores the previous text.', 'ok');
|
||||
});
|
||||
function progress() {
|
||||
const timed = LE.lines.filter((l) => l.kind === 'line' && l.t != null).length;
|
||||
const all = LE.lines.filter((l) => l.kind === 'line').length;
|
||||
@@ -919,10 +1113,10 @@
|
||||
}
|
||||
|
||||
/* undo / redo — whole-document snapshots, typing is coalesced */
|
||||
const snap = () => JSON.stringify({ l: LE.lines, t: LE.tags });
|
||||
const snap = () => JSON.stringify({ l: LE.lines, t: LE.tags, o: LE.offset });
|
||||
function pushUndo() { LE.undo.push(snap()); if (LE.undo.length > 100) LE.undo.shift(); LE.redo = []; undoBtns(); }
|
||||
function undoBtns() { $('bUndo').disabled = !LE.undo.length; $('bRedo').disabled = !LE.redo.length; }
|
||||
function restore(s) { const o = JSON.parse(s); LE.lines = o.l; LE.tags = o.t; $('tags').value = LE.tags.join(', '); render(); setDirty(true); undoBtns(); }
|
||||
function restore(s) { const o = JSON.parse(s); LE.lines = o.l; LE.tags = o.t; LE.offset = o.o ?? LE.offset; $('tags').value = LE.tags.join(', '); render(); setDirty(true); undoBtns(); }
|
||||
function doUndo() { if (!LE.undo.length) return; LE.redo.push(snap()); restore(LE.undo.pop()); say('Undone'); }
|
||||
function doRedo() { if (!LE.redo.length) return; LE.undo.push(snap()); restore(LE.redo.pop()); say('Redone'); }
|
||||
let typing = null;
|
||||
@@ -957,6 +1151,7 @@
|
||||
$('raw').classList.add('hidden'); $('lines').classList.remove('hidden'); $('rawBtn').textContent = '✎ Edit as text';
|
||||
await loadFlags();
|
||||
render(); setDirty(false); undoBtns(); syncBars();
|
||||
paintWhisper(); refreshWhisper(id);
|
||||
say('');
|
||||
if (!keepAudio) initAudio(id, streams);
|
||||
draw();
|
||||
@@ -993,7 +1188,7 @@
|
||||
row.innerHTML = `
|
||||
<div class="ln-main">
|
||||
<button class="tc" data-act="chip" type="button"></button>
|
||||
<input class="ln-text" value="${esc(line.text)}" placeholder="Type the line…" aria-label="Line text" />
|
||||
<textarea class="ln-text" rows="1" placeholder="Type the line… (Shift+Enter for a line break)" aria-label="Line text">${esc(line.text)}</textarea>
|
||||
<span class="flagchip hidden" data-flag><svg viewBox="0 0 24 24"><path d="M5 21V4M5 4h11l-2 4 2 4H5"/></svg><b></b></span>
|
||||
<button class="more" data-act="more" type="button" aria-label="Line options">⋯</button>
|
||||
</div>
|
||||
@@ -1005,9 +1200,12 @@
|
||||
<button class="btn" data-act="nudge" data-d="-0.2" type="button">−0.2</button>
|
||||
<button class="btn" data-act="nudge" data-d="0.2" type="button">+0.2</button>
|
||||
<select class="mini" data-kind aria-label="Line type"><option value="line">Line</option><option value="section">Section</option><option value="cue">Cue</option></select>
|
||||
<button class="btn" data-act="break" type="button" title="Same timestamp, both lines highlight together">↵ Line break</button>
|
||||
<button class="btn" data-act="add" type="button">＋ Below</button>
|
||||
<button class="btn dng" data-act="del" type="button">Delete</button>
|
||||
</div>`;
|
||||
const text = row.querySelector('.ln-text');
|
||||
text.rows = Math.max(1, line.text.split('\n').length);
|
||||
row.querySelector('[data-kind]').value = line.kind;
|
||||
return row;
|
||||
}
|
||||
@@ -1036,6 +1234,9 @@
|
||||
LE.rows = LE.lines.map((l, i) => { const r = rowEl(l, i); box.appendChild(r); return r; });
|
||||
if (!LE.lines.length) box.innerHTML = '<div class="empty">No lines yet — press “＋ Add line”, paste text with ✎ Edit as text, or pull them from LRCLIB (⋯ tools).</div>';
|
||||
LE.lines.forEach((_, i) => paintRow(i));
|
||||
const fields = LE.rows.map(row => row.querySelector('.ln-text'));
|
||||
const heights = fields.map(field => field.scrollHeight);
|
||||
fields.forEach((field, i) => { field.style.height = heights[i] + 'px'; });
|
||||
paintFlags(); progress(); lastIdx = -2; tick(); tapLabel();
|
||||
$('songMeta').textContent = $('songMeta').textContent.replace(/\d+ lines/, `${LE.lines.length} lines`);
|
||||
}
|
||||
@@ -1109,6 +1310,13 @@
|
||||
else if (act === 'set') stamp(i);
|
||||
else if (act === 'seek') { seek((LE.lines[i].t || 0) + LE.offset); if (LE.audio && LE.audio.paused) LE.audio.play().catch(() => {}); }
|
||||
else if (act === 'nudge') { if (LE.lines[i].t == null) return; edit(() => { LE.lines[i].t = r2(Math.max(0, LE.lines[i].t + Number(btn.dataset.d))); }); paintRow(i); draw(); }
|
||||
else if (act === 'break') {
|
||||
const text = LE.rows[i].querySelector('.ln-text');
|
||||
const pos = text.selectionStart;
|
||||
edit(() => { LE.lines[i].text = text.value.slice(0, pos) + '\n' + text.value.slice(text.selectionEnd); });
|
||||
render(); select(i);
|
||||
const target = LE.rows[i].querySelector('.ln-text'); target.focus(); target.setSelectionRange(pos + 1, pos + 1);
|
||||
}
|
||||
else if (act === 'add') { edit(() => LE.lines.splice(i + 1, 0, { t: null, text: '', kind: 'line' })); render(); select(i + 1); LE.rows[i + 1].querySelector('.ln-text').focus(); }
|
||||
else if (act === 'del') { edit(() => LE.lines.splice(i, 1)); LE.sel = -1; render(); draw(); }
|
||||
else if (act === 'resolve') {
|
||||
@@ -1118,7 +1326,7 @@
|
||||
});
|
||||
$('lines').addEventListener('input', (e) => {
|
||||
const row = e.target.closest('.ln'); if (!row) return; const i = Number(row.dataset.i);
|
||||
if (e.target.classList.contains('ln-text')) { typed(); LE.lines[i].text = e.target.value; setDirty(true); tapLabel(); }
|
||||
if (e.target.classList.contains('ln-text')) { typed(); LE.lines[i].text = e.target.value; e.target.rows = Math.max(1, e.target.value.split('\n').length); e.target.style.height = 'auto'; e.target.style.height = e.target.scrollHeight + 'px'; setDirty(true); tapLabel(); }
|
||||
});
|
||||
$('lines').addEventListener('change', (e) => {
|
||||
const row = e.target.closest('.ln'); if (!row) return; const i = Number(row.dataset.i);
|
||||
@@ -1128,6 +1336,7 @@
|
||||
$('lines').addEventListener('focusin', (e) => { const row = e.target.closest('.ln'); if (row && Number(row.dataset.i) !== LE.sel) select(Number(row.dataset.i)); });
|
||||
$('lines').addEventListener('keydown', (e) => {
|
||||
if (e.key !== 'Enter' || !e.target.classList.contains('ln-text')) return;
|
||||
if (e.shiftKey) return; // Native textarea break stays in the current timed cue.
|
||||
e.preventDefault();
|
||||
const i = Number(e.target.closest('.ln').dataset.i);
|
||||
edit(() => LE.lines.splice(i + 1, 0, { t: null, text: '', kind: LE.lines[i].kind === 'section' ? 'line' : LE.lines[i].kind }));
|
||||
@@ -1199,17 +1408,10 @@
|
||||
$('raw').classList.toggle('hidden', !LE.raw); $('lines').classList.toggle('hidden', LE.raw);
|
||||
$('rawBtn').textContent = LE.raw ? '≡ Back to lines' : '✎ Edit as text';
|
||||
if (LE.raw) {
|
||||
$('raw').value = LE.lines.map((l) => (l.t == null ? '' : `[${fmtT(l.t)}] `) + (l.kind === 'section' ? '# ' : l.kind === 'cue' ? '! ' : '') + l.text).join('\n');
|
||||
$('raw').value = LyricsCore.toText({ lines: LE.lines });
|
||||
} else {
|
||||
pushUndo();
|
||||
LE.lines = $('raw').value.split('\n').map((raw) => {
|
||||
let rest = raw.trim(), t = null;
|
||||
const m = rest.match(/^\[(\d{1,3}):(\d{1,2}(?:[.:]\d{1,3})?)\]\s*/);
|
||||
if (m) { t = r2(Number(m[1]) * 60 + Number(m[2].replace(':', '.'))); rest = rest.slice(m[0].length); }
|
||||
let kind = 'line';
|
||||
if (rest.startsWith('# ')) { kind = 'section'; rest = rest.slice(2); } else if (rest.startsWith('! ')) { kind = 'cue'; rest = rest.slice(2); }
|
||||
return { t, text: rest, kind };
|
||||
}).filter((l) => l.text);
|
||||
LE.lines = LyricsCore.fromText($('raw').value).lines;
|
||||
setDirty(true); render(); draw();
|
||||
}
|
||||
syncBars();
|
||||
@@ -1242,12 +1444,12 @@
|
||||
leOpen(id);
|
||||
});
|
||||
function leOpen(id, focusText) {
|
||||
if (LE.dirty && id !== LE.id && !confirm('Discard the unsaved changes to the song you have open?')) return;
|
||||
if (LE.dirty && id !== LE.id && !confirm('Discard the unsaved changes to the song you have open?')) return false;
|
||||
showTab('editor');
|
||||
if (id === LE.id && LE.loaded && focusText) {
|
||||
const i = LE.lines.findIndex((l) => l.text === focusText); if (i >= 0) select(i, true, true); return;
|
||||
}
|
||||
leLoad(id, false, focusText);
|
||||
return leLoad(id, false, focusText);
|
||||
}
|
||||
|
||||
function renderDocs() {
|
||||
|
||||
@@ -3247,7 +3247,12 @@ function fitLyricLines(list, { maxShare = 0.17, maxPx = 160, minPx = 18, uniform
|
||||
// re-laid-out every line (O(n²) — seconds on a phone for a long sermon).
|
||||
const fits = [];
|
||||
for (const el of rows) {
|
||||
widthAt.text = el.textContent;
|
||||
// Size a grouped cue by its widest visual line, preserving tight breaks.
|
||||
const parts = el.textContent.split('\n');
|
||||
widthAt.text = parts.reduce((widest, part) => {
|
||||
widthAt.text = part; const width = widthAt(100);
|
||||
return width > widest.width ? { text: part, width } : widest;
|
||||
}, { text: '', width: -1 }).text;
|
||||
const w100 = widthAt(100);
|
||||
let px = cap;
|
||||
if (w100) {
|
||||
@@ -3650,16 +3655,19 @@ const Notes = (() => {
|
||||
body.innerHTML = `
|
||||
<p class="lyr-editor-help">One lyric line per line. Optional: <code>[1:23.45]</code> times a line
|
||||
(or time them later with Tap-sync), <code># Chorus</code> is a section, <code>! Drums in</code> a band cue,
|
||||
<code>@ Key G, Capo 2, 70 BPM</code> adds tags. Pasted .lrc files work as-is.</p>
|
||||
<code>@ Key G, Capo 2, 70 BPM</code> adds tags. Use Shift+Enter or Line break to group words under the same timestamp (<code>| continuation</code> in text). Pasted .lrc files work as-is.</p>
|
||||
${note ? `<p class="lyr-editor-note">${escapeHtml(note)}</p>` : ''}
|
||||
<textarea class="lyr-editor-text" spellcheck="false" placeholder="Paste or type the lyrics…"></textarea>
|
||||
<div class="lyr-editor-tools">
|
||||
<button type="button" class="np-btn" data-act="captions">⤓ Import YouTube captions</button>
|
||||
<button type="button" class="np-btn" data-act="break">↵ Line break (same time)</button>
|
||||
<button type="button" class="np-btn" data-act="stamp">⏱ Stamp current time on this line</button>
|
||||
</div>`;
|
||||
const ta = body.querySelector('textarea');
|
||||
ta.value = prefill != null ? prefill : (st.lyrics ? LC.toText(st.lyrics.data) : '');
|
||||
ta.addEventListener('keydown', (e) => e.stopPropagation());
|
||||
const insertBreak = () => { ta.setRangeText('\n| ', ta.selectionStart, ta.selectionEnd, 'end'); ta.focus(); };
|
||||
body.querySelector('[data-act="break"]').addEventListener('click', insertBreak);
|
||||
ta.addEventListener('keydown', (e) => { e.stopPropagation(); if (e.key === 'Enter' && e.shiftKey) { e.preventDefault(); insertBreak(); } });
|
||||
const runCaptions = async () => {
|
||||
const text = await captionsText();
|
||||
if (text != null) ta.value = text;
|
||||
@@ -3667,7 +3675,8 @@ const Notes = (() => {
|
||||
body.querySelector('[data-act="captions"]').addEventListener('click', runCaptions);
|
||||
body.querySelector('[data-act="stamp"]').addEventListener('click', () => {
|
||||
const pos = ta.selectionStart;
|
||||
const start = ta.value.lastIndexOf('\n', pos - 1) + 1;
|
||||
let start = ta.value.lastIndexOf('\n', pos - 1) + 1;
|
||||
while (start > 0 && /^\s*\| /.test(ta.value.slice(start).split('\n')[0])) start = ta.value.lastIndexOf('\n', start - 2) + 1;
|
||||
const endNl = ta.value.indexOf('\n', start);
|
||||
const end = endNl < 0 ? ta.value.length : endNl;
|
||||
const lineText = ta.value.slice(start, end).replace(/^(\[[^\]]*\]\s*)+/, '');
|
||||
@@ -3675,7 +3684,11 @@ const Notes = (() => {
|
||||
ta.value = ta.value.slice(0, start) + stamp + lineText + ta.value.slice(end);
|
||||
// Move to the next line so repeated taps walk down the sheet.
|
||||
const next = ta.value.indexOf('\n', start);
|
||||
const caret = next < 0 ? ta.value.length : next + 1;
|
||||
let caret = next < 0 ? ta.value.length : next + 1;
|
||||
while (/^\s*\| /.test(ta.value.slice(caret).split('\n')[0])) {
|
||||
const following = ta.value.indexOf('\n', caret);
|
||||
caret = following < 0 ? ta.value.length : following + 1;
|
||||
}
|
||||
ta.focus();
|
||||
ta.setSelectionRange(caret, caret);
|
||||
});
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
* @ Key G, Capo 2 (Play F), 70 BPM tags
|
||||
* [00:12.00][01:30.00] repeated line LRC multi-stamp
|
||||
* [offset:+500] LRC offset (ms, + = sooner)
|
||||
* | You can do anything continuation, same cue
|
||||
* Other LRC headers ([ar:], [ti:], …) are ignored.
|
||||
*
|
||||
* Sync offset: a line is shown at (t + offset) seconds of playback, so a
|
||||
@@ -60,9 +61,15 @@
|
||||
const lines = [];
|
||||
const tags = [];
|
||||
let offset = null;
|
||||
let group = [];
|
||||
for (const rawLine of String(text || '').replace(/\r/g, '').split('\n')) {
|
||||
let rest = rawLine.trim();
|
||||
if (!rest) continue;
|
||||
if (rest.startsWith('| ') && group.length) {
|
||||
for (const line of group) line.text += '\n' + rest.slice(2).trim();
|
||||
continue;
|
||||
}
|
||||
group = [];
|
||||
const header = rest.match(HEADER_RE);
|
||||
if (header && !/^\d/.test(header[1])) {
|
||||
if (header[1].toLowerCase() === 'offset') {
|
||||
@@ -83,8 +90,9 @@
|
||||
if (rest.startsWith('#')) { kind = 'section'; rest = rest.replace(/^#+/, '').trim(); }
|
||||
else if (rest.startsWith('!')) { kind = 'cue'; rest = rest.slice(1).trim(); }
|
||||
if (!rest) continue;
|
||||
if (!stamps.length) lines.push({ t: null, text: rest, kind });
|
||||
for (const t of stamps) lines.push({ t: Math.round(t * 100) / 100, text: rest, kind });
|
||||
group = stamps.length ? stamps.map(t => ({ t: Math.round(t * 100) / 100, text: rest, kind }))
|
||||
: [{ t: null, text: rest, kind }];
|
||||
lines.push(...group);
|
||||
}
|
||||
// A fully timed sheet (typical .lrc with repeated-chorus stamps) is put in
|
||||
// time order; a partly timed one keeps the order it was written in.
|
||||
@@ -100,7 +108,8 @@
|
||||
for (const l of (doc && doc.lines) || []) {
|
||||
const stamp = l.t === null || l.t === undefined ? '' : `[${fmtStamp(l.t)}] `;
|
||||
const mark = l.kind === 'section' ? '# ' : l.kind === 'cue' ? '! ' : '';
|
||||
out.push(stamp + mark + l.text);
|
||||
const parts = String(l.text || '').split('\n');
|
||||
out.push(stamp + mark + parts[0], ...parts.slice(1).map(part => '| ' + part));
|
||||
}
|
||||
return out.join('\n');
|
||||
}
|
||||
|
||||
@@ -78,3 +78,16 @@ test('YouTube export adds a 0:00 entry only when needed', () => {
|
||||
assert.strictEqual(LC.activeMarker(items, 60).title, 'Intro');
|
||||
assert.strictEqual(LC.activeMarker(items, 10), null);
|
||||
});
|
||||
|
||||
|
||||
test('multiline cues round-trip without adding timestamps or active rows', () => {
|
||||
const doc = { lines: [{ t: 12, text: 'Because You are God\nYou can do anything', kind: 'line' }, { t: 18, text: 'Next lyric', kind: 'line' }], tags: [] };
|
||||
const copy = LC.fromText(LC.toText(doc));
|
||||
assert.deepStrictEqual(copy.lines, doc.lines);
|
||||
assert.strictEqual(LC.activeIndex(copy.lines, 15, 0), 0);
|
||||
assert.strictEqual(LC.activeIndex(copy.lines, 18, 0), 1);
|
||||
assert.strictEqual(LC.timedCount(copy.lines), 2);
|
||||
const repeated = LC.fromText('[0:12][0:30] Because You are God\n| You can do anything');
|
||||
assert.strictEqual(repeated.lines.length, 2);
|
||||
assert.strictEqual(repeated.lines[1].text, doc.lines[0].text);
|
||||
});
|
||||
|
||||
@@ -3522,7 +3522,7 @@ html[data-layout="bento-hub"] .mini-bar-inner {
|
||||
.lyr-line:hover { background: var(--bg-3); }
|
||||
.lyr-line.untimed { cursor: default; }
|
||||
.lyr-time { flex: none; width: 38px; font-family: var(--mono); font-size: 10px; color: var(--text-dim); }
|
||||
.lyr-text { font-size: 15px; line-height: 1.45; }
|
||||
.lyr-text { white-space: pre-line; font-size: 15px; line-height: 1.45; }
|
||||
.lyr-line.active { background: var(--bg-3); color: var(--text); }
|
||||
.lyr-line.active .lyr-text { font-weight: 700; font-size: 17px; }
|
||||
.lyr-line.active .lyr-time { color: var(--accent-bright); }
|
||||
@@ -3779,6 +3779,8 @@ body.remote-open { overflow: hidden; }
|
||||
font-weight: 800;
|
||||
line-height: 1.15;
|
||||
letter-spacing: -0.01em;
|
||||
white-space: pre-line;
|
||||
flex-shrink: 0;
|
||||
/* --fit = the size at which THIS line fills one row; others are fractions. */
|
||||
font-size: calc(var(--fit, 40px) * 0.48);
|
||||
color: var(--text-2);
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
"tauri": "tauri",
|
||||
"tauri:dev": "tauri dev",
|
||||
"tauri:build": "tauri build",
|
||||
"test": "node --test frontend/"
|
||||
"test": "node --test frontend/*.test.js"
|
||||
},
|
||||
"author": "",
|
||||
"license": "MIT",
|
||||
|
||||
3
playwright.admin.config.js
Normal file
3
playwright.admin.config.js
Normal file
@@ -0,0 +1,3 @@
|
||||
const { defineConfig } = require('@playwright/test');
|
||||
const classic = require('./playwright.classic.config');
|
||||
module.exports = defineConfig({ ...classic, testMatch: /admin-lyrics\.spec\.js/ });
|
||||
@@ -29,6 +29,7 @@ import urllib.error
|
||||
import urllib.request
|
||||
import glob
|
||||
import http.cookiejar
|
||||
import threading
|
||||
|
||||
FILLER = re.compile(r"^(?:(?:oh|ooh|ohh|oh-oh|ah|ahh|hey|yeah|mm|mm-mm|mm-mm-mm|hmm|whoa|woah|la|na|uh|come on)[\s,.!?-]*)+$", re.I)
|
||||
KEEP_CAP = {'I', "I'm", "I'll", "I've", "I'd", 'You', 'Your', "You're", 'Yours', 'Lord', 'God', 'Jesus', 'Christ',
|
||||
@@ -101,7 +102,7 @@ class Api:
|
||||
data=json.dumps(body).encode() if body is not None else None)
|
||||
try:
|
||||
with self.opener.open(req, timeout=600) as r:
|
||||
data = r.read()
|
||||
data = r.read(MAX_AUDIO_BYTES + 1) if raw else r.read()
|
||||
return r.status, data if raw else json.loads(data or b'{}')
|
||||
except urllib.error.HTTPError as e:
|
||||
data = e.read()
|
||||
@@ -193,9 +194,29 @@ def watch(args):
|
||||
'error': 'worker died while transcribing (likely out of memory)'}
|
||||
print(f'lyrics worker: {vid} crashed the previous run — backing off', flush=True)
|
||||
save_state(args.state, state)
|
||||
next_auto = 0
|
||||
|
||||
def get_model():
|
||||
nonlocal model
|
||||
if model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
print(f'lyrics worker: loading {args.model}', flush=True)
|
||||
model = WhisperModel(args.model, device='cpu', compute_type='int8', cpu_threads=args.threads)
|
||||
return model
|
||||
|
||||
while True:
|
||||
try:
|
||||
api = Api(args.base, token, password)
|
||||
# Explicit admin requests take priority and always use Whisper.
|
||||
# Poll every ten seconds, independently of the automatic interval.
|
||||
if token:
|
||||
st, request = api.call('POST', '/api/lyrics-worker/claim', {})
|
||||
if st == 200 and request.get('job'):
|
||||
process_requested(args, api, request['job'], get_model)
|
||||
continue
|
||||
if time.time() < next_auto:
|
||||
time.sleep(min(args.watch, 10))
|
||||
continue
|
||||
st, r = api.call('GET', '/api/admin/media')
|
||||
if st != 200:
|
||||
raise RuntimeError(f'listing failed ({st}): {r.get("error")}')
|
||||
@@ -205,10 +226,7 @@ def watch(args):
|
||||
and state.get(m['id'], {}).get('status') != 'instrumental'
|
||||
and state.get(m['id'], {}).get('retry_at', 0) <= now]
|
||||
if todo:
|
||||
if model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
print(f'lyrics worker: loading {args.model}', flush=True)
|
||||
model = WhisperModel(args.model, device='cpu', compute_type='int8', cpu_threads=args.threads)
|
||||
get_model()
|
||||
m = todo[0] # one song per cycle keeps the worker's footprint small
|
||||
entry = state.get(m['id'], {})
|
||||
state[m['id']] = {**entry, 'status': 'in_progress'}
|
||||
@@ -224,9 +242,58 @@ def watch(args):
|
||||
save_state(args.state, state)
|
||||
print(f'lyrics worker: {m["id"]} ({m.get("title", "")[:60]}): {result}', flush=True)
|
||||
continue # straight on to the next song
|
||||
next_auto = time.time() + args.watch
|
||||
except Exception as e: # never die: the next cycle retries
|
||||
print(f'lyrics worker: {e}', flush=True)
|
||||
time.sleep(args.watch)
|
||||
next_auto = time.time() + args.watch
|
||||
time.sleep(min(args.watch, 10))
|
||||
|
||||
|
||||
def process_requested(args, api, job, get_model):
|
||||
"""Lease a manual request, heartbeat through model loading, return a draft."""
|
||||
stopped = threading.Event()
|
||||
stage = ['loading-model']
|
||||
path = f'/api/lyrics-worker/jobs/{job["id"]}'
|
||||
|
||||
def report(extra=None):
|
||||
return api.call('POST', path, {'lease': job['lease'], 'stage': stage[0], **(extra or {})})
|
||||
|
||||
def heartbeat():
|
||||
while not stopped.wait(25):
|
||||
try:
|
||||
st, _ = report()
|
||||
if st == 409:
|
||||
stopped.set()
|
||||
except Exception:
|
||||
pass # transient connection failure; the durable lease handles recovery
|
||||
|
||||
thread = threading.Thread(target=heartbeat, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
model = get_model()
|
||||
def progress(value):
|
||||
stage[0] = value
|
||||
st, _ = report()
|
||||
if st == 409:
|
||||
raise RuntimeError('The transcription lease expired.')
|
||||
result = transcribe_one(args, api, model, job['videoId'], draft_job=job, on_stage=progress)
|
||||
stopped.set()
|
||||
thread.join(timeout=2)
|
||||
payload = {'status': 'complete', 'result': result} if isinstance(result, dict) else {'status': 'failed', 'error': result}
|
||||
st, response = report(payload)
|
||||
if st == 400 and payload['status'] == 'complete':
|
||||
report({'status': 'failed', 'error': response.get('error', 'The transcript was rejected.')[:300]})
|
||||
if st != 200:
|
||||
print(f'lyrics worker: request {job["id"]} could not finish ({st}): {response.get("error", "failed")}', flush=True)
|
||||
except Exception as e:
|
||||
stopped.set()
|
||||
try:
|
||||
report({'status': 'failed', 'error': str(e)[:300]})
|
||||
except Exception:
|
||||
pass # the next worker can reclaim an expired request
|
||||
finally:
|
||||
stopped.set()
|
||||
thread.join(timeout=2)
|
||||
|
||||
|
||||
def run_once(args, api):
|
||||
@@ -247,20 +314,22 @@ def run_once(args, api):
|
||||
print('\n'.join(f'{v} {s}' for v, s in summary))
|
||||
|
||||
|
||||
def transcribe_one(args, api, model, vid):
|
||||
def transcribe_one(args, api, model, vid, draft_job=None, on_stage=None):
|
||||
"""Give one saved song lyrics. Published (often synced) lyrics from LRCLIB
|
||||
beat a machine transcript, so that is tried first; transcription is the
|
||||
fallback. Returns a one-line result."""
|
||||
st, cur = api.call('GET', f'/api/notes/{vid}')
|
||||
live = (cur or {}).get('lyrics') if st == 200 else None
|
||||
if live and live['data']['lines'] and not args.overwrite:
|
||||
if not draft_job and live and live['data']['lines'] and not args.overwrite:
|
||||
return 'skip: has lyrics'
|
||||
if not getattr(args, 'no_web', False):
|
||||
if not draft_job and not getattr(args, 'no_web', False):
|
||||
st, r = api.call('POST', f'/api/notes/{vid}/lyrics/web', {'overwrite': bool(args.overwrite)})
|
||||
if st == 200:
|
||||
m = r.get('match') or {}
|
||||
return f"saved rev {r.get('rev')} — LRCLIB {'synced' if r.get('synced') else 'plain'}: {m.get('artist', '')} – {m.get('track', '')}"
|
||||
st, audio = api.call('GET', f'/api/media/{vid}?a=1', raw=True)
|
||||
if on_stage:
|
||||
on_stage('downloading-audio')
|
||||
st, audio = api.call('GET', draft_job['audioPath'] if draft_job else f'/api/media/{vid}?a=1', raw=True)
|
||||
if st != 200:
|
||||
return f'no cached audio ({st})'
|
||||
if len(audio) > MAX_AUDIO_BYTES:
|
||||
@@ -269,6 +338,8 @@ def transcribe_one(args, api, model, vid):
|
||||
f.write(audio)
|
||||
f.flush()
|
||||
t0 = time.time()
|
||||
if on_stage:
|
||||
on_stage('transcribing')
|
||||
# vad_filter must stay OFF: it classifies sung music as non-speech
|
||||
# and silently drops the whole song.
|
||||
segs, info = model.transcribe(f.name, language=args.language, word_timestamps=True, vad_filter=False,
|
||||
@@ -282,6 +353,8 @@ def transcribe_one(args, api, model, vid):
|
||||
doc = {'lines': lines, 'tags': ['auto-transcribed (whisper)'], 'offset': 0}
|
||||
head = ' / '.join(l['text'] for l in lines[:3])
|
||||
print(f'{vid}: {len(lines)} lines, lang={info.language}, {took:.0f}s for {info.duration:.0f}s audio | {head[:100]}', flush=True)
|
||||
if draft_job:
|
||||
return doc
|
||||
if args.dry_run:
|
||||
print(json.dumps(doc, ensure_ascii=False)[:2000])
|
||||
return f'dry-run {len(lines)} lines'
|
||||
|
||||
66
scripts/lyrics/test_auto_lyrics.py
Normal file
66
scripts/lyrics/test_auto_lyrics.py
Normal file
@@ -0,0 +1,66 @@
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import types
|
||||
import unittest
|
||||
|
||||
spec = importlib.util.spec_from_file_location('auto_lyrics', pathlib.Path(__file__).with_name('auto_lyrics.py'))
|
||||
worker = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(worker)
|
||||
|
||||
class FakeApi:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
def call(self, method, path, body=None, raw=False):
|
||||
self.calls.append((method, path, body))
|
||||
if raw:
|
||||
return 200, b'fixture audio'
|
||||
if path.startswith('/api/notes/'):
|
||||
return 200, {'lyrics': {'rev': 5, 'data': {'lines': [{'t': 0, 'text': 'Keep the original'}]}}}
|
||||
return 200, {'ok': True}
|
||||
|
||||
class FakeModel:
|
||||
def transcribe(self, path, **options):
|
||||
assert options['vad_filter'] is False
|
||||
words = [types.SimpleNamespace(word=word, start=i, end=i + .8) for i, word in enumerate('Because You are God You can do anything'.split())]
|
||||
return [types.SimpleNamespace(words=words)], types.SimpleNamespace(language='en', duration=10)
|
||||
|
||||
class WhisperRequests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.args = types.SimpleNamespace(overwrite=False, no_web=False, language=None, min_words=1, dry_run=False)
|
||||
self.api = FakeApi()
|
||||
self.job = {'id': 'fixture-job', 'videoId': '0gfX0dFLaBc', 'lease': 'fixture-lease', 'audioPath': '/api/media/0gfX0dFLaBc?a=1'}
|
||||
|
||||
def test_manual_request_uses_whisper_despite_existing_lyrics_and_never_publishes(self):
|
||||
result = worker.transcribe_one(self.args, self.api, FakeModel(), self.job['videoId'], draft_job=self.job)
|
||||
self.assertIsInstance(result, dict)
|
||||
self.assertTrue(result['lines'])
|
||||
self.assertFalse(any(method == 'PUT' or path.endswith('/lyrics/web') for method, path, body in self.api.calls))
|
||||
|
||||
def test_requested_job_reports_stages_and_completes_a_draft(self):
|
||||
worker.process_requested(self.args, self.api, self.job, lambda: FakeModel())
|
||||
posts = [body for method, path, body in self.api.calls if path.startswith('/api/lyrics-worker/jobs/')]
|
||||
self.assertEqual(posts[0]['stage'], 'downloading-audio')
|
||||
self.assertEqual(posts[1]['stage'], 'transcribing')
|
||||
self.assertEqual(posts[-1]['status'], 'complete')
|
||||
self.assertEqual(posts[-1]['lease'], 'fixture-lease')
|
||||
|
||||
def test_rejected_draft_is_reported_failed_instead_of_retrying_forever(self):
|
||||
original = self.api.call
|
||||
def reject(method, path, body=None, raw=False):
|
||||
if body and body.get('status') == 'complete':
|
||||
return 400, {'error': 'too many lines'}
|
||||
return original(method, path, body, raw)
|
||||
self.api.call = reject
|
||||
worker.process_requested(self.args, self.api, self.job, lambda: FakeModel())
|
||||
self.assertEqual(self.api.calls[-1][2]['status'], 'failed')
|
||||
self.assertEqual(self.api.calls[-1][2]['error'], 'too many lines')
|
||||
|
||||
def test_no_vocals_fails_without_touching_saved_lyrics(self):
|
||||
self.args.min_words = 25
|
||||
worker.process_requested(self.args, self.api, self.job, lambda: FakeModel())
|
||||
last = self.api.calls[-1][2]
|
||||
self.assertEqual(last['status'], 'failed')
|
||||
self.assertFalse(any(method == 'PUT' for method, path, body in self.api.calls))
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
149
server/admin-analytics.js
Normal file
149
server/admin-analytics.js
Normal file
@@ -0,0 +1,149 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { statfs, stat } from 'node:fs/promises';
|
||||
import { db } from './db.js';
|
||||
import { ingest } from './video-catalog.js';
|
||||
|
||||
const LEASE = 120000;
|
||||
const n = v => Number(v) || 0;
|
||||
const clean = value => String(value || '').trim().slice(0, 200);
|
||||
const parse = value => { try { return JSON.parse(value) || {}; } catch { return {}; } };
|
||||
function publicJob(row) {
|
||||
const state = parse(row.state);
|
||||
return { id: row.id, query: row.query, maxVideos: n(row.max_videos), depth: n(row.depth), status: row.status,
|
||||
collected: state.ids?.length || 0, enriched: state.enriched || 0, failed: state.failed || 0,
|
||||
searches: state.searches || 0, current: state.current || '', error: row.error, errors: state.errors || [],
|
||||
createdAt: n(row.created_at), updatedAt: n(row.updated_at) };
|
||||
}
|
||||
export function descriptiveMetadata(raw) {
|
||||
const out = {};
|
||||
for (const key of ['id','title','fulltitle','description','channel','channel_id','channel_url','uploader','uploader_id','uploader_url','upload_date','release_date','timestamp','duration','view_count','like_count','comment_count','tags','categories','language','live_status','availability','age_limit','license','chapters','thumbnails','thumbnail','webpage_url','artist','artists','album','track','release_year']) {
|
||||
if (raw[key] != null) out[key] = raw[key];
|
||||
}
|
||||
// Retain format specs, excluding expiring media URLs and request headers.
|
||||
if (Array.isArray(raw.formats)) out.formats = raw.formats.map(f => Object.fromEntries(['format_id','format_note','ext','width','height','fps','vcodec','acodec','abr','tbr','filesize','filesize_approx','audio_channels','asr'].filter(k => f[k] != null).map(k => [k, f[k]])));
|
||||
if (JSON.stringify(out).length > 500000) throw Error('Extracted metadata exceeds 500 KB.');
|
||||
return out;
|
||||
}
|
||||
function card(raw) {
|
||||
return { ...raw, channel: raw.channel || raw.uploader || '', channelId: raw.channel_id,
|
||||
channelUrl: raw.channel_url, thumbnail: raw.thumbnail || raw.thumbnails?.at(-1)?.url };
|
||||
}
|
||||
export async function storageAnalytics(paths = []) {
|
||||
const queries = [
|
||||
"SELECT status,COUNT(*) AS count,COALESCE(SUM(size),0) AS bytes,COALESCE(SUM(duration),0) AS seconds FROM media_cache GROUP BY status",
|
||||
"SELECT kind,COUNT(*) AS count,COALESCE(SUM(size),0) AS bytes FROM uploads GROUP BY kind",
|
||||
];
|
||||
// Keep each aggregate on its own table, so thumbnail bytes and plays never multiply.
|
||||
const [media, uploads, catalog, thumbs, details, sources, plays, top] = await Promise.all([
|
||||
db.execute(queries[0]), db.execute(queries[1]),
|
||||
db.execute('SELECT COUNT(*) AS videos,COALESCE(SUM(length(CAST(card AS BLOB))),0) AS bytes,MAX(updated_at) AS lastSeen FROM video_meta'),
|
||||
db.execute('SELECT COUNT(*) AS total,SUM(CASE WHEN data IS NOT NULL THEN 1 ELSE 0 END) AS saved,COALESCE(SUM(size),0) AS bytes,SUM(CASE WHEN data IS NULL AND retry_at<9007199254740991 THEN 1 ELSE 0 END) AS pending FROM video_thumbnails'),
|
||||
db.execute('SELECT COUNT(*) AS videos,COALESCE(SUM(length(CAST(metadata AS BLOB))),0) AS bytes FROM video_details'),
|
||||
db.execute('SELECT source,COUNT(*) AS videos,SUM(discoveries) AS discoveries,MAX(last_seen) AS lastSeen FROM video_meta_sources GROUP BY source ORDER BY videos DESC'),
|
||||
db.execute('SELECT COALESCE(SUM(plays),0) AS plays,COUNT(DISTINCT video_id) AS videos FROM listening_daily'),
|
||||
db.execute('SELECT l.video_id,SUM(l.plays) AS plays,m.card FROM listening_daily l LEFT JOIN video_meta m ON m.id=l.video_id GROUP BY l.video_id ORDER BY plays DESC LIMIT 10'),
|
||||
]);
|
||||
const volumes = await Promise.all(paths.map(async ({ label, path }) => {
|
||||
try { const [fs, file] = await Promise.all([statfs(path), stat(path)]); return { label, device: String(file.dev), total: n(fs.blocks) * n(fs.bsize), free: n(fs.bavail) * n(fs.bsize) }; }
|
||||
catch { return { label, unavailable: true }; }
|
||||
}));
|
||||
return { media: media.rows, uploads: uploads.rows, catalog: catalog.rows[0], thumbnails: thumbs.rows[0], details: details.rows[0], sources: sources.rows, listening: plays.rows[0], topPlayed: top.rows.map(r => ({ id: r.video_id, plays: n(r.plays), title: parse(r.card).title || r.video_id })), volumes };
|
||||
}
|
||||
|
||||
export function registerAnalyticsRoutes(app, { adminAuth, runYtdlp, paths = [], autoStart = true }) {
|
||||
let running = null;
|
||||
const owner = randomUUID();
|
||||
async function runNext() {
|
||||
if (running) return running;
|
||||
running = execute().finally(() => { running = null; });
|
||||
return running;
|
||||
}
|
||||
async function execute() {
|
||||
const time = Date.now();
|
||||
const job = (await db.execute({ sql: `UPDATE metadata_collections SET status='running',owner=?,lease_until=?,updated_at=?
|
||||
WHERE id=(SELECT id FROM metadata_collections WHERE status='queued' OR (status='running' AND lease_until<?) ORDER BY created_at LIMIT 1) RETURNING *`, args: [owner, time + LEASE, time, time] })).rows[0];
|
||||
if (!job) return false;
|
||||
const state = { ids: [], queries: [{ q: job.query, level: 0 }], pending: [], enriched: 0, failed: 0, searches: 0, errors: [], ...parse(job.state) };
|
||||
const seen = new Set(state.ids);
|
||||
async function checkpoint(status = 'running', error = null) {
|
||||
const result = await db.execute({ sql: "UPDATE metadata_collections SET state=?,status=?,error=?,lease_until=?,updated_at=? WHERE id=? AND owner=? AND status='running'", args: [JSON.stringify(state), status, error, Date.now() + LEASE, Date.now(), job.id, owner] });
|
||||
if (!result.rowsAffected) { const error = Error('Collection cancelled or reassigned.'); error.name = 'CollectionStopped'; throw error; }
|
||||
}
|
||||
try {
|
||||
while (state.pending.length || (state.queries.length && seen.size < job.max_videos && state.searches < 24)) {
|
||||
if (!state.pending.length) {
|
||||
const { q, level } = state.queries[0]; state.current = q; await checkpoint();
|
||||
const remaining = job.max_videos - seen.size;
|
||||
const count = Math.max(1, Math.ceil(remaining / (job.depth - level + 1)));
|
||||
const output = await runYtdlp([`ytsearch${count}:${q}`, '--dump-json', '--flat-playlist', '--skip-download', '--no-warnings', '--ignore-errors'], { signal: AbortSignal.timeout(60000) });
|
||||
await checkpoint(); // Recheck cancellation/ownership before storing discoveries.
|
||||
const unique = new Map();
|
||||
for (const raw of output.split('\n').filter(Boolean).map(parse)) {
|
||||
if (/^[\w-]{11}$/.test(raw.id) && !seen.has(raw.id)) unique.set(raw.id, raw);
|
||||
}
|
||||
const results = [...unique.values()].slice(0, remaining);
|
||||
await ingest(results.map(card), 'collector');
|
||||
for (const raw of results) { if (seen.has(raw.id)) continue; seen.add(raw.id); state.ids.push(raw.id); state.pending.push({ id: raw.id, level }); }
|
||||
state.queries.shift(); state.searches++; await checkpoint();
|
||||
continue;
|
||||
}
|
||||
const item = state.pending[0]; state.current = item.id; await checkpoint();
|
||||
try {
|
||||
const raw = JSON.parse(await runYtdlp(['-J', '--skip-download', '--no-playlist', '--no-warnings', `https://www.youtube.com/watch?v=${item.id}`], { signal: AbortSignal.timeout(60000) }));
|
||||
if (raw.id !== item.id) throw Error('Extractor returned a different video.');
|
||||
await checkpoint();
|
||||
const metadata = descriptiveMetadata(raw);
|
||||
await ingest([card(raw)], 'collector');
|
||||
await db.execute({ sql: 'INSERT INTO video_details (video_id,metadata,updated_at) VALUES (?,?,?) ON CONFLICT(video_id) DO UPDATE SET metadata=excluded.metadata,updated_at=excluded.updated_at', args: [item.id, JSON.stringify(metadata), Date.now()] });
|
||||
state.enriched++;
|
||||
if (item.level < job.depth && state.queries.length < 24) {
|
||||
for (const q of [raw.channel || raw.uploader, ...(raw.tags || []).slice(0, 2)]) {
|
||||
if (state.queries.length >= 24) break;
|
||||
const term = clean(q); if (!term) continue;
|
||||
state.visited ??= [job.query.toLowerCase()];
|
||||
if (state.visited.includes(term.toLowerCase())) continue;
|
||||
state.visited.push(term.toLowerCase()); state.queries.push({ q: term, level: item.level + 1 });
|
||||
}
|
||||
}
|
||||
} catch (error) { if (error.name === 'CollectionStopped') throw error; state.failed++; if (state.errors.length < 20) state.errors.push({ id: item.id, error: String(error.message).slice(0, 200) }); }
|
||||
state.pending.shift(); await checkpoint();
|
||||
}
|
||||
state.current = ''; await checkpoint('complete');
|
||||
} catch (error) {
|
||||
if (error.name === 'CollectionStopped') return true;
|
||||
await db.execute({ sql: "UPDATE metadata_collections SET status='failed',state=?,error=?,updated_at=? WHERE id=? AND owner=? AND status='running'", args: [JSON.stringify(state), String(error.message).slice(0, 300), Date.now(), job.id, owner] });
|
||||
}
|
||||
return true;
|
||||
}
|
||||
const kick = () => runNext().catch(error => console.error('[metadata collector]', error.message));
|
||||
if (autoStart) { const timer = setInterval(kick, 10000); timer.unref?.(); kick(); }
|
||||
app.get('/api/admin/analytics', adminAuth, async c => c.json({ ok: true, ...await storageAnalytics(paths) }, 200, { 'Cache-Control': 'no-store' }));
|
||||
app.get('/api/admin/metadata', adminAuth, async c => {
|
||||
const query = clean(c.req.query('q')), offset = Math.max(0, Math.min(500000, parseInt(c.req.query('offset')) || 0));
|
||||
const where = query ? 'WHERE m.hay LIKE ?' : '', args = query ? ['%' + query.replace(/[\\%_]/g, '\\$&') + '%'] : [];
|
||||
const filter = where ? where + " ESCAPE '\\'" : '';
|
||||
const rows = (await db.execute({ sql: `SELECT m.id,m.card,m.updated_at,d.updated_at AS enriched,t.size AS thumbnailBytes FROM video_meta m LEFT JOIN video_details d ON d.video_id=m.id LEFT JOIN video_thumbnails t ON t.video_id=m.id ${filter} ORDER BY m.updated_at DESC LIMIT 50 OFFSET ?`, args: [...args, offset] })).rows;
|
||||
const total = n((await db.execute({ sql: `SELECT COUNT(*) AS n FROM video_meta m ${filter}`, args })).rows[0].n);
|
||||
return c.json({ ok: true, total, offset, videos: rows.map(r => ({ ...parse(r.card), updatedAt: n(r.updated_at), enriched: !!r.enriched, thumbnailBytes: n(r.thumbnailBytes) })) });
|
||||
});
|
||||
app.get('/api/admin/metadata/:id', adminAuth, async c => {
|
||||
const row = (await db.execute({ sql: 'SELECT m.card,d.metadata FROM video_meta m LEFT JOIN video_details d ON d.video_id=m.id WHERE m.id=?', args: [c.req.param('id')] })).rows[0];
|
||||
return row ? c.json({ ok: true, metadata: row.metadata ? parse(row.metadata) : parse(row.card) }) : c.json({ ok: false, error: 'Not found.' }, 404);
|
||||
});
|
||||
app.get('/api/admin/collections', adminAuth, async c => c.json({ ok: true, jobs: (await db.execute('SELECT * FROM metadata_collections ORDER BY created_at DESC LIMIT 30')).rows.map(publicJob) }));
|
||||
app.post('/api/admin/collections', adminAuth, async c => {
|
||||
let body; try { body = await c.req.json(); } catch { return c.json({ ok: false, error: 'Invalid JSON.' }, 400); }
|
||||
const query = clean(body?.query), limit = Number(body?.maxVideos), depth = Number(body?.depth);
|
||||
if (!query || !Number.isInteger(limit) || limit < 1 || limit > 500 || !Number.isInteger(depth) || depth < 0 || depth > 3) return c.json({ ok: false, error: 'Enter a search, 1–500 videos and depth 0–3.' }, 400);
|
||||
const time = Date.now(), id = randomUUID();
|
||||
const result = await db.execute({ sql: "INSERT INTO metadata_collections (id,query,max_videos,depth,created_at,updated_at) SELECT ?,?,?,?,?,? WHERE (SELECT COUNT(*) FROM metadata_collections WHERE status IN ('queued','running'))<3", args: [id, query, limit, depth, time, time] });
|
||||
if (!result.rowsAffected) return c.json({ ok: false, error: 'Three collections are already active.' }, 429);
|
||||
if (autoStart) kick();
|
||||
return c.json({ ok: true, id }, 202);
|
||||
});
|
||||
app.post('/api/admin/collections/:id/cancel', adminAuth, async c => {
|
||||
const result = await db.execute({ sql: "UPDATE metadata_collections SET status='cancelled',updated_at=? WHERE id=? AND status IN ('queued','running')", args: [Date.now(), c.req.param('id')] });
|
||||
return c.json({ ok: !!result.rowsAffected });
|
||||
});
|
||||
return { runNext };
|
||||
}
|
||||
83
server/admin-analytics.test.js
Normal file
83
server/admin-analytics.test.js
Normal file
@@ -0,0 +1,83 @@
|
||||
import { test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { Hono } from 'hono';
|
||||
const root = mkdtempSync(join(tmpdir(), 'ytp-analytics-'));
|
||||
process.env.DB_PATH = join(root, 'test.db');
|
||||
const { db, initDb, upsertMedia } = await import('./db.js');
|
||||
const { registerAnalyticsRoutes, storageAnalytics } = await import('./admin-analytics.js');
|
||||
const catalog = await import('./video-catalog.js');
|
||||
const originalFetch = globalThis.fetch;
|
||||
globalThis.fetch = async () => new Response(new Uint8Array([1,2,3]), { headers: { 'Content-Type': 'image/jpeg' } });
|
||||
let app, runner, calls = [], serial = 0;
|
||||
const adminAuth = async (c, next) => c.req.header('x-test-admin') === 'yes' ? next() : c.json({ ok: false }, 401);
|
||||
const get = path => app.request(path, { headers: { 'x-test-admin': 'yes' } });
|
||||
const post = (path, body = {}) => app.request(path, { method: 'POST', headers: { 'x-test-admin': 'yes', 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const fakeExtractor = async args => {
|
||||
calls.push(args);
|
||||
if (args[0].startsWith('ytsearch')) return Array.from({ length: Number(args[0].match(/^ytsearch(\d+)/)[1]) }, () => ({ id: String(++serial).padStart(11, '0'), title: 'Worship ' + serial, channel: 'Channel ' + serial })).map(JSON.stringify).join('\n');
|
||||
const id = args.at(-1).split('v=')[1];
|
||||
return JSON.stringify({ id, title: 'Enriched ' + id, channel: 'Related ' + id, tags: ['Topic ' + id], duration: 60, description: 'Full description '.repeat(200), view_count: 1234, formats: [{ format_id: '140', acodec: 'aac', url: 'https://temporary.example', http_headers: { Cookie: 'omitted' } }] });
|
||||
};
|
||||
beforeAll(async () => { await initDb(); app = new Hono(); runner = registerAnalyticsRoutes(app, { adminAuth, runYtdlp: fakeExtractor, autoStart: false, paths: [{ label: 'DB', path: root }] }); });
|
||||
afterAll(async () => { await catalog.drainThumbnails(); globalThis.fetch = originalFetch; db.close(); rmSync(root, { recursive: true, force: true }); });
|
||||
test('admin routes reject unauthenticated requests and validate collector limits', async () => {
|
||||
for (const path of ['/api/admin/analytics','/api/admin/metadata','/api/admin/metadata/00000000001','/api/admin/collections']) expect((await app.request(path)).status).toBe(401);
|
||||
expect((await app.request('/api/admin/collections', { method: 'POST' })).status).toBe(401);
|
||||
for (const body of [{ query: '', maxVideos: 5, depth: 0 }, { query: 'worship', maxVideos: 501, depth: 0 }, { query: 'worship', maxVideos: 5, depth: 4 }]) expect((await post('/api/admin/collections', body)).status).toBe(400);
|
||||
for (let i = 0; i < 3; i++) expect((await post('/api/admin/collections', { query: 'queue ' + i, maxVideos: 1, depth: 0 })).status).toBe(202);
|
||||
expect((await post('/api/admin/collections', { query: 'queue 4', maxVideos: 1, depth: 0 })).status).toBe(429);
|
||||
for (const job of (await (await get('/api/admin/collections')).json()).jobs) await post('/api/admin/collections/' + job.id + '/cancel');
|
||||
});
|
||||
test('depth zero stores full descriptive metadata and actual thumbnail bytes', async () => {
|
||||
calls = []; await post('/api/admin/collections', { query: 'root worship', maxVideos: 4, depth: 0 }); await runner.runNext(); await catalog.drainThumbnails();
|
||||
const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.query === 'root worship');
|
||||
expect(job.status).toBe('complete'); expect(job.collected).toBe(4); expect(job.enriched).toBe(4); expect(calls.filter(a => a[0].startsWith('ytsearch'))).toHaveLength(1);
|
||||
const videos = (await (await get('/api/admin/metadata')).json()).videos;
|
||||
expect(videos).toHaveLength(4); expect(videos.every(v => v.enriched && v.thumbnailBytes === 3)).toBe(true);
|
||||
const detail = (await (await get('/api/admin/metadata/' + videos[0].id)).json()).metadata;
|
||||
expect(detail.description.length).toBeGreaterThan(1200); expect(detail.view_count).toBe(1234); expect(detail.formats[0].url).toBeUndefined(); expect(detail.formats[0].http_headers).toBeUndefined();
|
||||
});
|
||||
test('related depth follows channels/topics but caps unique videos across all searches', async () => {
|
||||
calls = []; await post('/api/admin/collections', { query: 'deep worship', maxVideos: 10, depth: 2 }); await runner.runNext();
|
||||
const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.query === 'deep worship');
|
||||
expect(job.status).toBe('complete'); expect(job.collected).toBe(10); expect(job.enriched).toBe(10);
|
||||
const searches = calls.filter(a => a[0].startsWith('ytsearch')); expect(searches.length).toBeGreaterThan(1); expect(searches[1][0]).toContain('Related'); expect(searches.length).toBeLessThanOrEqual(24);
|
||||
});
|
||||
test('expired jobs resume pending videos without repeating their search', async () => {
|
||||
await db.execute({ sql: "INSERT INTO metadata_collections (id,query,max_videos,depth,status,state,lease_until,created_at,updated_at) VALUES ('restart','resume',1,0,'running',?,0,0,0)", args: [JSON.stringify({ ids: ['99999999999'], queries: [], pending: [{ id: '99999999999', level: 0 }] })] });
|
||||
calls = []; await runner.runNext(); const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.id === 'restart');
|
||||
expect(job.status).toBe('complete'); expect(job.collected).toBe(1); expect(job.enriched).toBe(1); expect(calls).toHaveLength(1);
|
||||
});
|
||||
test('storage totals count table aggregates once and use actual disk capacity', async () => {
|
||||
await upsertMedia('aaaaaaaaaaa', { status: 'ready', size: 2048, duration: 120 });
|
||||
await db.execute("INSERT INTO listening_daily (fingerprint,day,video_id,plays) VALUES ('listener','2026-10-03','aaaaaaaaaaa',3)");
|
||||
const result = await storageAnalytics([{ label: 'DB', path: root }, { label: 'Absent', path: root + '/missing' }]);
|
||||
expect(Number(result.media.find(r => r.status === 'ready').bytes)).toBe(2048); expect(Number(result.listening.plays)).toBe(3); expect(result.volumes[0].free).toBeGreaterThan(0); expect(result.volumes[1].unavailable).toBe(true);
|
||||
expect(result.sources.find(r => r.source === 'collector')).toBeDefined(); expect((await get('/api/admin/metadata?q=Related&offset=0')).status).toBe(200); expect((await get('/api/admin/metadata?q=%25')).status).toBe(200);
|
||||
});
|
||||
test('extraction errors remain visible while discovered cards stay stored', async () => {
|
||||
const failureApp = new Hono(); const failed = registerAnalyticsRoutes(failureApp, { adminAuth, autoStart: false, runYtdlp: async a => a[0].startsWith('ytsearch') ? JSON.stringify({ id: 'failure0001', title: 'Discovered before failure' }) : Promise.reject(Error('Unavailable video')) });
|
||||
await post('/api/admin/collections', { query: 'failure', maxVideos: 1, depth: 0 }); await failed.runNext();
|
||||
const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.query === 'failure');
|
||||
expect(job.status).toBe('complete'); expect(job.failed).toBe(1); expect(job.collected).toBe(1); expect(job.errors[0].error).toBe('Unavailable video');
|
||||
});
|
||||
|
||||
test('duplicate search cards do not consume slots before unique results', async () => {
|
||||
const a = { id: 'unique00001', title: 'Unique A' }, b = { id: 'unique00002', title: 'Unique B' };
|
||||
const worker = registerAnalyticsRoutes(new Hono(), { adminAuth, autoStart: false, runYtdlp: async args => args[0].startsWith('ytsearch') ? [a,a,b].map(JSON.stringify).join('\n') : JSON.stringify(args.at(-1).endsWith(a.id) ? a : b) });
|
||||
await post('/api/admin/collections', { query: 'duplicates', maxVideos: 2, depth: 0 }); await worker.runNext();
|
||||
const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.query === 'duplicates');
|
||||
expect(job.collected).toBe(2); expect(job.enriched).toBe(2);
|
||||
});
|
||||
test('cancelling an in-flight search prevents storing its returned cards', async () => {
|
||||
let started, release;
|
||||
const began = new Promise(resolve => { started = resolve; }), pause = new Promise(resolve => { release = resolve; });
|
||||
const worker = registerAnalyticsRoutes(new Hono(), { adminAuth, autoStart: false, runYtdlp: async () => { started(); await pause; return JSON.stringify({ id: 'cancel00001', title: 'Should not save' }); } });
|
||||
const queued = await (await post('/api/admin/collections', { query: 'cancel while searching', maxVideos: 1, depth: 0 })).json();
|
||||
const work = worker.runNext(); await began; await post('/api/admin/collections/' + queued.id + '/cancel'); release(); await work;
|
||||
const job = (await (await get('/api/admin/collections')).json()).jobs.find(j => j.id === queued.id);
|
||||
expect(job.status).toBe('cancelled'); expect(job.failed).toBe(0);
|
||||
expect((await db.execute("SELECT * FROM video_meta WHERE id='cancel00001'")).rows).toHaveLength(0);
|
||||
});
|
||||
22
server/db.js
22
server/db.js
@@ -157,6 +157,28 @@ export async function initDb() {
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_video_channel ON video_channels (channel,updated_at DESC,video_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS video_details (
|
||||
video_id TEXT PRIMARY KEY, metadata TEXT NOT NULL, updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS metadata_collections (
|
||||
id TEXT PRIMARY KEY, query TEXT NOT NULL, max_videos INTEGER NOT NULL, depth INTEGER NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'queued', state TEXT NOT NULL DEFAULT '{}', error TEXT,
|
||||
lease_until INTEGER NOT NULL DEFAULT 0, owner TEXT,
|
||||
created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_collections_queue ON metadata_collections (status,created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS lyric_transcriptions (
|
||||
id TEXT PRIMARY KEY, video_id TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'queued',
|
||||
base_rev INTEGER NOT NULL DEFAULT 0, result TEXT, error TEXT, stage TEXT NOT NULL DEFAULT 'queued',
|
||||
attempts INTEGER NOT NULL DEFAULT 0, lease_token TEXT, lease_until INTEGER NOT NULL DEFAULT 0,
|
||||
created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_transcription_active ON lyric_transcriptions (video_id)
|
||||
WHERE status IN ('queued','running');
|
||||
CREATE INDEX IF NOT EXISTS idx_transcription_queue ON lyric_transcriptions (status,created_at);
|
||||
CREATE TABLE IF NOT EXISTS lyrics_worker_state (id INTEGER PRIMARY KEY CHECK(id=1), last_seen INTEGER NOT NULL);
|
||||
|
||||
-- Shared per-video documents (kind = lyrics | chapters), visible to every
|
||||
-- user. The live copy is here; every save also lands in video_note_revs
|
||||
-- as a full snapshot, which is the server-side backup and undo history.
|
||||
|
||||
@@ -42,6 +42,8 @@ import { createHash, createHmac, randomBytes, timingSafeEqual } from 'node:crypt
|
||||
import { mkdirSync, readdirSync, unlinkSync, writeFileSync, readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { getCookie, setCookie, deleteCookie } from 'hono/cookie';
|
||||
import { registerAnalyticsRoutes } from './admin-analytics.js';
|
||||
import { registerTranscriptionRoutes } from './transcriptions.js';
|
||||
|
||||
export const NOTE_KINDS = new Set(['lyrics', 'chapters']);
|
||||
// A YouTube id or one of the server's own uploads (see uploads.js).
|
||||
@@ -59,6 +61,12 @@ function cleanText(v, max) {
|
||||
return String(v == null ? '' : v).replace(/[\u0000-\u001f\u007f]+/g, ' ').trim().slice(0, max);
|
||||
}
|
||||
|
||||
function cleanLyricText(value) {
|
||||
return String(value ?? '').replace(/\r\n?/g, '\n')
|
||||
.replace(/[\u0000-\u0009\u000b-\u001f\u007f]+/g, ' ')
|
||||
.split('\n').map(part => part.trim()).filter(Boolean).join('\n').trim().slice(0, MAX_LINE_CHARS);
|
||||
}
|
||||
|
||||
function cleanTime(v) {
|
||||
if (v === null || v === undefined || v === '') return null;
|
||||
const n = Number(v);
|
||||
@@ -76,7 +84,7 @@ export function sanitizeLyrics(input) {
|
||||
const lines = [];
|
||||
for (const l of rawLines) {
|
||||
if (!l || typeof l !== 'object') continue;
|
||||
const text = cleanText(l.text, MAX_LINE_CHARS);
|
||||
const text = cleanLyricText(l.text);
|
||||
if (!text) continue;
|
||||
const kind = l.kind === 'section' || l.kind === 'cue' ? l.kind : 'line';
|
||||
lines.push({ t: cleanTime(l.t), text, kind });
|
||||
@@ -426,7 +434,7 @@ export function registerNoteRoutes(app, deps) {
|
||||
if (!who) return c.json({ ok: false, error: 'link an online profile to report a lyric line' }, 401);
|
||||
if (who.invalid) return c.json({ ok: false, error: 'invalid API token' }, 401);
|
||||
if (flagOverBudget(who.by)) return c.json({ ok: false, error: 'too many reports — wait a few minutes' }, 429);
|
||||
const text = cleanText(body.text, MAX_LINE_CHARS);
|
||||
const text = cleanLyricText(body.text);
|
||||
if (!text) return c.json({ ok: false, error: 'which line? (text is missing)' }, 400);
|
||||
try {
|
||||
const lyr = (await db.getNotes(id)).lyrics;
|
||||
@@ -662,6 +670,18 @@ export function registerNoteRoutes(app, deps) {
|
||||
await next();
|
||||
};
|
||||
|
||||
if (deps.analytics) registerAnalyticsRoutes(app, { adminAuth: requireAdminOrToken, runYtdlp, ...deps.analytics });
|
||||
|
||||
registerTranscriptionRoutes(app, {
|
||||
db, adminAuth: requireAdminOrToken, sanitizeLyrics,
|
||||
workerEnabled: !!workerToken && workerToken.length >= 24,
|
||||
workerAuth: async (c, next) => {
|
||||
const token = (c.req.header('authorization') || '').match(/^Bearer\s+(\S+)$/i)?.[1];
|
||||
if (!workerToken || workerToken.length < 24 || !token || !safeEqual(token, workerToken)) return c.json({ ok: false, error: 'lyrics worker token required' }, 401);
|
||||
await next();
|
||||
},
|
||||
});
|
||||
|
||||
// Saved (server-cached) videos with whether each already has lyrics — the
|
||||
// work list for batch lyric injection.
|
||||
app.get('/api/admin/media', requireAdminOrToken, async (c) => {
|
||||
|
||||
@@ -84,6 +84,12 @@ describe('pure helpers', () => {
|
||||
expect(d.offset).toBe(30);
|
||||
});
|
||||
|
||||
test('lyrics retain tight line breaks while other metadata strips controls', () => {
|
||||
const result = N.sanitizeLyrics({ lines: [{ t: 12, text: ' Because You are God\r\n You can do anything\u0007 ', kind: 'line' }], tags: ['Key\nG'] });
|
||||
expect(result.lines).toEqual([{ t: 12, text: 'Because You are God\nYou can do anything', kind: 'line' }]);
|
||||
expect(result.tags).toEqual(['Key G']);
|
||||
});
|
||||
|
||||
test('sanitizeChapters requires time + title and sorts', () => {
|
||||
const d = N.sanitizeChapters({ items: [{ t: 30, title: 'B' }, { t: 5, title: 'A', note: 'n' }, { t: null, title: 'no time' }] });
|
||||
expect(d.items).toEqual([{ t: 5, title: 'A', note: 'n' }, { t: 30, title: 'B', note: '' }]);
|
||||
@@ -245,3 +251,19 @@ describe('routes', () => {
|
||||
expect(await res.text()).toContain('admin');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
test('saving a grouped cue preserves one timestamp in current and revision documents', async () => {
|
||||
const id = 'groupedCue1';
|
||||
const lines = [{ t: 12, text: 'Because You are God\nYou can do anything', kind: 'line' }];
|
||||
const response = await app.request(`/api/notes/${id}/lyrics`, json('PUT', { data: { lines }, baseRev: 0, profile: 'josh' }));
|
||||
expect(response.status).toBe(200);
|
||||
const current = await (await app.request(`/api/notes/${id}`)).json();
|
||||
expect(current.lyrics.data.lines).toEqual(lines);
|
||||
const revision = await (await app.request(`/api/notes/${id}/lyrics/revs/1`)).json();
|
||||
expect(revision.data.lines).toEqual(lines);
|
||||
const flag = await app.request(`/api/notes/${id}/flags`, json('POST', { text: lines[0].text, profile: 'josh', reason: 'words' }));
|
||||
expect(flag.status).toBe(200);
|
||||
const flags = await (await app.request(`/api/notes/${id}/flags`)).json();
|
||||
expect(flags.flags[0].text).toBe(lines[0].text);
|
||||
});
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
"scripts": {
|
||||
"start": "bun server.js",
|
||||
"dev": "bun --hot server.js",
|
||||
"test": "bun test ./recommendations.test.js && bun test --timeout 60000 ./media-cache.test.js && bun test ./notes.test.js && bun test ./remote.test.js && bun test ./party.test.js && bun test ./uploads.test.js && bun test ./innertube.test.js && bun test ./ytdlp-pool.test.js && bun test ./p2p-db.test.js && bun test ./p2p-admit.test.js && bun test ./p2p-retention.test.js && bun test ./p2p-routes.test.js && bun test ./p2p-hub.test.js && bun test --timeout 60000 ./p2p-intake.test.js && bun test ./flags.test.js"
|
||||
"test": "bun test ./recommendations.test.js && bun test --timeout 60000 ./media-cache.test.js && bun test ./notes.test.js && bun test ./transcriptions.test.js && bun test ./admin-analytics.test.js && bun test ./remote.test.js && bun test ./party.test.js && bun test ./uploads.test.js && bun test ./innertube.test.js && bun test ./ytdlp-pool.test.js && bun test ./p2p-db.test.js && bun test ./p2p-admit.test.js && bun test ./p2p-retention.test.js && bun test ./p2p-routes.test.js && bun test ./p2p-hub.test.js && bun test --timeout 60000 ./p2p-intake.test.js && bun test ./flags.test.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^1.14.0",
|
||||
|
||||
@@ -84,7 +84,7 @@ export function registerCatalogRoutes(app, { resolveListener = async (c, name, f
|
||||
? c.req.raw.clone() : null;
|
||||
await next();
|
||||
if (!c.res.ok) return;
|
||||
const discovery = c.req.method === 'GET' && !path.startsWith('/api/catalog/') && path !== '/api/recommendations';
|
||||
const discovery = c.req.method === 'GET' && !/^\/api\/admin\/(analytics|metadata|collections)(\/|$)/.test(path) && !path.startsWith('/api/catalog/') && path !== '/api/recommendations';
|
||||
try {
|
||||
if (discovery && c.res.headers.get('content-type')?.includes('application/json')) {
|
||||
const result = await c.res.clone().json();
|
||||
|
||||
@@ -2328,6 +2328,11 @@ const notes = registerNoteRoutes(app, {
|
||||
backupDir: pathJoin(dirname(process.env.DB_PATH || './data/ytplayer.db'), 'backups'),
|
||||
adminHtmlPath: './public/admin.html',
|
||||
workerToken: process.env.LYRICS_WORKER_TOKEN || '',
|
||||
analytics: { paths: [
|
||||
{ label: 'Video cache', path: MEDIA_DIR },
|
||||
{ label: 'Uploads', path: process.env.UPLOAD_DIR || pathJoin(dirname(process.env.DB_PATH || './data/ytplayer.db'), 'uploads') },
|
||||
{ label: 'Database', path: dirname(process.env.DB_PATH || './data/ytplayer.db') },
|
||||
] },
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
|
||||
79
server/transcriptions.js
Normal file
79
server/transcriptions.js
Normal file
@@ -0,0 +1,79 @@
|
||||
// Explicit Whisper requests produce reviewable drafts, never published notes.
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { db as sql } from './db.js';
|
||||
const ID = /^([\w-]{11}|upl_[a-f0-9]{12})$/;
|
||||
const LEASE_MS = 120000;
|
||||
function publicJob(row) {
|
||||
if (!row) return null;
|
||||
return { id: row.id, videoId: row.video_id, status: row.status, stage: row.stage,
|
||||
baseRev: Number(row.base_rev), result: row.result ? JSON.parse(row.result) : null,
|
||||
error: row.error, createdAt: Number(row.created_at), updatedAt: Number(row.updated_at) };
|
||||
}
|
||||
|
||||
export function registerTranscriptionRoutes(app, { db, adminAuth, workerAuth, workerEnabled, sanitizeLyrics, now = Date.now }) {
|
||||
const seen = () => sql.execute({ sql: 'INSERT INTO lyrics_worker_state (id,last_seen) VALUES (1,?) ON CONFLICT(id) DO UPDATE SET last_seen=excluded.last_seen', args: [now()] });
|
||||
async function source(id) {
|
||||
if (id.startsWith('upl_')) return db.getUpload(id);
|
||||
const row = await db.getMedia(id); return row?.status === 'ready' ? row : null;
|
||||
}
|
||||
app.post('/api/admin/transcriptions/:video', adminAuth, async c => {
|
||||
const id = c.req.param('video');
|
||||
if (!ID.test(id)) return c.json({ ok: false, error: 'invalid video id' }, 400);
|
||||
if (!workerEnabled) return c.json({ ok: false, error: 'Whisper is unavailable: the lyrics worker is not configured.' }, 503);
|
||||
const audio = await source(id);
|
||||
if (!audio) return c.json({ ok: false, error: 'Save this video on the server before transcribing its audio.' }, 409);
|
||||
if (Number(audio.duration) > 3600) return c.json({ ok: false, error: 'Whisper requests are limited to one hour of audio.' }, 422);
|
||||
const notes = await db.getNotes(id);
|
||||
// One active job per song and a bounded queue, including across restarts.
|
||||
const existing = (await sql.execute({ sql: "SELECT * FROM lyric_transcriptions WHERE video_id=? AND status IN ('queued','running')", args: [id] })).rows[0];
|
||||
if (existing) return c.json({ ok: true, job: publicJob(existing) });
|
||||
const active = Number((await sql.execute("SELECT COUNT(*) AS n FROM lyric_transcriptions WHERE status IN ('queued','running')")).rows[0].n);
|
||||
if (active >= 25) return c.json({ ok: false, error: 'The transcription queue is full. Try again after a job finishes.' }, 429);
|
||||
const time = now();
|
||||
await sql.execute({ sql: `INSERT OR IGNORE INTO lyric_transcriptions (id,video_id,base_rev,created_at,updated_at)
|
||||
SELECT ?,?,?,?,? WHERE (SELECT COUNT(*) FROM lyric_transcriptions WHERE status IN ('queued','running')) < 25`,
|
||||
args: [randomUUID(), id, notes.lyrics?.rev || 0, time, time] });
|
||||
const row = (await sql.execute({ sql: "SELECT * FROM lyric_transcriptions WHERE video_id=? AND status IN ('queued','running')", args: [id] })).rows[0];
|
||||
return row ? c.json({ ok: true, job: publicJob(row) }, 202) : c.json({ ok: false, error: 'The transcription queue is full.' }, 429);
|
||||
});
|
||||
app.get('/api/admin/transcriptions/:video', adminAuth, async c => {
|
||||
const id = c.req.param('video');
|
||||
if (!ID.test(id)) return c.json({ ok: false, error: 'invalid video id' }, 400);
|
||||
const row = (await sql.execute({ sql: 'SELECT * FROM lyric_transcriptions WHERE video_id=? ORDER BY created_at DESC,rowid DESC LIMIT 1', args: [id] })).rows[0];
|
||||
const worker = (await sql.execute('SELECT last_seen FROM lyrics_worker_state WHERE id=1')).rows[0];
|
||||
return c.json({ ok: true, job: publicJob(row), workerOnline: !!worker && now() - Number(worker.last_seen) < 90000, enabled: !!workerEnabled }, 200, { 'Cache-Control': 'no-store' });
|
||||
});
|
||||
app.post('/api/lyrics-worker/claim', workerAuth, async c => {
|
||||
await seen(); const time = now();
|
||||
await sql.execute({ sql: "UPDATE lyric_transcriptions SET status='failed',stage='failed',error='The worker stopped repeatedly. Please retry.',updated_at=? WHERE status='running' AND lease_until<? AND attempts>=3", args: [time, time] });
|
||||
// UPDATE RETURNING makes claiming atomic for multiple worker processes.
|
||||
const row = (await sql.execute({ sql: `UPDATE lyric_transcriptions SET status='running',stage='loading-model',attempts=attempts+1,
|
||||
lease_token=?,lease_until=?,updated_at=? WHERE id=(SELECT id FROM lyric_transcriptions
|
||||
WHERE (status='queued' OR (status='running' AND lease_until<?)) AND attempts<3 ORDER BY created_at LIMIT 1) RETURNING *`,
|
||||
args: [randomUUID(), time + LEASE_MS, time, time] })).rows[0];
|
||||
return c.json({ ok: true, job: row ? { ...publicJob(row), lease: row.lease_token,
|
||||
audioPath: row.video_id.startsWith('upl_') ? `/api/uploads/${row.video_id}` : `/api/media/${row.video_id}?a=1` } : null });
|
||||
});
|
||||
app.post('/api/lyrics-worker/jobs/:job', workerAuth, async c => {
|
||||
let body;
|
||||
try { const raw = await c.req.text(); if (raw.length > 400000) return c.json({ ok: false, error: 'transcript too large' }, 413); body = JSON.parse(raw); }
|
||||
catch { return c.json({ ok: false, error: 'invalid JSON' }, 400); }
|
||||
if (!body || typeof body.lease !== 'string') return c.json({ ok: false, error: 'missing lease' }, 400);
|
||||
const time = now();
|
||||
let result = null, status = 'running', error = null;
|
||||
let stage = ['loading-model', 'downloading-audio', 'transcribing'].includes(body.stage) ? body.stage : 'transcribing';
|
||||
if (body.status === 'complete') {
|
||||
try { result = sanitizeLyrics(body.result); if (!result.lines.length) throw Error('No sung lyrics were found.'); }
|
||||
catch (e) { return c.json({ ok: false, error: e.message }, 400); }
|
||||
result.tags = ['auto-transcribed (whisper)']; status = 'complete'; stage = 'complete';
|
||||
} else if (body.status === 'failed') { status = 'failed'; stage = 'failed'; error = String(body.error || 'Transcription failed.').slice(0, 300); }
|
||||
const changed = await sql.execute({ sql: `UPDATE lyric_transcriptions SET status=?,stage=?,result=?,error=?,lease_until=?,updated_at=?
|
||||
WHERE id=? AND status='running' AND lease_token=? AND lease_until>=?`,
|
||||
args: [status, stage, result ? JSON.stringify(result) : null, error, time + LEASE_MS, time, c.req.param('job'), body.lease, time] });
|
||||
if (!changed.rowsAffected) return c.json({ ok: false, error: 'This job lease has expired or finished.' }, 409);
|
||||
await seen();
|
||||
// Bound completed draft retention; active jobs are never deleted.
|
||||
await sql.execute({ sql: "DELETE FROM lyric_transcriptions WHERE status IN ('complete','failed') AND updated_at<?", args: [time - 30 * 86400000] });
|
||||
return c.json({ ok: true });
|
||||
});
|
||||
}
|
||||
97
server/transcriptions.test.js
Normal file
97
server/transcriptions.test.js
Normal file
@@ -0,0 +1,97 @@
|
||||
import { test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { Hono } from 'hono';
|
||||
const root = mkdtempSync(join(tmpdir(), 'ytp-transcriptions-'));
|
||||
process.env.DB_PATH = join(root, 'test.db');
|
||||
const db = await import('./db.js');
|
||||
const { registerNoteRoutes } = await import('./notes.js');
|
||||
const VID = '0gfX0dFLaBc';
|
||||
const TOKEN = 'worker-test-token-'.repeat(3);
|
||||
const worker = { Authorization: 'Bearer ' + TOKEN };
|
||||
const json = (body = {}, headers = {}) => ({ method: 'POST', headers: { 'Content-Type': 'application/json', ...headers }, body: JSON.stringify(body) });
|
||||
let app, cookie, job, lease;
|
||||
beforeAll(async () => {
|
||||
await db.initDb(); await db.upsertMedia(VID, { status: 'ready', duration: 240 });
|
||||
await db.saveNote({ videoId: VID, kind: 'lyrics', baseRev: 0, source: 'user', updatedBy: 'fixture', data: { lines: [{ t: 1, text: 'Original saved lyrics', kind: 'line' }], tags: [], offset: 0 } });
|
||||
writeFileSync(join(root, 'admin.html'), '<title>admin</title>');
|
||||
app = new Hono();
|
||||
registerNoteRoutes(app, { db, getProfile: async () => null, profileNameRe: /^[\w-]{3,40}$/, runYtdlp: async () => '{}', adminPassword: 'test-password', workerToken: TOKEN, backupDir: join(root, 'backups'), adminHtmlPath: join(root, 'admin.html') });
|
||||
const login = await app.request('/api/admin/login', json({ password: 'test-password' }));
|
||||
cookie = login.headers.get('set-cookie').split(';')[0];
|
||||
});
|
||||
afterAll(() => { db.db.close(); rmSync(root, { recursive: true, force: true }); });
|
||||
const admin = () => ({ Cookie: cookie });
|
||||
const draft = { lines: [{ t: 2, text: 'Because You are God', kind: 'line' }, { t: 6, text: 'You can do anything', kind: 'line' }], tags: ['wrong source'], offset: 0 };
|
||||
|
||||
test('admin and worker authentication are enforced separately', async () => {
|
||||
expect((await app.request('/api/admin/transcriptions/' + VID, json())).status).toBe(401);
|
||||
expect((await app.request('/api/admin/transcriptions/' + VID)).status).toBe(401);
|
||||
expect((await app.request('/api/lyrics-worker/claim', json({}, admin()))).status).toBe(401);
|
||||
});
|
||||
test('two button requests deduplicate and preserve published lyrics', async () => {
|
||||
const first = await app.request('/api/admin/transcriptions/' + VID, json({}, admin()));
|
||||
expect(first.status).toBe(202); job = (await first.json()).job;
|
||||
const second = await (await app.request('/api/admin/transcriptions/' + VID, json({}, admin()))).json();
|
||||
expect(second.job.id).toBe(job.id); expect(job.baseRev).toBe(1);
|
||||
expect((await db.getNotes(VID)).lyrics.data.lines[0].text).toBe('Original saved lyrics');
|
||||
await db.initDb();
|
||||
expect((await (await app.request('/api/admin/transcriptions/' + VID, { headers: admin() })).json()).job.id).toBe(job.id);
|
||||
});
|
||||
test('claiming is atomic and does not expose leases to the admin UI', async () => {
|
||||
const responses = await Promise.all([app.request('/api/lyrics-worker/claim', json({}, worker)), app.request('/api/lyrics-worker/claim', json({}, worker))]);
|
||||
const jobs = await Promise.all(responses.map(r => r.json()));
|
||||
expect(jobs.filter(r => r.job)).toHaveLength(1); lease = jobs.find(r => r.job).job.lease;
|
||||
expect(jobs.find(r => r.job).job.audioPath).toBe('/api/media/' + VID + '?a=1');
|
||||
const view = await (await app.request('/api/admin/transcriptions/' + VID, { headers: admin() })).json();
|
||||
expect(view.job.lease).toBeUndefined(); expect(view.workerOnline).toBe(true);
|
||||
});
|
||||
test('heartbeat advances stages and invalid leases cannot finish jobs', async () => {
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease: 'wrong', status: 'complete', result: draft }, worker))).status).toBe(409);
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, stage: 'transcribing' }, worker))).status).toBe(200);
|
||||
const view = await (await app.request('/api/admin/transcriptions/' + VID, { headers: admin() })).json();
|
||||
expect(view.job.stage).toBe('transcribing');
|
||||
});
|
||||
test('completion returns a sanitized review draft without overwriting saved lyrics', async () => {
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, status: 'complete', result: draft }, worker))).status).toBe(200);
|
||||
const view = await (await app.request('/api/admin/transcriptions/' + VID, { headers: admin() })).json();
|
||||
expect(view.job.status).toBe('complete'); expect(view.job.result.lines).toHaveLength(2);
|
||||
expect(view.job.result.tags).toEqual(['auto-transcribed (whisper)']);
|
||||
expect((await db.getNotes(VID)).lyrics.rev).toBe(1);
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, status: 'failed' }, worker))).status).toBe(409);
|
||||
});
|
||||
test('expired jobs are reclaimed with a new lease; stale results are rejected', async () => {
|
||||
const queued = await (await app.request('/api/admin/transcriptions/' + VID, json({}, admin()))).json();
|
||||
expect(queued.job.id).not.toBe(job.id); job = queued.job;
|
||||
const first = (await (await app.request('/api/lyrics-worker/claim', json({}, worker))).json()).job;
|
||||
await db.db.execute({ sql: 'UPDATE lyric_transcriptions SET lease_until=0 WHERE id=?', args: [job.id] });
|
||||
const reclaimed = (await (await app.request('/api/lyrics-worker/claim', json({}, worker))).json()).job;
|
||||
expect(reclaimed.lease).not.toBe(first.lease);
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease: first.lease, status: 'complete', result: draft }, worker))).status).toBe(409);
|
||||
lease = reclaimed.lease;
|
||||
});
|
||||
test('empty transcripts and oversized drafts are rejected, and failures retain saved lyrics', async () => {
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, status: 'complete', result: { lines: [] } }, worker))).status).toBe(400);
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, result: 'x'.repeat(400001) }, worker))).status).toBe(413);
|
||||
expect((await app.request('/api/lyrics-worker/jobs/' + job.id, json({ lease, status: 'failed', error: 'No sung vocals were detected.' }, worker))).status).toBe(200);
|
||||
const view = await (await app.request('/api/admin/transcriptions/' + VID, { headers: admin() })).json();
|
||||
expect(view.job.error).toBe('No sung vocals were detected.'); expect((await db.getNotes(VID)).lyrics.rev).toBe(1);
|
||||
});
|
||||
test('uncached songs, invalid IDs, and overlong audio give usable errors', async () => {
|
||||
expect((await app.request('/api/admin/transcriptions/nope', json({}, admin()))).status).toBe(400);
|
||||
expect((await app.request('/api/admin/transcriptions/aaaaaaaaaaa', json({}, admin()))).status).toBe(409);
|
||||
await db.upsertMedia('bbbbbbbbbbb', { status: 'ready', duration: 4000 });
|
||||
expect((await app.request('/api/admin/transcriptions/bbbbbbbbbbb', json({}, admin()))).status).toBe(422);
|
||||
});
|
||||
test('uploads use the upload media endpoint and repeated worker crashes fail visibly', async () => {
|
||||
const id = 'upl_0123456789ab';
|
||||
await db.createUpload({ id, kind: 'audio', title: 'Fixture upload', duration: 30, ext: 'm4a', mime: 'audio/mp4', size: 100 });
|
||||
await app.request('/api/admin/transcriptions/' + id, json({}, admin()));
|
||||
const claim = (await (await app.request('/api/lyrics-worker/claim', json({}, worker))).json()).job;
|
||||
expect(claim.audioPath).toBe('/api/uploads/' + id);
|
||||
await db.db.execute({ sql: 'UPDATE lyric_transcriptions SET lease_until=0,attempts=3 WHERE id=?', args: [claim.id] });
|
||||
await app.request('/api/lyrics-worker/claim', json({}, worker));
|
||||
const view = await (await app.request('/api/admin/transcriptions/' + id, { headers: admin() })).json();
|
||||
expect(view.job.status).toBe('failed');
|
||||
});
|
||||
@@ -2,7 +2,7 @@
|
||||
import { db } from './db.js';
|
||||
|
||||
const VIDEO_ID = /^[\w-]{11}$/;
|
||||
const SOURCES = new Set(['search', 'search-cache', 'client-search', 'channel', 'streams', 'playlist', 'profile', 'sync', 'related', 'backfill']);
|
||||
const SOURCES = new Set(['search', 'search-cache', 'client-search', 'channel', 'streams', 'playlist', 'profile', 'sync', 'related', 'backfill', 'collector']);
|
||||
const text = (v, n = 300) => typeof v === 'string' ? v.trim().slice(0, n) : '';
|
||||
const positive = v => Number.isFinite(Number(v)) && Number(v) > 0 ? Number(v) : 0;
|
||||
const canonicalThumb = id => `https://i.ytimg.com/vi/${id}/hqdefault.jpg`;
|
||||
@@ -90,6 +90,7 @@ export async function trimCatalog(max = Number(process.env.VIDEO_META_MAX) || 50
|
||||
'DELETE FROM video_meta_sources WHERE video_id NOT IN (SELECT id FROM video_meta)',
|
||||
'DELETE FROM video_thumbnails WHERE video_id NOT IN (SELECT id FROM video_meta)',
|
||||
'DELETE FROM video_channels WHERE video_id NOT IN (SELECT id FROM video_meta)',
|
||||
'DELETE FROM video_details WHERE video_id NOT IN (SELECT id FROM video_meta)',
|
||||
], 'write');
|
||||
return count - max;
|
||||
}
|
||||
|
||||
152
tests/admin-lyrics.spec.js
Normal file
152
tests/admin-lyrics.spec.js
Normal file
@@ -0,0 +1,152 @@
|
||||
const { test, expect } = require('@playwright/test');
|
||||
const id = '0gfX0dFLaBc';
|
||||
async function fixture(page) {
|
||||
const state = { job: null, starts: 0, saves: [] };
|
||||
await page.route('**/api/**', async route => {
|
||||
const req = route.request(), url = new URL(req.url()); let body = { ok: true };
|
||||
if (url.pathname === '/api/admin/me') body = { enabled: true, admin: true };
|
||||
else if (url.pathname === '/api/admin/media') body = { ok: true, media: [{ id, title: 'Because You are God', channel: 'Cathedral of Praise Worship', lyricsLines: 47 }] };
|
||||
else if (url.pathname === '/api/uploads') body.uploads = [];
|
||||
else if (url.pathname === '/api/admin/flags') body = { ok: true, flags: [], open: 0 };
|
||||
else if (url.pathname === '/api/notes/' + id) body = { lyrics: { rev: 8, data: { lines: Array.from({ length: 47 }, (_, i) => ({ t: i * 5, text: 'Original cue ' + i, kind: 'line' })), tags: ['Key G'], offset: 0 } } };
|
||||
else if (url.pathname === '/api/streams') body = { data: { meta: { title: 'Because You are God', duration: 240 }, qualities: [] } };
|
||||
else if (url.pathname === '/api/admin/transcriptions/' + id) {
|
||||
if (req.method() === 'POST') { state.starts++; state.job = { id: 'job', videoId: id, status: 'queued', baseRev: 8 }; }
|
||||
body = { ok: true, job: state.job };
|
||||
} else if (url.pathname === '/api/notes/' + id + '/lyrics' && req.method() === 'PUT') { state.saves.push(req.postDataJSON()); body.rev = 9; }
|
||||
return route.fulfill({ contentType: 'application/json', body: JSON.stringify(body) });
|
||||
});
|
||||
await page.goto('/admin.html#editor');
|
||||
await expect(page.locator('[data-open="' + id + '"]').first()).toBeVisible();
|
||||
return state;
|
||||
}
|
||||
function complete(state) {
|
||||
state.job = { ...state.job, status: 'complete', result: { lines: [{ t: 1, text: 'Because You are God', kind: 'line' }, { t: 4, text: 'You can do anything', kind: 'line' }], tags: ['auto-transcribed (whisper)'], offset: 0 } };
|
||||
}
|
||||
test('row and editor buttons queue one job and require review before saving', async ({ page }) => {
|
||||
const state = await fixture(page);
|
||||
await page.locator('[data-whisper]').click();
|
||||
await expect(page.locator('#whisperBtn')).toBeDisabled();
|
||||
expect(state.starts).toBe(1);
|
||||
await expect(page.locator('.ln')).toHaveCount(47);
|
||||
complete(state);
|
||||
await expect(page.locator('#whisperUse')).toBeVisible({ timeout: 10000 });
|
||||
await expect(page.locator('.ln')).toHaveCount(47); expect(state.saves).toHaveLength(0);
|
||||
await page.locator('#whisperUse').click();
|
||||
await expect(page.locator('.ln')).toHaveCount(2); expect(state.saves).toHaveLength(0);
|
||||
await page.locator('#bUndo').click(); await expect(page.locator('.ln')).toHaveCount(47);
|
||||
await page.locator('#bRedo').click(); await expect(page.locator('.ln')).toHaveCount(2);
|
||||
await page.locator('#save').click();
|
||||
await expect.poll(() => state.saves.length).toBe(1);
|
||||
expect(state.saves[0].baseRev).toBe(8); expect(state.saves[0].data.lines[0].text).toBe('Because You are God');
|
||||
});
|
||||
test('editor button keeps unsaved words until the draft is explicitly applied', async ({ page }) => {
|
||||
const state = await fixture(page);
|
||||
await page.locator('[data-open="' + id + '"]').first().click();
|
||||
await page.locator('.ln-text').first().fill('My correction');
|
||||
await page.locator('#whisperBtn').click(); complete(state);
|
||||
await expect(page.locator('#whisperUse')).toBeVisible({ timeout: 10000 });
|
||||
await expect(page.locator('.ln-text').first()).toHaveValue('My correction');
|
||||
page.once('dialog', dialog => dialog.dismiss()); await page.locator('#whisperUse').click();
|
||||
await expect(page.locator('.ln-text').first()).toHaveValue('My correction');
|
||||
expect(state.saves).toHaveLength(0);
|
||||
});
|
||||
test('mobile buttons remain reachable and failed requests allow retry', async ({ page }) => {
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
await fixture(page);
|
||||
const button = page.locator('[data-whisper]'); await expect(button).toBeVisible();
|
||||
const box = await button.boundingBox(); expect(box.height).toBeGreaterThanOrEqual(44); expect(box.x + box.width).toBeLessThanOrEqual(390);
|
||||
await page.route('**/api/admin/transcriptions/' + id, route => route.fulfill({ status: 503, contentType: 'application/json', body: JSON.stringify({ ok: false, error: 'Lyrics worker is not configured.' }) }));
|
||||
await button.click(); await expect(page.locator('#msg')).toContainText('not configured');
|
||||
await expect(page.locator('#whisperBtn')).toBeEnabled();
|
||||
});
|
||||
|
||||
for (const width of [390, 1440]) {
|
||||
test(`grouped cue keeps one time through text editing and save at ${width}px`, async ({ page }) => {
|
||||
await page.setViewportSize({ width, height: 900 });
|
||||
const state = await fixture(page);
|
||||
await page.locator('[data-open="' + id + '"]').first().click();
|
||||
const input = page.locator('.ln-text').first();
|
||||
await input.fill('Because You are God'); await input.press('End');
|
||||
if (width === 390) await page.locator('.ln').first().locator('[data-act="break"]').click();
|
||||
else await input.press('Shift+Enter');
|
||||
await input.press('ControlOrMeta+End'); await input.pressSequentially('You can do anything');
|
||||
await expect(input).toHaveValue('Because You are God\nYou can do anything');
|
||||
await expect(page.locator('.ln')).toHaveCount(47);
|
||||
await page.locator('#bTools').click(); await page.locator('#rawBtn').click();
|
||||
await expect(page.locator('#raw')).toContainText('');
|
||||
expect(await page.locator('#raw').inputValue()).toContain('| You can do anything');
|
||||
await page.locator('#rawBtn').click();
|
||||
await expect(page.locator('.ln-text').first()).toHaveValue('Because You are God\nYou can do anything');
|
||||
await page.locator('#save').click(); await expect.poll(() => state.saves.length).toBe(1);
|
||||
expect(state.saves[0].data.lines[0]).toEqual({ t: 0, text: 'Because You are God\nYou can do anything', kind: 'line' });
|
||||
});
|
||||
}
|
||||
|
||||
const { openClassic } = require('./helpers/classic-fixture');
|
||||
for (const width of [390, 1440]) {
|
||||
test(`service mode highlights the whole grouped cue with a tight internal gap at ${width}px`, async ({ page }) => {
|
||||
await page.setViewportSize({ width, height: 900 });
|
||||
await openClassic(page);
|
||||
await page.route('**/api/notes/' + id, route => route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, lyrics: { rev: 1, data: { lines: [{ t: 0, text: 'Because You are God\nYou can do anything', kind: 'line' }, { t: 30, text: 'Next lyric', kind: 'line' }], tags: [], offset: 0 } } }) }));
|
||||
await page.evaluate(async id => {
|
||||
current = { meta: { id, title: 'Because You are God' }, qualities: [], audioUrl: '' };
|
||||
await Notes.load(current.meta); enterServiceMode(); setServiceView('lyrics'); Notes.tick(0, true);
|
||||
}, id);
|
||||
const cue = page.locator('#serviceLyricsList .sl-line').first();
|
||||
await expect(cue).toHaveClass(/active/); await expect(cue).toHaveText('Because You are God\nYou can do anything');
|
||||
await expect(page.locator('#serviceLyricsList .sl-line')).toHaveCount(2);
|
||||
await page.waitForTimeout(450);
|
||||
const metrics = await cue.evaluate(el => {
|
||||
const style = getComputedStyle(el), next = el.nextElementSibling;
|
||||
return { space: style.whiteSpace, leading: parseFloat(style.lineHeight) - parseFloat(style.fontSize), gap: next.getBoundingClientRect().top - el.getBoundingClientRect().bottom, height: el.clientHeight, lineHeight: parseFloat(style.lineHeight) };
|
||||
});
|
||||
expect(metrics.space).toBe('pre-line'); expect(metrics.gap).toBeGreaterThan(metrics.leading);
|
||||
expect(metrics.height).toBeGreaterThan(metrics.lineHeight * 1.8);
|
||||
});
|
||||
}
|
||||
|
||||
for (const width of [320, 390, 820, 1440]) {
|
||||
test(`admin analytics exposes storage and collector controls at ${width}px`, async ({ page }) => {
|
||||
await page.setViewportSize({ width, height: 900 }); await fixture(page);
|
||||
let request = null, cancelled = false;
|
||||
await page.route('**/api/admin/analytics', route => route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, media: [{ status: 'ready', count: 2, bytes: 1048576, seconds: 3600 }], uploads: [{ count: 1, bytes: 1024 }], catalog: { videos: 15, bytes: 3000 }, details: { videos: 4, bytes: 10000 }, thumbnails: { saved: 12, pending: 3, bytes: 1000 }, listening: { plays: 47, videos: 2 }, topPlayed: [{ title: 'Because You are God', plays: 47 }], sources: [{ source: 'search', videos: 15, discoveries: 23 }], volumes: [{ label: 'Video cache', device: '1', total: 1000000000, free: 500000000 }] }) }));
|
||||
await page.route('**/api/admin/metadata?**', route => route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, total: 1, videos: [{ id, title: 'Because You are God', channel: 'Cathedral of Praise Worship', enriched: true, thumbnailBytes: 1024, duration: 240 }] }) }));
|
||||
await page.route('**/api/admin/metadata/' + id, route => route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, metadata: { id, description: 'Stored descriptive metadata', tags: ['worship'] } }) }));
|
||||
await page.route('**/api/admin/collections', route => {
|
||||
if (route.request().method() === 'POST') request = route.request().postDataJSON();
|
||||
return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, id: 'job', jobs: request ? [{ id: 'job', query: request.query, maxVideos: request.maxVideos, depth: request.depth, status: cancelled ? 'cancelled' : 'queued', collected: 0, enriched: 0, failed: 0, searches: 0 }] : [] }) });
|
||||
});
|
||||
await page.route('**/api/admin/collections/job/cancel', route => { cancelled = true; return route.fulfill({ contentType: 'application/json', body: '{"ok":true}' }); });
|
||||
await page.locator('[data-tab="analytics"]:visible').click();
|
||||
await expect(page.locator('#analyticsMetrics .metric')).toHaveCount(8);
|
||||
await expect(page.locator('#analyticsVolumes')).toContainText('available');
|
||||
await page.locator('#collectorQuery').fill('Cathedral worship'); await page.locator('#collectorLimit').fill('45'); await page.locator('#collectorDepth').selectOption('2');
|
||||
await page.locator('#collectorStart').click();
|
||||
await expect(page.locator('#collectorStatus')).toContainText('queued'); expect(request).toEqual({ query: 'Cathedral worship', maxVideos: 45, depth: 2 });
|
||||
await page.getByRole('button', { name: 'Stop collection' }).click(); await expect(page.locator('#collectionJobs')).toContainText('cancelled');
|
||||
await page.locator('[data-metadata-id]').click(); await expect(page.locator('#metadataDetail')).toContainText('Stored descriptive metadata');
|
||||
const overflow = await page.evaluate(() => document.documentElement.scrollWidth > innerWidth + 1); expect(overflow).toBe(false);
|
||||
await page.locator('#analyticsRefresh').scrollIntoViewIfNeeded();
|
||||
await page.screenshot({ path: `/tmp/ytplayer-admin-analytics-${width}.png`, fullPage: width >= 1000 });
|
||||
if (width < 1000) {
|
||||
await page.locator('#collectorForm').evaluate(el => el.scrollIntoView({ block: 'center' }));
|
||||
await page.screenshot({ path: `/tmp/ytplayer-admin-collector-${width}.png` });
|
||||
const reached = await page.locator('#collectorStart').evaluate(el => { const r = el.getBoundingClientRect(); return el.contains(document.elementFromPoint(r.x + r.width / 2, r.y + r.height / 2)); });
|
||||
expect(reached).toBe(true);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
test('player lyric editor groups Shift+Enter words and stamps their shared cue', async ({ page }) => {
|
||||
await openClassic(page);
|
||||
await page.route('**/api/notes/' + id, route => route.fulfill({ contentType: 'application/json', body: JSON.stringify({ ok: true, lyrics: { rev: 1, data: { lines: [{ t: 12, text: 'Because You are God', kind: 'line' }, { t: 30, text: 'Next lyric', kind: 'line' }], tags: [], offset: 0 } } }) }));
|
||||
await page.evaluate(async id => { data.profile = { name: 'fixture' }; current = { meta: { id, title: 'Song' } }; await Notes.load(current.meta); document.getElementById('lyricsEditBtn').click(); }, id);
|
||||
const text = page.locator('.lyr-editor-text'); await expect(text).toBeVisible();
|
||||
await text.evaluate(el => { el.focus(); const pos = el.value.indexOf('\n'); el.setSelectionRange(pos, pos); });
|
||||
await text.press('Shift+Enter'); await text.pressSequentially('You can do anything');
|
||||
expect(await text.inputValue()).toContain('Because You are God\n| You can do anything');
|
||||
await page.locator('.lyr-editor [data-act="stamp"]').click();
|
||||
const document = await text.inputValue(); expect(document).toContain('[0:00.00] Because You are God\n| You can do anything');
|
||||
expect(document).not.toContain('[0:00.00] |');
|
||||
});
|
||||
Reference in New Issue
Block a user