diff --git a/.agents/skills/lyrics-agy/SKILL.md b/.agents/skills/lyrics-agy/SKILL.md new file mode 100644 index 0000000..8cf0c48 --- /dev/null +++ b/.agents/skills/lyrics-agy/SKILL.md @@ -0,0 +1,118 @@ +--- +name: lyrics-agy +description: Find lyrics (with timings when they exist) for worship.hesed.sbs songs using agy, the flat-rate Antigravity CLI, via the agy-bridge MCP server. Use when LRCLIB has no match, when a song still has no lyrics or bad machine-transcribed ones, or when the user says "ask agy for the lyrics". Also how to paste lyrics you already have into a song. +--- + +# Lyrics from agy + +agy searches the open web and, when the song has a caption track or a published +sync, returns **timed** lyrics in LRC form. It is flat-rate, so a run costs +nothing per song — but it is the **last** resort, after LRCLIB (`lyrics-lookup`, +`lyrics-regenerate`), because LRCLIB's synced lyrics are published data while +agy's are derived. + +Script: `scripts/lyrics/agy_lyrics.py`. + +## Run it + +```bash +cd ~/development/personal/ytplayer +export YTP_ADMIN_PASSWORD='…' # or YTP_TOKEN=ytp_… + +# every song that still has no lyrics — dry run first, ALWAYS +python3 scripts/lyrics/agy_lyrics.py --missing +python3 scripts/lyrics/agy_lyrics.py --missing --apply + +# one song, replacing lyrics that are wrong +python3 scripts/lyrics/agy_lyrics.py --ids ZHl6EwSwjv0 --overwrite --apply + +# lyrics you already have (LRC or plain text), no agy call at all +python3 scripts/lyrics/agy_lyrics.py --ids ZHl6EwSwjv0 --from-file words.txt --overwrite --apply +``` + +Every run backs the current lyrics up to +`Documents/ytplayer-lyrics-backup-.json` before writing, and the server +keeps each previous version as a restorable revision. + +**One song takes ~2 minutes per attempt**, and `--tries` defaults to 3, so a +batch is slow. Run it detached rather than in a foreground command that will hit +a timeout: + +```bash +setsid nohup python3 scripts/lyrics/agy_lyrics.py --missing --apply \ + > /tmp/agy-lyrics.log 2>&1 < /dev/null & +``` + +## How it calls agy + +Through MCPJungle, so no agy CLI contract is hard-coded here: + +```bash +mcpjungle invoke agy-bridge__agy_ask --input '{"dir": "", "prompt": "…"}' +mcpjungle invoke agy-bridge__fetch_output --input '{"keep_id": "ask-…"}' +``` + +Four things about that interface cost real debugging time — do not rediscover them: + +1. **`dir` is required.** Without it the call fails schema validation. +2. **The answer arrives on STDERR**, not stdout. Read both streams or you get + an empty string and conclude, wrongly, that agy found nothing. +3. **Long answers are truncated** with `fetch_output(keep_id='…') for more`. + A full set of lyrics is almost always longer than the cap, so always follow + the `keep_id` — otherwise you silently save half a song. +4. **agy can fail and still look like a success.** A quota-exhausted instance + returns prose like `WHY: agy-ask failed (rc=1)` inside a `STATUS: ok` + envelope. One such line even carried a `[05:37.76]` stamp and parsed as a + perfectly good synced lyric. `FAILED` in the script rejects those; keep it. + +## It is not deterministic — that is the main gotcha + +The same question can come back synced, plain, or empty on consecutive calls. +Observed in one sitting on "Jesus At The Centre": 39 timed lines, then 43 +untimed, then nothing. So the script asks up to `--tries` times and keeps the +**best** answer (`score()`: any timed lines beats none, then more lines beats +fewer), stopping early once a timed answer arrives. + +If a song saves untimed and you believe a timed version exists, just run it +again with `--overwrite`. + +## What it filters out, and why + +agy streams its own progress into the answer. Everything below is dropped +before parsing, and every pattern is there because it once ended up saved as a +lyric line: + +- `STATUS:` / `SUMMARY:` / `MODEL:` / `INSTANCE:` / `AGY-META:` / `WHY:` envelopes +- `Waiting for task execution…`, `Background task completed with…` +- section labels that are not sung (`Verse 1`, `[Chorus]`, `x2`) +- anything over 200 characters (a paragraph of commentary, not a sung line) + +After filtering, an answer shorter than `--min-lines` (6) is rejected outright. + +## Reviewing before you trust it + +agy derives timings, so check them once per song before relying on them in a +service: + +- The last timestamp should land near the song's length (a 384 s song ending at + 367 s is right; one ending at 120 s means it only got a verse). +- Timestamps must increase monotonically. +- Open `/admin?v=`, press play and watch the highlighted line track the + vocal. Fix drift with **Shift all**, or retime individual lines with **Set** / + **Tap mode**. + +Untimed results are fine — the app shows them as a plain scrolling list, and +Tap mode turns them into synced lyrics in one pass of the song. + +## Provenance tags + +Written into `data.tags` so a later run can tell where lyrics came from: + +| Tag | Meaning | +|---|---| +| `from the web via agy (synced)` | agy, with timings | +| `from the web via agy (untimed) — check and Tap-sync` | agy, words only | +| `from a file () (synced\|untimed)` | `--from-file` | + +Related: `lyrics-lookup` (LRCLIB first, then this), `lyrics-regenerate` +(replace wrong lyrics from LRCLIB), `deploy-prod`. diff --git a/.claude/skills/lyrics-agy b/.claude/skills/lyrics-agy new file mode 120000 index 0000000..b300112 --- /dev/null +++ b/.claude/skills/lyrics-agy @@ -0,0 +1 @@ +../../.agents/skills/lyrics-agy \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 242adc5..acab354 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -405,6 +405,11 @@ Skills live in `.agents/skills/` (symlinked into `.claude/skills/`): - **mobile-app-ui-design** — UI/UX design work on the PWA screens. - **lyrics-lookup** — fill in songs that have NO lyrics (LRCLIB → agy web search → local faster-whisper). - **lyrics-regenerate** — back up, then replace wrong/mistimed lyrics from LRCLIB (`scripts/lyrics/lrclib_regen.py`). +- **lyrics-agy** — last resort when LRCLIB has nothing: agy (flat-rate) searches the web and often returns + LRC-timed lyrics (`scripts/lyrics/agy_lyrics.py`). Also the `--from-file` path for pasting lyrics you already have. + agy is NOT deterministic — the script asks up to `--tries` times and keeps the best (timed > untimed > longer). + Its answer arrives on **stderr**, long answers are truncated behind a `keep_id`, and a quota-failed instance + returns error prose inside a `STATUS: ok` envelope that will parse as lyrics if you let it. ### Matching a song on LRCLIB (learned the hard way) `/api/streams` gives the YouTube **channel**, not the artist, so a title+artist diff --git a/scripts/lyrics/agy_lyrics.py b/scripts/lyrics/agy_lyrics.py new file mode 100644 index 0000000..b24991b --- /dev/null +++ b/scripts/lyrics/agy_lyrics.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""agy_lyrics.py — find lyrics for saved songs with agy (the Antigravity CLI). + +agy is flat-rate, so this costs nothing per song. It is the LAST resort: +LRCLIB first (free and usually SYNCED), then this. agy searches the open web +(Genius, AZLyrics, hymnary, the artist's own site, YouTube transcripts) and is +asked for LRC-format timings when it can find or derive them; most of the time +it comes back with plain text, which lands UNTIMED and can be timed afterwards +in the admin lyric editor's Tap mode. + + YTP_ADMIN_PASSWORD=… python3 scripts/lyrics/agy_lyrics.py --missing + YTP_ADMIN_PASSWORD=… python3 scripts/lyrics/agy_lyrics.py --ids ZHl6EwSwjv0 --apply + YTP_ADMIN_PASSWORD=… python3 scripts/lyrics/agy_lyrics.py --ids ID --from-file words.txt --apply + +Nothing is written without --apply, and the current lyrics of every song the run +touches are backed up to a JSON file first (the server also keeps every previous +version as a revision). +""" +import argparse +import datetime as dt +import json +import os +import re +import subprocess +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from auto_lyrics import Api # noqa: E402 +from lrclib_regen import clean_artist, clean_title, parse_lrc # noqa: E402 + +ASK_TOOL = 'agy-bridge__agy_ask' +FETCH_TOOL = 'agy-bridge__fetch_output' +# agy prepends STATUS/SUMMARY and appends its own metadata; neither is lyrics. +META = re.compile(r'^(STATUS:|SUMMARY:|MODEL:|INSTANCE:|AGY-META:|WOULD RUN:|NEXT:|WHY:|ERROR:|\$ )', re.I) +# agy can fail per instance (quota) and still exit 0 through the bridge; its +# error prose would otherwise be parsed as lyrics — one such line even carried +# a [05:37.76] stamp and looked like a perfectly good synced lyric. +FAILED = re.compile(r'(STATUS:\s*error|agy-ask failed|quota exhausted|rc=[1-9])', re.I) +# agy streams its own progress chatter into the answer ("Waiting for task +# execution...", "Background task completed"). None of it is lyrics. +CHATTER = re.compile( + r'(background task|waiting for task|task execution|completed with|' + r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|' + r'^\.\.\.|^\s*task-\d+|tool call|searching the web|let me |i (will|.ll) )', re.I) +SECTION = re.compile(r'^\[?\(?(verse|chorus|bridge|intro|outro|pre-?chorus|refrain|tag|vamp|instrumental|repeat|x\d)[^\]\)]*\)?\]?:?$', re.I) + + +def mcp(tool, payload, timeout=900): + """Call an MCP tool through the mcpjungle CLI and return `result` text.""" + proc = subprocess.run( + ['mcpjungle', 'invoke', tool, '--input', json.dumps(payload)], + capture_output=True, text=True, timeout=timeout, + ) + # mcpjungle prints the tool's answer on STDERR, not stdout — read both. + out = (proc.stderr or '') + (proc.stdout or '') + # The CLI prints a human block, then "** Structured Content **" + JSON. + m = re.search(r'\*\*\s*Structured Content\s*\*\*\s*(\{.*)', out, re.S) + if m: + try: + return str(json.loads(m.group(1)).get('result', '')) + except json.JSONDecodeError: + pass + return out + + +def ask_agy(title, artist, duration=0, workdir='.'): + """Ask agy for one song's lyrics. Returns the raw text ('' when not found).""" + dur = f' (about {int(duration) // 60} minutes {int(duration) % 60} seconds long)' if duration else '' + prompt = ( + f'Find the full lyrics for the song "{title}"' + (f' by {artist}' if artist else '') + dur + '. ' + 'Search the web (Genius, AZLyrics, hymnary, worshiptogether, the artist\'s own site, YouTube transcripts). ' + 'Return ONLY the lyrics. If you can find or derive a timing for each line, return them in LRC format: ' + 'every line prefixed with [mm:ss.xx] and one sung line per line. If no timings exist anywhere, return the ' + 'plain lyrics, one sung line per line, with NO timestamps and NO invented ones. ' + 'No commentary, no chords, no section labels unless they are actually sung. ' + 'If you cannot find this exact song with confidence, reply exactly: NOT FOUND' + ) + text = mcp(ASK_TOOL, {'dir': os.path.abspath(workdir), 'prompt': prompt}) + # Long answers come back truncated with a keep_id — fetch the rest. + keep = re.search(r"fetch_output\(keep_id='([^']+)'", text) + if keep: + full = mcp(FETCH_TOOL, {'keep_id': keep.group(1)}) + if len(full) > len(text): + text = full + if FAILED.search(text): + raise RuntimeError('agy did not answer: ' + ' '.join(text.split())[:160]) + return '' if 'NOT FOUND' in text.upper()[:400] else text + + +def clean_lines(text): + """agy's answer -> lyric lines. Keeps LRC stamps when it supplied them.""" + body = [] + for raw in str(text or '').splitlines(): + t = raw.strip().strip('*_`') + if not t or META.match(t) or SECTION.match(t) or CHATTER.search(t): + continue + if t.startswith(('#', '>', '|', 'http')) or re.fullmatch(r'[-=_*]{3,}', t): + continue + if len(t) > 200: # a paragraph of commentary, not a sung line + continue + body.append(t) + lines = parse_lrc('\n'.join(body)) + # Trim a repeated tail ("In Jesus' name" x4 is real; 20 identical lines is not) + return lines[:400] + + +def score(lines): + """Rank two agy answers: timed beats untimed, then longer beats shorter.""" + return (sum(1 for l in lines if l['t'] is not None) > 0, len(lines)) + + +def summarise(lines): + timed = sum(1 for l in lines if l['t'] is not None) + return f'{len(lines)} lines ({timed} timed)' if timed else f'{len(lines)} lines (untimed)' + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--base', default=os.environ.get('YTP_BASE', 'https://worship.hesed.sbs')) + pick = ap.add_mutually_exclusive_group(required=True) + pick.add_argument('--missing', action='store_true', help='every saved song with no lyrics') + pick.add_argument('--ids', help='comma-separated video ids') + ap.add_argument('--from-file', help='use this file\'s text instead of asking agy (LRC or plain; one song, needs --ids)') + ap.add_argument('--overwrite', action='store_true', help='also replace lyrics a song already has') + ap.add_argument('--apply', action='store_true', help='actually save (default: dry run)') + ap.add_argument('--min-lines', type=int, default=6, help='reject an answer shorter than this') + ap.add_argument('--tries', type=int, default=3, help='ask agy up to N times and keep the best answer') + ap.add_argument('--backup-dir', default=os.environ.get('YTP_BACKUP_DIR', '')) + args = ap.parse_args() + + if args.from_file and not args.ids: + sys.exit('--from-file needs --ids (it is one song\'s words)') + api = Api(args.base, os.environ.get('YTP_TOKEN'), os.environ.get('YTP_ADMIN_PASSWORD')) + + if args.missing: + st, r = api.call('GET', '/api/admin/media') + if st != 200: + sys.exit(f'listing failed ({st}): {r.get("error")} — set YTP_TOKEN or YTP_ADMIN_PASSWORD') + todo = [m['id'] for m in r['media'] if args.overwrite or not m.get('lyricsLines')] + else: + todo = [x.strip() for x in args.ids.split(',') if x.strip()] + if not todo: + sys.exit('nothing to do — every song already has lyrics') + + # Back up whatever these songs have now, before anything is replaced. + backup = {} + for vid in todo: + st, n = api.call('GET', f'/api/notes/{vid}') + cur = (n or {}).get('lyrics') if st == 200 else None + if cur: + backup[vid] = {'savedAt': cur['updatedAt'], 'rev': cur['rev'], 'data': cur['data']} + if backup: + out_dir = args.backup_dir or ('/mnt/c/Users/josh/Documents' if os.path.isdir('/mnt/c/Users/josh/Documents') else os.path.expanduser('~')) + path = os.path.join(out_dir, f'ytplayer-lyrics-backup-{dt.datetime.now():%Y%m%d-%H%M%S}.json') + with open(path, 'w', encoding='utf-8') as f: + json.dump({'exportedAt': dt.datetime.now().isoformat(), 'base': args.base, 'songs': backup}, f, ensure_ascii=False, indent=1) + print(f'backup: {len(backup)} songs → {path}\n') + + saved = skipped = 0 + for vid in todo: + st, sres = api.call('GET', f'/api/streams?v={vid}') + meta = ((sres or {}).get('data') or {}).get('meta') or {} + title, artist = clean_title(meta.get('title') or vid), clean_artist(meta.get('channel')) + st, n = api.call('GET', f'/api/notes/{vid}') + cur = (n or {}).get('lyrics') if st == 200 else None + if cur and not args.overwrite: + print(f'{vid} skip: already has {len(cur["data"]["lines"])} lines (use --overwrite)') + skipped += 1 + continue + + if args.from_file: + lines = clean_lines(open(args.from_file, encoding='utf-8').read()) + source = f'from a file ({os.path.basename(args.from_file)})' + else: + print(f'{vid} asking agy about "{title}"' + (f' by {artist}' if artist else '') + ' …', flush=True) + # agy is not deterministic: the same question can come back synced, + # plain, or empty. Ask a few times and keep the best answer — most + # timed lines wins, then most lines. + best, why = [], '' + for attempt in range(max(1, args.tries)): + try: + got = clean_lines(ask_agy(title, artist, float(meta.get('duration') or 0))) + except RuntimeError as err: + why = str(err) + continue + if score(got) > score(best): + best = got + if any(l['t'] is not None for l in best) and len(best) >= args.min_lines: + break # a timed answer is as good as it gets + if attempt + 1 < max(1, args.tries): + print(f' try {attempt + 1}: {summarise(got) if got else "nothing"} — asking again') + lines = best + if not lines and why: + print(f' {why}') + source = 'from the web via agy' + if len(lines) < args.min_lines: + print(f'{vid} no usable answer ({len(lines)} lines) — left alone') + skipped += 1 + continue + + synced = any(l['t'] is not None for l in lines) + tag = f'{source} ({"synced" if synced else "untimed"})' + ('' if synced else ' — check and Tap-sync') + print(f'{vid} {summarise(lines)} | {title[:40]}') + if not args.apply: + saved += 1 # "would save" — the dry-run summary says so + print(' ' + ' / '.join(l['text'] for l in lines[:3])[:110]) + continue + doc = {'lines': lines, 'tags': [tag], 'offset': 0} + st, rr = api.call('PUT', f'/api/notes/{vid}/lyrics', {'data': doc, 'baseRev': cur['rev'] if cur else 0}) + if st == 200: + saved += 1 + print(f' saved as rev {rr.get("rev")}') + else: + print(f' write failed {st}: {rr.get("error")}') + print(f'\n{saved} {"saved" if args.apply else "would be saved"}, {skipped} left alone' + + ('' if args.apply else ' (dry run — add --apply)')) + + +if __name__ == '__main__': + main()