The Song Analyzer would not work in the browser. It would fail after about two minutes, every time, with a generic Failed to fetch. And yet every single thing I tried by hand worked perfectly: yt-dlp from the command line, the bare Flask dev server, even gunicorn — the exact production server — started by hand. Each downloaded the audio and analyzed it in a clean ten to twenty seconds. Only requests through the installed systemd service stalled and died.
That asymmetry is the most disorienting shape a bug can take. When a reproduction succeeds, your instinct is to trust it and conclude the code is fine. But the thing that mattered — the actual deployed service — kept failing identically while every reproduction sailed through. The temptation was to keep blaming the obvious suspects: YouTube throttling the IP, a flaky network, the WSL2 virtual switch. I lost hours to each of those before accepting they couldn't be it, because the manual reproductions disproved every one.
The breakthrough came from instrumenting the worker to dump its own Python stack on a signal, and then doing one clean comparison: run the identical gunicorn with its output going to a file instead of to the logging daemon. It worked. Pointed back at the daemon, it hung. That was the entire difference.
Under systemd, a service's standard output is a pipe to journald, which rate-limits how fast it will accept log lines. yt-dlp streams a download progress bar — rapid carriage-return updates — to that pipe. When the writes outpace what journald will drain, the pipe fills, and the next write() simply blocks. The download loop freezes mid-transfer, waiting on a log line nobody is reading, until the worker timeout fires and kills it. Run the same code with output to a terminal or a regular file, where write() never blocks, and it flies. That is why every manual test passed: none of them logged into a backpressured pipe.
"The reproduction that always succeeds and the system that always fails are not contradicting each other. They are telling you the difference between them is the bug."
The fix is two-sided: tell yt-dlp to stop emitting the progress bar, and route the service's output to a plain file instead of journald. Neither alone is the whole story — the progress bar was the loudest writer, but the audio post-processing subprocess could fill the pipe too. Sending the worker's output somewhere that can't apply backpressure closes the door for good.
For a while even the correct fix looked like it had failed, because a second problem was layered on top. Every failed attempt had left workers tied up and the browser retrying, and with the backend finally healthy, that backlog of overlapping requests dogpiled the machine — four workers all downloading and running CPU-heavy analysis at once, none finishing in time, each timeout spawning another retry. A clean restart cleared it instantly, which is the tell of a contention spiral rather than a code bug: the same request that fails under load succeeds the moment it runs alone.
The durable answer wasn't more workers, it was fewer concurrent analyses. A cross-process limit — built on the same kind of file lock already used to dedup identical songs — now lets overlapping requests queue and run one at a time, in exactly the clean conditions that always worked, instead of thrashing each other into failure.
With it finally working, the tuning was almost relaxing by comparison. Chord, key, and tempo detection all run on low-rate mono audio, so the analyzer had been downloading and processing far more data than it ever used. Pulling a smaller audio stream, having ffmpeg emit the exact 22 kHz mono the analysis wants, and computing the expensive chroma transform once instead of twice took a fresh analysis from around twenty seconds down to roughly twelve — and repeats of the same song still return instantly from cache. The embedded player now starts loading the instant you hit analyze, in parallel with the work, instead of only spinning up afterward.
None of the time lost was wasted on the wrong layer by carelessness; it was wasted because the evidence genuinely pointed away from the answer. A bug that only manifests through the production launcher, and disappears under every hand-run reproduction, trains you to distrust the one signal that matters. The lesson wasn't about journald specifically. It was that when a reproduction and the real system disagree, the gap between them is not noise to be ignored — it is the shortest path to the cause, if you're willing to test it directly instead of arguing with it.