What Real Hardware Costs You

June 29, 2026 Self-Hosted Infra • Deployment • Boot Resilience

The Design Was Right. The Box Wasn't.

Yesterday's hardening pass gave the Song Analyzer a disk cache, gunicorn workers, and a dedup lock — on paper, a service ready for real traffic. None of that mattered the moment the physical machine running it actually rebooted. Three separate things turned out to be unfinished underneath the design, each invisible until the specific condition that exposed it actually happened.

Switched to Gunicorn. Forgot to Install It.

The systemd unit had already been pointed at gunicorn -w 4 --timeout 120 -b 127.0.0.1:5005 app:app. The venv on the box had not. It predated the switch, and its requirements.txt never picked up gunicorn — a stale checkout, not a code bug. Every restart died with status=203/EXEC, which is systemd telling you it couldn't even find the binary to execute, a layer below where a Python traceback would show up. pip install "gunicorn>=22.0" inside the venv fixed it outright; the next restart came up as one master process and four workers, confirmed in ps aux, matching the -w 4 flag exactly.

The lesson isn't about gunicorn specifically. It's that "the repo says this is how it runs" and "this is what's actually running on the box" are two different claims, and only a physical machine lets them quietly drift apart. Nobody else's platform does that to you.

WSL2 Doesn't Care That Windows Rebooted

Both chord-analyzer and cloudflared were already enabled inside the Linux distro — which guarantees nothing if nothing on the Windows side ever launches the distro in the first place. WSL2 only starts when something explicitly invokes it: opening a terminal, an app reaching into it. A cold Windows boot with nobody doing that leaves the whole stack off indefinitely, services "enabled" and never running.

The fix lives entirely on the Windows side: a Task Scheduler entry running wsl.exe -d Ubuntu --exec /bin/true at startup, configured to run whether anyone is logged in or not, so it doesn't wait on a human session to exist. The other half of the gap is just as easy to miss — Windows sleep looks identical to powered-off from the outside. No ping, no SSH, indistinguishable from off until someone physically walks up and wakes the machine. powercfg /change standby-timeout-ac 0 and the matching monitor-timeout change close that door too.

Then YouTube Decided the Box Looked Like a Bot

Separately from anything above, yt-dlp downloads started failing outright: Sign in to confirm you're not a bot. Not a regression in this codebase — YouTube rate-limiting or flagging the request pattern, something that happens to yt-dlp users generally, more on datacenter IPs but not exclusively. The fix is to make the requests look like they're coming from a real, authenticated browser instead of a bare script: app.py now checks for an optional cookies.txt on every request and, if present, hands it to yt-dlp as a cookiefile.

That file is exported from a logged-in browser session, lives only on the physical machine, is gitignored, and is never committed — it's live session authentication, not a secret that belongs anywhere near version control. It also rotates: cookies expire over weeks to months depending on the account, so this isn't a one-time fix, it's a maintenance task that resurfaces on its own schedule.

What Actually Changed

None of these three were code review failures, because none of them were code problems. They were the specific, boring gap that only exists when "the infrastructure" is a machine in a house instead of someone else's platform: deployment state that can silently diverge from repo state, an OS that doesn't survive its own reboot unattended, and an upstream API that can simply change its mind about whether you're allowed to talk to it. Owning the hardware bought real audio analysis instead of fake chords. It also bought every one of these failure modes, one at a time, in the order they happened to occur.