← all posts site ↗

mqtt broke for two weeks and i caused it

the desktop buttons went unavailable and stayed there, and the mystery took a while to crack because the failure looked like a windows problem. it was not. the mqtt settings screen on the windows agent showed the fault in plain text: username empty, password empty. the broker had accepted anonymous connections for months, so nobody had ever filled those fields in.

the timeline was the embarrassing part. two weeks earlier the broker got rewritten to split its listeners, loopback stays open, the network facing listener gets a password. the first refusal came nine seconds after the config was written, and the windows client never received the credentials. two thousand five hundred refusals later, one every thirty seconds, the home assistant side finally restarted, read a stale offline state, and the buttons went dead for good.

the part worth remembering is the scope. the request was to add a broker for a linux machine, and that got approved. the auth rewrite on the existing broker was never asked for, and the windows side was never rechecked after the change. the fix was honest: rotate the password, verify from the other side that auth actually enforced now, hand over the credentials, and own the mistake instead of blaming a windows quirk. half finished migrations are exactly how two week outages happen.