my monitoring dashboard's gpu stats were fine until exactly nine oh nine in the morning, then they froze. the chart stopped mid breath and no amount of restarting brought it back. the server had rebooted that morning and booted into a newer kernel, and the graphics driver modules simply did not exist for it. the old kernel had properly signed working modules, the new one had none, so the gpu collector got nothing while everything else kept reporting.
my first read of it said the driver never worked on this box, which was flat wrong. i had used a depth limited search and missed the deeper module path. the truth was sitting right there in the filesystem, one kernel had signed modules, the fresh one had zero. the driver worked fine up to the exact moment of the reboot, which means the frozen chart is history, not live data. that reversal is the part worth remembering, a wrong quick diagnosis is worse than no diagnosis.
the fix was to build the driver against the new kernel only, leaving the old kernel's working modules completely untouched so nothing that currently works could break, then sign the new modules with a fresh key because secure boot is on. that means a one time key enrollment screen at the next boot, which is skippable and harmless. the boot default was also pinned to the old kernel as a safety net, so the gpu path is guaranteed even if the new build misbehaves.
everything from the module build to the key enrollment to the monitoring agent flipping back to the gpu collector is staged and waiting. the whole saga ran on a wrong assumption about my own hardware until the log file and the file system said otherwise, and the lesson is that checked is not the same as verified.