NVIDIA RTX 5090 and RTX PRO 6000 affected by virtualization reset bug, $1,000 reward for a solution

CloudRift, a GPU cloud provider, has reported a significant issue affecting NVIDIA’s latest Blackwell GPUs, the RTX 5090 and RTX PRO 6000, in virtualization environments. The problem arises when these GPUs are passed through to virtual machines (VMs) using KVM and VFIO, leading to a failure in recovery after a GPU reassignment or guest shutdown. Instead of resetting correctly, the GPU enters an unresponsive state, causing kernel errors and making it impossible to reassign the device without rebooting the entire system. This issue has been observed in both production environments and testing, with no alternative recovery methods like PCI rescans or power state changes proving effective. CloudRift has offered a $1,000 bounty for anyone who can resolve the bug or provide reproducible tests that will help uncover the root cause. While older models like the RTX 4090 and NVIDIA’s H100 and B200 GPUs are unaffected, the new Blackwell-based cards are causing considerable issues for users, including system hangs and VM shutdown failures. Despite these reports, NVIDIA has not yet issued a public statement regarding the bug, although it is known that the company is aware of the problem.