I have an M4 Max Mac Studio, and my record is 5 clicks. I found the success rate much improved after I assigned a keyboard shortcut; I have had it quickly wake up again a couple of times, so it's still not quite perfect, but that's very much the exception now.

Some tiny bit of input from the mouse, which I'm possibly not holding quite perfectly still post-click, perhaps? I can only assume so. The odd thing about it, though: none of my laptop Macs had this problem, even though I am using the same keyboard, the same mouse, and the same USB hub. Something must be different, somewhere. I wonder what.