by mkagenius 4 hours ago
It looks like the author is specifically avoiding model's name, because results are really weird.
Opus 4.8/4.7 scored 28%
Opus 4.6 score 37%
Not weird at all, given the variance in Opus' quality over the last few months.
wild guess - I wouldn't be surprised if Opus 4.6 was run quantized for a while, and 4.7/4.8 have QAT for that nerfed size.
Where is the weird part?
many people think opus 4.6 was the best