Opus 4.6 is still my preferred model for work, so this is great to hear.

Man, there is exactly zero information on your site about how your benchmarks work. Why should one trust your numbers when there is no way to verify them?

Why Deepseek v4 flash is better than pro in your benchmarks?

After having used GLM 5.2 and Opus 4.8 for enough time, I'm very unconvinced of the benchmark maxxing claims - if anything, GLM 5.2's rather lackluster performance on benchmarks compared to Opus 4.8 paints the opposite picture when compared to the subjective experience.

When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pretty much just studying what AI agents can do for my own curiosity and no other reason. Opus 4.8 was one of the first models I ever snuck in there that basically ran out of control. No previous Opus or Sonnet model I had used ever did this. Within hours every agent I had running was writing non-sense tool calls that echoed pretend commands that didn't exist, like 10 in a row, and talking about the "tool channel" being dirty. I switched back to Opus 4.7 and assumed Opus 4.8 was legitimately just broken.

I did come back to Opus 4.8 and found that it was indeed, pretty powerful. But that initial experience has stuck with me on just how narrow of a perspective any given test or benchmark is guaranteed to have. LLMs are too broad, it really doesn't matter what you try to do in your benchmark, you will necessarily get a limited view of what the model is capable of and its shortcomings. This will remain true for at least as long as models are susceptible to massive swings in performance based on randomness and minor differences in prompts and other environmental factors.

I'm not saying benchmarks are useless or that your benchmarks are not possibly closer to the truth either. All evidence at least points to the idea that Chinese models perform very well in coding but often have more mixed results on other tasks. I'm just saying that at this point, benchmarks feel like they have limited connection to my actual real experiences. GLM 5.2 actually scored kinda meh on a lot of benchmarks (compared to closed frontier models) but my actual experience using it does not match this.

And I'm definitely not saying GLM 5.2 is better than the frontier LLMs here, just that the race is close. I still prefer GPT 5.5 right now for code review, I think, and Opus clearly has some advantages depending on the task. It's just no longer a given that Opus 4.8 will perform better than GLM 5.2 on any given task, so to me the calculus behind "using the best model available" is getting complex and you might need to get a feel for what models have what strengths to really figure it out.

I do feel like the "use the best model available" mentality is not going to die any time soon, but if it does die, it will be gradual and start soon for programming. Modern LLMs are still not a full superset of what human programmers can do, but still larger models are definitely starting to hit diminishing returns for tasks at the lower end of complexity, and that is a big deal. It's a weird world where some tasks you can feel kinda confident just throwing Gemma 4 at it and not sweating whether you should use a better model; I've certainly done it for some quick Python scripts or getting an overview of some code I'm unfamiliar with.

Notice the website url is the same name as the commentor.

Notice he's using "trust me bro" benchmarks.

Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.

Everyone is grinding and marketing nobody is actually discussing anything for real.

My increasing frustration with these plans is the harness lock in.

Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates.

So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.

There is a whole iceberg topic on subsidizing.

So your question is really “if they’re giving free usage, why not take advantage of it?”

I do, so I don’t know the reasons not to, other than to experiment.

[dead]

Oh interesting. I basically chose Matrix because setting anything up with Whatsapp or signal was kind of painful and telegram doesn't make it easy to use encryption with bots.

I kind of wanted to see if I can make a Matrix agent from scratch with Rust with GLM and it was surprisingly easy. Just make something for myself how I want it. Maybe I'll take a look on Hermes later...

Very interesting—Element X solved a lot of the pains of Element (iOS), could be a good solution!

I use an observability tool with claude code [1] that shows me usage including prompt and session cost. Even though I use a max subscription, it's interesting to see what it would cost me if I was using API directly.

My typical session ranges from $100-$400 - higher end when using workflows with lots of subagents. $100/session is expected when using the API without the subsidized subscription pricing. Most larger orgs have to use API pricing AFAIK.

[1] https://github.com/simple10/agents-observe

I have Claude max plan and the vscode claude dashboard plugin has logged about $4k worth of tokens in the past 2 months. I upgraded because I was using my weekly basic plan tokens in like 5 hours.

Likewise, I don't understand how anyone survives on the basic plans. It's funny seeing these two camps not understanding what the other is doing :)

It's really interesting what "normal" is for folks. I use the $200/month Anthropic subscription and use it within a few percentages of my limit every week.

I'd blow through $20/month plan in hours.

API, pay per token.

Opencode and Zed about 40/60.

It's pretty simple. There are things that I do because it's fun, like gamedev. I hand code that, and don't use LLM tools because I like learning and building. I do lots of utility stuff coding for my wife's business, most of that is stuff I could do in a few hours. It's worth $20 to not spend a few hours doing it. It's a cost benefit tradeoff. I won't learn much fixing WordPress themes or adding a feature to her web page, or setting up an automation for her, so I don't see the point of doing that.

Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.

Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.

$20 is really cheap for the amount of work saved, considering you're in the US.

Is spending $20 considered "super rich"?