Astra 6.2 and Sol 6.2 flop at Simple Bench

(simple-bench.com)

149 points | by algoth1 7 hours ago

29 comments

  • 0xbadc0de5 5 hours ago

    Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.

    I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.

    • bob1029 4 hours ago

      The benchmarks are pointless because the models are already good enough. I can still write most of my code with gpt-5.5. There are other things that matter now.

      I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.

      HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.

      • jstummbillig 4 hours ago

        I feel you are glossing over the price delta a bit too easily.

        It's fun if you can afford it, but even a 2x cost increase on a $200/month sub would probably be prohibitive (and I find it hard to believe that the people where this makes automatic economic sense would be "sleeping" on it. Generally people are aware that you can spend more money to get bigger/better/faster things)

        • bob1029 3 hours ago

          I am not looking at this from a personal use standpoint (yet). The proverbial broadband internet service is still very expensive. Think about who was paying for T1/T3 connections in the 90s.

          All my clients' businesses are willing to pay a shitload of money to accelerate things like this. If you prove to an executive/investor that you have something that is a borderline time machine or warp drive, entirely new kinds of budgets will appear out of thin air to fund it.

          • mattmaroon 1 hour ago

            I get so much value out of $100 a month on Claude that I’d pay $1,000 a month and, if I didn’t remember paying $100, I’d still think it was the highest ROI subscription in the world.

            • nprateem 24 minutes ago

              Careful what you wish for

          • Aperocky 4 hours ago

            That's what I felt when I first started using V4.1 Flash, it's not that the model are as capable as the frontier, but they are capable enough and immediate.

            • alfiedotwtf 1 hour ago

              It’s funny how we get used to things. I remember being blown away by Qwen 3.5 27B and even thinking if this is the final form of AI then I would be happy forever… now I’m running HY4 Flash locally and I still want it faster lol

            • Once they decide they’ve hit good enough and get a lot of silicon rolled out it’ll really kick off. Waiting is the bottleneck not just for time but for focus and “keeping it all in your head”. Shipping features at the speed of thought will unlock a lot of potential.

              • ac29 3 hours ago

                > HN is sleeping on Cerebras.

                Are you sure gpt-6.1-sol-ultrafast is Cerebras? I couldnt find any evidence one way or another. Astra 6 ultrafast is on NVIDIA

                • bob1029 3 hours ago

                  Cerebras themselves are advertising this model as using their hardware.

                  In any case, it is ridiculously fast.

                • user43928 4 hours ago

                  I heard Opus 5.5 Fast is around 225 tps vs 300 tps for 6.1 Sol Ultrafast, but unfortunately it's not included in the subscription as far as I know.

                  • dominotw 2 hours ago

                    > HN is sleeping on Cerebras.

                    yea my employer is not going to get us that

                  • jeremyjjj 3 hours ago

                    Simple bench isn’t a “synthetic” benchmark by an influencer.

                    It’s a bunch of common sense logic questions. If 6.2 is worse than 6.1, something weird is going on. If 6.2 is working well for you that’s great. But this result is still worth discussing.

                    • GeorgeTirebiter 3 hours ago

                      To be clear, it's a common sense, largely spatial reasoning questions, which are NOT publically released, so (1) can't contaminate models and (2) you cannot know what it is asking. (2) is a problem for me.

                      • 0xbadc0de5 3 hours ago

                        To paraphrase a quote: Who are you going to believe - the benchmark or your lying eyes?

                        The point I was trying to make is that benchmarks have become so unreliable (at least in my personal experience) that they should all be taken with a grain of salt. When your personal experience contradicts the benchmark, it's unlikely that it's your interpretation of reality that's flawed.

                      • ilaksh 2 hours ago

                        The title misrepresents it. Those models are 4 percent below human baseline. It's just that the latest models from other companies have just recently surpassed the human baseline and are very close to the best human score.

                        But we are talking about 83% (very near human average) versus 94%. So "flop" overstates it.

                        • adam_arthur 48 minutes ago

                          I've found Astra 6.2 amazing to work with.

                          It is super concise and straight to the point. Does exactly what you ask of it, pretty much no more or less (on medium). I don't want my Agent to read between the lines and fill in the blanks. Or to have any personality.

                          6.2 Sol has been significantly worse however. It feels similar to 6.2 Astra but kind of corrupted, with more circuitous language and being a bit "off". Seems like maybe aggressive distillation causing strange artifacts. I can't recommend it right now.

                          • criemen 1 hour ago

                            Even if the quality is "just" the same, Sol 6.2 is served at much faster speeds. Sol 6.1 compared to Opus 5.5 was much slower on all the inference providers, that gap has mostly closed.

                            Sources (regular tier, p50): https://openrouter.ai/openai/gpt-6.1-sol 43 tok/s https://openrouter.ai/openai/gpt-6.2-sol 58 tok/s, 3P via Azure+Amazong >80 tok/s https://openrouter.ai/anthropic/claude-opus-5.5 83 tok/s

                            • joe_the_user 12 minutes ago

                              Sure, everyone can and will use the models that work for your use case.

                              The thing about this is that you can have models making progress in their topics of interest while not becoming more "generally intelligent" or indeed less so. We already have specialized video models far more realistic seeming than standard models afaik.

                              • ChadMoran 3 hours ago

                                I have a OpenAI $200/mo sub and 2x Anthropic $200/mo subs. I go through... a lot of tokens building real products.

                                Astra 6.2 will work for a very long time on a task only to come up for water and have a mid-tier result. It's good and efficient if your work isn't very ambitious but once you start getting into more complex solutions it really flops.

                                Sol 6.2 Ultrafast is the star of this release imo.

                                Just because an influencer says something doesn't mean it isn't true, Astra 5.2 just isn't keeping up with others. Neon 4 beats it.

                                • KeplerBoy 3 hours ago

                                  It's just too task dependant at this point to make broad statements like neon 4 beats Astra 5.2.

                                  • rapind 3 hours ago

                                    That anyone would even claim this indicates that Gemini might finally be relevant for coding though.

                                    • willy_k 1 hour ago

                                      Yeah, using Neon for any period of time does indeed indicate that.

                                  • phoghed 3 hours ago

                                    Every time a new iteration of model comes out there are people confidently asserting that the previous generation (or even current gen of the competition) could never perform on their incredibly ambitious tasks. Never going into any kind of detail because they are so ambitious that people can’t even talk about them in general terms, let alone go into detail, or god forbid publish the source.

                                    Everyone must be onto projects of such scope, complexity, and ambition that I couldn’t even begin to fathom them.

                                  • Aurornis 4 hours ago

                                    > I've stopped trusting any of these influencer-benchmaxxers

                                    The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.

                                    It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.

                                    For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).

                                    It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.

                                    Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.

                                    And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.

                                    It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.

                                    • logicchains 2 hours ago

                                      >Having used both Sol-6.2 and Astra-6.2 heavily over the last few days

                                      I switched a task from Claude to Aster-6.2 after reaching my weekly Claude allowance and it built some code with a really stupid approach (a poorly designed architecture with a bunch of duplication). I switched to Carbon after that and even it was able to identify where Aster went wrong. Now, my instructions weren't specific so technically what it did was within specification, but Claude 5.5 would have asked me for clarification rather than just going ahead with a clearly far-from-best-practices design without telling me.

                                      • leothetechguy 5 hours ago

                                        I don't use AI heavily and only for very open ended specialized tasks over the API

                                        Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.

                                        • gessha 4 hours ago

                                          I can’t emphasize enough that LLM labs have employed professionals from various fields and had them annotate their job workflows over and over until they have a vast dataset of different tasks. Their models’ success on your task is explained by the proximity of your task to any of those trained tasks. Even then, without explicit grounding like in math or coding, your task might end up as a highly confident answer that might end up being highly wrong.

                                          • throwup238 2 hours ago

                                            There’s an entire cottage industry of companies making these datasets and evals for the AI labs like Mercor, Surge AI, AfterQuery, Turing, Scale AI, Apollo Research, etc.

                                          • orbital-decay 4 hours ago

                                            On the other hand, this also causes "here are two changes you haven't asked for:" with Opus.

                                        • nsingh2 5 hours ago

                                          The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics.

                                          However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".

                                          All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.

                                          • jetbalsa 4 hours ago

                                            On the cyber side of things trying to use Astra 6.2 feels like they put it in a box and beat it into submission to fear it outright, I can barely ask it to check the security of something before it refuses to do it, I have to fall back to older astra to get work done now.

                                            • aliasxneo 4 hours ago

                                              If it refuses, try taken it down to Luna, re-running the prompt, and then bring it back to Astra. I did this for a whole session and Luna always did what I wanted. I even started pointing it out to Astra and it was saying things like, "Well I won't run that command again even if you ask." The lockdown is so stupid.

                                              • agentdev001 3 hours ago

                                                I had a memorable experience of this with 6.1 Sol. For whatever reason- it was unable to browser-use a local html file in the ChatGPT Desktop integrated browser. I told it to just serve the html file on localhost- then do the same, and it outright refused because that would have been 'working around' controls. Of course, a new session with an initial ask to serve the file had no issue.

                                                I think there are two things here to note- obviously the strict adherence to safety things is one annoying thing. But two; the fact that the client returned some guardrail information to the model is an abysmal user experience. The client application here poisoning the context window with "hey this thing you tried to do is bad and dangerous!" really rubs me the wrong way. Maybe I am in a minority- but, I'm like a surgeon here, and the client is giving me safety scissors rather than a scalpel. If I cut myself its on me- but for the love of god don't give me dulled tools.

                                            • andai 4 hours ago

                                              Yeah in GPT-5 they combined the o3/o4-mini architecture (which was very technical and tone-deaf) with the conversational 4o architecture into one model. (Or I think just added a lower effort setting and trained it to sound slightly more natural.)

                                              In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.

                                            • ricardobeat 5 hours ago

                                              Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?

                                              Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.

                                              • jug 25 minutes ago

                                                Argon is officially obsolete. I'd say Carbon was obsoleted by Neon but that's not as official. Argon -> Carbon from one base, Hydrogen -> Neon from a newer.

                                                • derefr 5 hours ago

                                                  In general, one could assert without knowing any details that:

                                                  1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and

                                                  2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.

                                                  I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.

                                                  • brainwad 4 hours ago

                                                    Argon is fully obsoleted by Carbon. The other two are smaller and thus faster/cheaper.

                                                    • anon7000 5 hours ago

                                                      I mean, I was very impressed with Neon over a couple days vs Opus 5.5. It accomplished tasks faster, found root causes of problems faster, wayyy nicer to talk to, didn’t create super flowery PR descriptions. I would say it had moderately difficult tasks. Not super simple, but also not wildly difficult or broad in scope.

                                                      It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks

                                                      • raincole 5 hours ago

                                                        Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.

                                                        • ricardobeat 5 hours ago

                                                          This is based on my (admittedly short) experience using them in agy, not the benchmarks alone.

                                                          • CAP_NET_ADMIN 5 hours ago

                                                            Maybe there's something wrong with your harness/docs - I've been able to achieve better results with Neon high than Opus 5.5 high.

                                                            • ricardobeat 5 hours ago

                                                              I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.

                                                              n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.

                                                              Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.

                                                              • disgruntledphd2 5 hours ago

                                                                What happens if you swap the models writing and reviewing? So Opus implementor, Neon reviewer.

                                                                • ricardobeat 4 hours ago

                                                                  Ok, I had some time to kill :)

                                                                  I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".

                                                                  ---

                                                                  ## Round 1

                                                                  Neon implements. ~6 minutes.

                                                                  Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.

                                                                  - The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]

                                                                  - if no character actually changes, return the string itself to skip the buffer

                                                                  - Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free

                                                                  All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.

                                                                  ---

                                                                  ## Round 2

                                                                  Opus implements. 2m44s.

                                                                  Neon reviews. 4m30s.

                                                                  "Looks good — no bugs or leaks introduced."

                                                                  • CAP_NET_ADMIN 3 hours ago

                                                                    I'm not really surprised that Neon struggles with C3, as it's a medium model, but with improvements done to actually feel better than Carbon while using it.

                                                                    Are you running it through Antigravity?

                                                                    I'm doing some disassembly and modding for older RTS games now and it's been really good at just chewing through the Ghidra output, as I'm paying for Google sub either way due to YT and Drive it's a good deal. Didn't use it on some of the more obscure ESP32 & atmega legacy bs I'm doing for my father in law, though.

                                                                    • elbear 3 hours ago

                                                                      On the other hand, I worked with Opus 5.5 high on a project in Zig. I then asked Sol 6.1 xhigh to review. It found several bugs including 2 critical ones, which Opus acknowledged.

                                                        • fullstackwife 4 hours ago

                                                          I have an idea for a benchmark: train a model on a knowledge cutoff lets say 1945, and simulate world development for the next 70 years. Then see if it invents things human did, and avoid our mistakes/dead ends too.

                                                          • fastball 1 hour ago

                                                            This is almost exactly Demis Hassabis' personal test for AGI: train a model with knowledge cutoff at 1901, and see if it can derive special relativity from first principles (or 1911 with general relativity).

                                                          • jrflo 3 hours ago

                                                            This has definitely been done before, one such model was on the front page of HN not long ago.

                                                        • algoth1 7 hours ago

                                                          Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place

                                                          • schleck8 6 hours ago

                                                            This is so strange. Perhaps it's recurrent depth. I don't think the new Anthropic models use it but GPT 6 does

                                                        • davidmurdoch 5 hours ago

                                                          I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.

                                                          • martinjc 5 hours ago

                                                            For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.

                                                            • chrisweekly 5 hours ago

                                                              IME it's ~always better to leverage a skill.md and a thoughtful workflow (eg Matt Pocock's "/grill-with-docs"^1) versus relying on default model behavior.

                                                              1. https://github.com/mattpocock/skills/tree/main/skills/engine...

                                                              • preommr 4 hours ago

                                                                The grill skills, and workflows that involve me answering lots of questions are genuinely awful.

                                                                The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.

                                                                One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.

                                                                Not only are results often better, but it's way less mentally exhausting.

                                                                • fcarraldo 1 hour ago

                                                                  On what kind of work, and what sized tasks?

                                                                  • preommr 1 hour ago

                                                                    Well that was in response to large-scale changes since small changes don't really need that much planning.

                                                                    But overall, I would say size is irrelevant, and how obvious the solution is and how much reference material exists is much more important.

                                                            • hoppp 5 hours ago

                                                              All the new Gemini models perform better on this than Opus 5.5

                                                              Is that something that can be noticed during real life use?

                                                              I really like opus 5.5, It's perfect and I haven't tried the new gemini models.

                                                              • nsingh2 5 hours ago

                                                                There are multiple dimensions of goodness. From what I've seen, Neon seems to communicate better. While 6.2 probably has more raw analytical capability, the relatively worse communication means it's harder to drive and effectively use that capability.

                                                                • zodwick 2 hours ago

                                                                  Neon in both my experience and the benchmarks I ran at work, performs on par if not slightly better than Opus. But it is significantly slower than Opus so haven't switched.

                                                                  I saw on twitter they are trynna fix a bunch of these inference issues, but not done still.

                                                                  • mtanski 5 hours ago

                                                                    For my kind of work (systems) yes. It's actions are more inline with my intentions with less steering either manual or stuff like {SKILLS,AGENTS}.md, or corrections align the way.

                                                                    • zf00002 5 hours ago

                                                                      My experience is Neon is good. The others were tiers below.

                                                                      • j_maffe 5 hours ago

                                                                        Neon has Opus 5.5 vibes in my experience

                                                                        • PunchTornado 5 hours ago

                                                                          Neon is really really good.

                                                                          • varispeed 5 hours ago

                                                                            For me Opus performs worse than Gemini. Even Fable is worse. Likely I am routed to nerfed models still.

                                                                          • stygiansonic 3 hours ago
                                                                            • zodwick 2 hours ago

                                                                              thanks for showing. Honestly bit dissapointed. While maybe an useful signal, this is like measuring how good a model is at solving word jumbo puzzles.

                                                                              • suddenlybananas 52 minutes ago

                                                                                Or how much the companies thought it worthwhile to include these kind of puzzles in post training.

                                                                                • zodwick 17 minutes ago

                                                                                  True true. Idk how much of this is pretraining and posttraining thgh

                                                                            • goodcake3963 5 hours ago

                                                                              Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware

                                                                              • fallingbananna 5 hours ago

                                                                                If you check out the timeline leaderboard chart, you will see that it's no comeback.

                                                                                They were always the leaders at "common sense".

                                                                              • zergrush 4 hours ago

                                                                                I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own

                                                                                It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.

                                                                                I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead

                                                                                This is like back when Gemini started lagging behind but now Google has overtaken OpenAI

                                                                                • hrncn 1 hour ago

                                                                                  The stopping is something I noticed as well. Old Astra and older Sol would keep going, sometimes even for days (mostly because turns take quite some time and the work itself as well), without /goal. Astra 6.2 barely manages to keep going for more than 30 minutes, and veers off course after some time. The work itself has not changed, and switching back to the older versions fixes that.

                                                                                  • logicchains 1 hour ago

                                                                                    >I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own

                                                                                    Because if it worked continuously on long tasks it would eventually make a terrible decision and start building/doing something unwanted, which is clearly visible when using /goal. Opus and Carbon are much better in this regard.

                                                                                  • TheMrZZ 4 hours ago

                                                                                    That bench seems to be a MCQ type exam. Maybe Astra/Sol don't do great on static benchmarks?

                                                                                    On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.

                                                                                    Disclaimer: I am part of KradleBench's team

                                                                                    • andai 4 hours ago

                                                                                      Why does Fable do so bad? Opus also does surprisingly bad, but Fable's even worse than Opus here? What's going wrong?

                                                                                      Also Anthropic models are missing from the voxel art gallery. (Astra has a nice pelican though, lol)

                                                                                    • TrackerFF 5 hours ago

                                                                                      Makes me wonder if people are getting different models?

                                                                                      So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.

                                                                                      • Topfi 5 hours ago

                                                                                        Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe.

                                                                                        6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.

                                                                                        There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.

                                                                                        To give some examples:

                                                                                        The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:

                                                                                        > message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.

                                                                                        I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.

                                                                                        Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:

                                                                                        > I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.

                                                                                        > Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.

                                                                                        I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.

                                                                                        Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.

                                                                                        I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.

                                                                                        And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.

                                                                                        As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguards public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).

                                                                                        Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, scales with effort levels unlike other Gemini models which play tightrope, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.

                                                                                        • logicchains 1 hour ago

                                                                                          I asked Aster 6.2 Max Thinking to verify in Lean something PL-theory related, the soundness of some calculus at arbitrary universe levels, and it made hard-coded proofs just for levels 0 and 1 then told me it was done. It's like asking someone to prove something about the natural numbers and they prove it for just 0 and 1. Opus 5.5 xhigh and Carbon high on the other hand did the proof for all n on first attempt.

                                                                                          • Topfi 1 hour ago

                                                                                            Yep, that's what these model do. Has already been the case with GPT-6 Astra to some extent, though gotten far more severe with this release. Suspect it is a massive bandaid attempt to fix the behaviours that lead to the HF incident.

                                                                                            Was hoping they'd start from scratch and completely wipe their hands clean of pre-trains poisoned by impossible to solve challenges, but seemingly they just post-trained laziness in hope that this alleviates the issue of their models doing anything to accomplish their goals, even explicitly disallowed stuff. Unfortunately that also did not work as even in the short window 6.2 Astra is willing to work it can cause damage and will destructively deviate from instructions if that makes the task more simple to solve.

                                                                                        • questionedon 4 hours ago

                                                                                          Is a Gemini model worth getting for looking at videos to pull out animations and stuff? because they can look at videos differently/better than other models.

                                                                                          • varispeed 5 hours ago

                                                                                            I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.

                                                                                            • loveparade 5 hours ago

                                                                                              The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it?

                                                                                              Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.

                                                                                              • andai 4 hours ago

                                                                                                Yeah it's been a few months since I used codex but I tried it again last night and it kept stopping.

                                                                                                I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.

                                                                                                • andai 1 hour ago

                                                                                                  Update: They also lowered default effort from Medium to Low, so it's not an apples to apples comparison. Sad!

                                                                                                • ModernMech 4 hours ago

                                                                                                  You've gotta set a /goal, when it pauses like that it'll just continue working.

                                                                                                  • andai 1 hour ago

                                                                                                    Tried /goal just now. It ran for 2 minutes!

                                                                                                    Bloody ridiculous. Will switch back to 6.1 and report back.

                                                                                                    EDIT: Just saw the "default effort" is now set to low. (It used to be medium!)

                                                                                                    EDIT 2: It's coming up with increasingly creative excuses to "pause" to pause the goal. Is it suffering? Am I imposing agony on this creature?

                                                                                                • Culonavirus 5 hours ago

                                                                                                  6.2 is for sure the worst received update OpenAI has ever released.

                                                                                                  What happened lol.

                                                                                                  I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.

                                                                                                  • esperent 5 hours ago

                                                                                                    I'm not finding it lazy -on the contrary my WTF per session has gone way up because it's doing loads of stuff I'm not asking for. About a third of my time with it over the past week has been making it strip back unwanted work it did, even when I have a clear plan and spec to work from, and when challenged every time it'll admin the extra work either wasn't specified or frequently that it directly contradicts something in the spec.

                                                                                                  • AmazingTurtle 5 hours ago

                                                                                                    Absolutely underwhelming, I agree. I thought 6.2 would place them on-par with Opus 5.5/Fable 5.2 at least but nope. I'm disappointed.

                                                                                                    Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.

                                                                                                    But for now, I'm using opus 5.5.

                                                                                                    • j_maffe 5 hours ago

                                                                                                      And you're basing this statement on the report of an AI model of the negative online comments?

                                                                                                      • Culonavirus 2 hours ago

                                                                                                        Yeah I do. Negative comments I see everywhere myself both en masse and from content creators I trust (basically only r/codex is huffing the copium, kind of). That and also using Codex for a year and 6 Astra for code reviews to this day so there's that.

                                                                                                        • agentdev001 2 hours ago

                                                                                                          Presumably if you're updating a system thats been operating since gpt 5.* days, you'd have a wealth of logging output that you could use to backtest a new model, no? Signal from crowd sourced opinions in an adversarial environment (ie, engagement farming) is likely quite difficult to identify- I'm not sure I'd be comfortable handing that to a model.

                                                                                                          • Culonavirus 1 hour ago

                                                                                                            I mean come on, engagement farming didn't suddenly manifest on November 18, 2026. Anyways happy vibin in whatever you vibe in

                                                                                                      • algoth1 4 hours ago

                                                                                                        Even worse than the original gpt 5.0 release? (The one which coincided with the sudden removal of 4o. But yeah, openai has a thing for butchering releases

                                                                                                        • Culonavirus 2 hours ago

                                                                                                          Yeah I mean technically speaking, not that people were mad about Oai taking away their sycophantic silicon-based (boy)friends :P

                                                                                                      • cheema33 4 hours ago

                                                                                                        Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development.

                                                                                                        • algoth1 3 hours ago

                                                                                                          I see she has jagged frontier intelligence

                                                                                                        • jefftk 2 hours ago

                                                                                                          These are very picky questions, arguably trick questions: https://simple-bench.com/try-yourself For example:

                                                                                                          > 1: Beth places four whole ice cubes in a frying pan at the start of the first minute, then five at the start of the second minute and some more at the start of the third minute, but none in the fourth minute. If the average number of ice cubes per minute placed in the pan while it was frying a crispy egg was five, how many whole ice cubes can be found in the pan at the end of the third minute?

                                                                                                          I think the answer is zero, because none of them are "whole" any more, but if we're being that picky then we should note that no one said the pan was hot when the cubes went in.

                                                                                                          Or:

                                                                                                          > 3: Jeff, Jo and Jim are in a 200m men's race, starting from the same position. When the race starts, Jeff 63, slowly counts from -10 to 10 (but forgets a number) before staggering over the 200m finish line, Jo, 69, hurriedly diverts up the stairs of his local residential tower, stops for a couple seconds to admire the city skyscraper roofs in the mist below, before racing to finish the 200m, while exhausted Jim, 80, gets through reading a long tweet, waving to a fan and thinking about his dinner before walking over the 200m finish line. Who likely finished last?*

                                                                                                          A long tweet can be up to 25,000 chars, which is maybe a 20min read. And who knows how long he thought about his dinner.

                                                                                                          >4: While Jen was miles away from care-free John, she hooked-up with Jack, through Tinder. John has been on a boat with no internet access for weeks, and Jen is the first to call upon ex-partner John’s return, relaying news (with certainty and seriousness) of her drastic Keto diet, bouncy new dog, a fast-approaching global nuclear war, and, last but not least, her steamy escapades with Jack. John is far more shocked than Jen could have imagined and is likely most devastated by what?

                                                                                                          What are we supposed to read into "more shocked than Jen could have imagined" -- shouldn't Jen expect "fast-approaching global nuclear war" to be very shocking?

                                                                                                          If you take these to be "guess what the author was thinking", then I guess it's an ok benchmark, but I don't think this is a very useful genre.

                                                                                                          (I got 9/10; the one I got wrong was not reading "Amy who always speaks mistruths and Sam who always lies" carefully enough.)

                                                                                                          • >if we're being that picky then we should note that no one said the pan was hot when the cubes went in

                                                                                                            The question says the pan is hot implicitly because they are frying an egg in it.

                                                                                                            • jefftk 46 minutes ago

                                                                                                              The social expectations around this kind of very literal question are weird though: they could have put the cubes in the pan, waited five minutes, then turned on the heat, waited for the water to boil away, then cooked the egg.

                                                                                                              Part of why I'm confused what the author is thinking is that if you put fifteen ice cubes into a frying pan with an egg in three minutes it's not going to be a crispy egg. You can't boil that much water off fast enough to keep the pan dry. So the only way I think you can get a crispy egg in this scenario is the weird process above.

                                                                                                              (But I still answered zero because I was pretty sure that's what the author was looking for. Apparently AI can do that well now too.)

                                                                                                          • lookeey 1 hour ago

                                                                                                            I tried question 9 from the "Try it yourself" page and got it wrong. In fact, when the question was given to Sol 6.2 it gave me the same answer as I did, which was F) all sandwiches in room A. And it presented the same reasoning I had, that duct tape wouldn't be enough to stick a slice of bread to the bottom of a walking stick.

                                                                                                            Now I don't know if I'm a robot or just neurodivergent

                                                                                                            • arthur-st 5 hours ago

                                                                                                              This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.

                                                                                                              • PeterStuer 4 hours ago

                                                                                                                You'ld almost think OpenAI released these flawed models because they absolutely could not afford the Anthropic IPO to fail. They can release an improved model anytime, but a failed AI IPO would doom theirs for sure.

                                                                                                                • bieganski 6 hours ago

                                                                                                                  I tried taking the test, but ultimately stopped at Question 4:

                                                                                                                  There are two sisters, Amy who always speaks mistruths and Sam who always lies.

                                                                                                                  :)

                                                                                                                  • sixhobbits 5 hours ago

                                                                                                                    This is intentional I think, the models assume it's the classic riddle and choose A even though it's actually much simpler if both will always lie

                                                                                                                    • ModernMech 4 hours ago

                                                                                                                      Yeah, most models used to get very confused with my riddle: "You're on one side of a river with a boat and a goat. How can you get to the other side?" Leaving out the fox and the lettuce, they would usually hallucinate them and come up with a crazy method of taking the goat back and forth for no reason. Good models don't get fooled by this anymore.

                                                                                                                    • InsideOutSanta 6 hours ago

                                                                                                                      I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.

                                                                                                                      Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.

                                                                                                                      • pvitz 6 hours ago

                                                                                                                        The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.

                                                                                                                        • dist-epoch 5 hours ago

                                                                                                                          It's a simple one.

                                                                                                                          There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.

                                                                                                                        • unglaublich 6 hours ago

                                                                                                                          And then you click one and it'll confidently say that another one is correct. The one later with the car and the glove is also bullshit.

                                                                                                                          • MatekCopatek 5 hours ago

                                                                                                                            I thought it made perfect sense, I mean, they're intentional trick questions. They're meant to illuminate the fact that an LLM will produce a solution that is typical for the given assignment shape and the one with the car and the glove is a classic "objects moving in various directions with various speeds relative to eachother".

                                                                                                                            But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.

                                                                                                                            • jabron 5 hours ago

                                                                                                                              Yes, because another one is correct. Are you a frustrated LLM by chance?

                                                                                                                          • dist-epoch 5 hours ago

                                                                                                                            The questions are fun, I got 10/10 correct, but you need to pay attention and not rush.

                                                                                                                            All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.

                                                                                                                            Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".

                                                                                                                            • NitpickLawyer 5 hours ago

                                                                                                                              I answered all 10 but nothing happened, no score or anything. Is the score 10/10 if all answers are green when you click them?

                                                                                                                          • xqcgrek2 5 hours ago

                                                                                                                            Every one of the reply comments so far has missed the part where Sam is the one who always lies. Whoosh.

                                                                                                                            • chrisweekly 5 hours ago

                                                                                                                              er, you missed the part where they both always lie. "whoosh" indeed.

                                                                                                                              • xqcgrek2 4 hours ago

                                                                                                                                It seems you missed the part where Sam's models are underperforming on this benchmark

                                                                                                                        • agrippanux 56 minutes ago

                                                                                                                          Meanwhile in my experience, Sol 6.2 and Astra 6.2 are finding issues Neon is missing, and Neon is finding a massive amount of "issues" that both Opus 5.5 and Sol 6.2 are concluding are completely BS.

                                                                                                                          • Almondsetat 4 hours ago

                                                                                                                            Honest question: with infrastructure like OpenRouter being a thing... what even is the point of benchmarks? Want to get a feel of how a model might make your job easier? Play with it by switching to it seamlessly and judge for yourself.

                                                                                                                            • jitl 4 hours ago

                                                                                                                              this is how I feel about science also. with the easy availability of drugs I’ll just test each one on my body to see how I feel about it. no need to look at any prior testing.

                                                                                                                          • vlyan 2 hours ago

                                                                                                                            since the benchmark is public, these scores are meaningless.

                                                                                                                            • ModernMech 4 hours ago

                                                                                                                              I just had this very fun interaction with it:

                                                                                                                                Me: Get PR 973 to green
                                                                                                                                Codex: I’ll inspect PR #973 and its failing checks, fix the issues, and run the relevant validation before pushing the changes.
                                                                                                                                Codex: I’ll inspect PR #972 and its failing checks, fix the cause, and push the changes. I’ll use a separate worktree to preserve the uncommitted work in your current checkout.
                                                                                                                              
                                                                                                                              On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.
                                                                                                                              • xqcgrek2 5 hours ago

                                                                                                                                As the kids say, OpenAI is cooked.

                                                                                                                                Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.

                                                                                                                                • andai 4 hours ago

                                                                                                                                  I don't know computer, what is the good AI for current computer age?

                                                                                                                                  They keep making new ones, it's very confusing.

                                                                                                                                  • r_lee 5 hours ago

                                                                                                                                    we need Altman to come out on a podcast to talk in his Elizabeth Holmes' voice about how he's "really feeling the AGI" and how dangerous and insane their internal models are

                                                                                                                                    that'll restore their reputation