Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant.
My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review
They do catch important things but it’s really contextual. You can’t grab a model slap it on top and say code review . Hence a dedicated review tool is almost dead . Code review should be part of your pipeline and consume test results from the original task , open spec etc . If you do not have that code review will not help if you do , what is the point of task rabbit just slap <your harness in the sandbox> review against <goal>
Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. I also still manually review code, and usually also catch issues, but the severity of what I find shrinks ever further as agents get better.
The sprawling code comments are becoming the most draining part of code review though, that's really killing me from the inside.
> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly.
No, none of today's AI would give you enough signal around "should this thing be built in the first place" nor if it's the correct solution on a high level.
They don't understand why you are doing what you are doing, and even if you explain it, they still don't actually understand the motivation and lots of other things.
You'll get them to do guesses and pretend they actually know how to prioritize and will tell you it makes lots of sense, whatever they come up with. But try following it blindly and you'll see where you end up.
The AI review are still quite far from having the same level of critical thinking and high level knowledge of your application, what you have done in the past and want to do next etc.
If you don't master this for your own project, what's even the point of your job.
That's a thoroughly uncharitable view. Especially in smaller orgs with a minimum velocity dictated by the company's need to survive, the amount of code required to be written just to keep up with your competitors is massive. Trying to review that all by hand, thoroughly, is draining, thankless, and tedious. You end up with a few fast movers producing most of the code, and some slower movers forced into a reviewer role they never signed up for. It's an unhealthy dynamic.
In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way.
Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC backend. We do not quite know what do to about it.
I know we are not the only ones in the situation. What's your experience and context ? What do you do ? What works for you what doesn't ?
We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.
Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.
The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.
Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)
I have AI confirm the logic works as expected, but review for system design.
Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself.
That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on
Generally speaking 'the Fable/Astra built GTA 6' videos are a new phenomenon, so it's clear these models have new capabilities and people will need new ways of interacting with them if they want to leverage these imo.
That likely won’t change if other competitors don’t take the lead at some point. If companies are willing to pay top dollar for the best models AND they get to extract as much money from Chinese labs distilling Astra/Fable it makes no sense to lower prices. Obviously not great for everyday users who don’t have unlimited money.
That should be expected based on the scaling laws that we expect; larger models are more intelligent and cost more. Now it's very unfortunately that they don't publish the size of their models.
Astra seems to be really slow. Maybe it intends to read more context. But from my experience it is definitely slower than 5.6 sol when handling same tasks.
Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant.
My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review
They do catch important things but it’s really contextual. You can’t grab a model slap it on top and say code review . Hence a dedicated review tool is almost dead . Code review should be part of your pipeline and consume test results from the original task , open spec etc . If you do not have that code review will not help if you do , what is the point of task rabbit just slap <your harness in the sandbox> review against <goal>
Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
What really important things are human reviews catching in your org?
I just feel more and more like the effort invested in manual reviews is not worth it
1. Does the implementation fit in the architecture/style of the project?
2. Are there potential security, accessibility, performance, etc. issues?
3. Domain specific knowledge (SQL, ASP.NET, XQuery, etc.) where there are better ways of solving a problem, or possible issues not handled.
4. Sense checking ... is the code easy to read? does it need an explanatory comment? does it need named parameters? etc.
1. Whether the thing should be done in the first place
2. If it's the correct solution on a high level
3. Whether it conflicts with or duplicates other parts of the system
4. Whether the comments are actually useful or restating the LLM chat
Also many others but these are the most common IME
All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. I also still manually review code, and usually also catch issues, but the severity of what I find shrinks ever further as agents get better.
The sprawling code comments are becoming the most draining part of code review though, that's really killing me from the inside.
> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly.
No, none of today's AI would give you enough signal around "should this thing be built in the first place" nor if it's the correct solution on a high level.
They don't understand why you are doing what you are doing, and even if you explain it, they still don't actually understand the motivation and lots of other things.
You'll get them to do guesses and pretend they actually know how to prioritize and will tell you it makes lots of sense, whatever they come up with. But try following it blindly and you'll see where you end up.
The AI review are still quite far from having the same level of critical thinking and high level knowledge of your application, what you have done in the past and want to do next etc.
If you don't master this for your own project, what's even the point of your job.
Code review tools are designed for less organised dev teams that don’t do PRs and mandatory human reviews already.
It is papering over a lower level of competency without having to invest in actual human oversight or real process improvement.
That's a thoroughly uncharitable view. Especially in smaller orgs with a minimum velocity dictated by the company's need to survive, the amount of code required to be written just to keep up with your competitors is massive. Trying to review that all by hand, thoroughly, is draining, thankless, and tedious. You end up with a few fast movers producing most of the code, and some slower movers forced into a reviewer role they never signed up for. It's an unhealthy dynamic.
It does add lot of noise after a point you start ignoring the suggestions and findings.
Code generated these days with fable and sol are near perfect. What issues they might have is logical errors.
You have a very interesting definition of "perfect", then.
> Code generated these days with fable and sol are near perfect.
If you're doing a simple CRUD app, sure.
If you're doing anything more involved they get the job done with dozens of shortcuts that bite you in the ass the moment you have on-call duty.
Way too much code and repetition and hacks.
Especially in GPU code, but also in other fields.
How do you guys review AI-generated code ?
In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way.
Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC backend. We do not quite know what do to about it.
I know we are not the only ones in the situation. What's your experience and context ? What do you do ? What works for you what doesn't ?
We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.
Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.
The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.
throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it.
have him start with an overall design doc if his change is 15k, it's definitely worth a design doc.
and then have his contributions reviewed in pieces of 200-300 LoC PRs.
any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to review it
Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)
I have AI confirm the logic works as expected, but review for system design.
Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself.
That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on
Both OAI and Anthropic seem to have released a model that is slightly better but cost ~2x the previous iteration. Interesting play
Generally speaking 'the Fable/Astra built GTA 6' videos are a new phenomenon, so it's clear these models have new capabilities and people will need new ways of interacting with them if they want to leverage these imo.
That likely won’t change if other competitors don’t take the lead at some point. If companies are willing to pay top dollar for the best models AND they get to extract as much money from Chinese labs distilling Astra/Fable it makes no sense to lower prices. Obviously not great for everyday users who don’t have unlimited money.
That should be expected based on the scaling laws that we expect; larger models are more intelligent and cost more. Now it's very unfortunately that they don't publish the size of their models.
Astra and Sol are the same price when you factor in token efficiency
I don't know, in the Codex app, it burns the limit much faster.
Bullshit.
Roughly how we price (high skilled) human labor.
Interesting comment because it is true that Astra is costlier for the same intelligence tasks as Sol.
But this is not the same for Fable at all.
Astra seems to be really slow. Maybe it intends to read more context. But from my experience it is definitely slower than 5.6 sol when handling same tasks.
It's a bigger model, of course it's slower.
Given that Fable is a Sol-class model, should Astra not be compared to Mythos in those tests?
From what I can tell, its shit.