Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together
> The main way they can surprise you today is pulling in information that’s new TO YOU
Many comments here are speaking in absolutes and being myopic about what's "novel". The truth is yes, LLMs output is essentially derivative, but it's also true that a lot of what we consider novel is also derivative. The two aren't mutually exclusive at all.
Had a novel idea? You didn't get it from thin air, everyone is standing on the shoulders of everyone who came before. LLMs most of all, but that doesn't mean it can't bring disparate bits of word-encoded knowledge together and burp out something we'd consider "novel" if a person had made the same connections.
That was actually the basis of some of the mathematics proofs. It made connections from existing material. Nothing a human couldn't do looking at the same material, but we're talking speed and scale here - same as all computational processes.
A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results. The only difference is LLMs are intentionally nondeterministic, as that behaviour can be usefully harnessed, pun intended.
So there's no need to argue about what's novel or not. LLMs can make novel connections via raw speed and breadth of disparate data. The human brain is more complex and "imaginative" for whatever your definition of that is - I mean fundamentally one either believes we are just physical neurons, or that there's a spiritual aspect on top, the end result is the same.
A six sided dice can also be novel if it happens to unexpectedly roll your phone number. It's a novel occurrence, same as an idea coming to mind that, TO YOU, seems novel... but we can't ask the other 8 billion people on the planet so who are we to judge what's a novel idea anyway?
This is a common rebuttal. But it falls flat, in my opinion, because LLMs appear to be a compression of language-based knowledge, and that compression is lossy. The result is that "knowledge" retrieved from the language-based storage model is biased towards the "average" idea.
As a concrete example, when you ask Claude Sonnet 5.5 to generate a small-to-medium-sized one-shot Python project, it will always fall into the same code patterns. Same structures of functions. Same file layout. Same package layout. They're not even good patterns, because if you throw a few common linters at it (e.g. flake8, isort, mypy) there are a ton of linting errors to clean up.
The only way that it escapes this mediocrity is through context. If you give it enough guardrails or sample data (e.g. Python code to emulate), it will start to go off in a "novel" direction that is defined by that context.
Now, what's the most likely way that "novelty" will be generated in an automated system like this? Randomly pulling in context and going for a random walk through the compressed language-based knowledge storage. It remains to be seen whether that's an effective way of producing more than just parlor tricks for investors. Human+LLM is overwhelmingly more novel and effective than only LLM.
This doesn't even address the direct point. The question is how any of this is fundamentally different (difference in type and not just of "scale") from how human cognition works.
Well, I've addressed it implicitly. Every human collects different context by living one of 8.3 billion+ unique lives. We don't have 8.3 billion+ different LLMs. The only way to simulate that is to perform a random walk through the compressed language-based knowledge storage by injecting random context.
Also, those 8.3 billion+ unique lives already produce contexts that are highly optimized for solving human problems that modern humans experience in the physical universe most relevant to humans.
All knowledge storage and retrieval is lossy. Oral histories are lossy. Lectures are lossy. On on one mentorships are lossy, writing or reading a textbook is lossy.
Agree with this as language use is inherently an attempt to compress observations into a shared semantics
The syntax of math is overloaded so you need less syntax to write out something that would be verbose in English ...no one wants to make sense of arithmetic expressed in German syntax. May as well write all software in brainfuck.
Written literacy on a core language like we focus on today is not the historical norm. Creoles and local dialects have vanished in the US as we hyper-normalize everyone into Children of the Corn like carbon copy zombies of character traits
If you want to argue, as I think you do, that there are no new ideas other than new combinations of old ideas, then you have to explain where the old ideas came from. It can't just be combinations all the way down. At some point, some initial set of ideas must have been created where none existed before otherwise we would have nothing to combine. After all the combinatoric closue of ∅ is ∅.
Ultimately any claim that there exist only combinations of old ideas must tackle this chicken-and-egg problem convincingly. For example- explain art. There was no art until it appeared fully formed and out of nowhere in early anatomically modern humans' dwellings (Neanderthals also had art but it was simpler and it is subject to the same chicken-and-eggness). So where did art come from? What pre-existing ideas were combined to bring it into existence from non-existence?
Btw, since I've had this conversation before- if the answer is that we looked around us in the world and saw things; yes, but we didn't see art. Where did art come from? Where did the idea come from of abstracting shapes and colours so that they resemble the shapes and colours of existing things, without actually being those things, or being composed of pieces of those things? How about numbers? How about writing? How about all our technology and science? Where did those things come from? What did we combine to create them?
Obviously I think that the argument that there are only new combinations is wrong: we are a creative species. We can create new things. Somehow. It would be very interesting to understand how. We have not yet created machines that can match us in that and we can't understand the "how" by studying the machines we have now. By denying the mystery of this our ability, by denying that it is a mystery, by dismissing it and accepting any facile explanation that presents itself in our current era -it's god, it's nature, we're like LLMs- we are making ourselves a great epistemic disservice, denying ourselves the chance to find out what it all means, to understand what kind of creature we truly are.
> then you have to explain where the old ideas came from
Its fairly plausible that the original ideas could be sensory gradient-triggered impulses in simpler creatures' proto-minds that (physical and cultural) evolution has gradually built up through more complex neural machinery supporting more complex representational language.
> where did art come from
Similarly, its not a stretch to see art as built from earlier attempts at representation in physical space - starting with mundane attempts at information exchange by colony insects or pack creatures. The breadth and depth of expression has evolved alongside more complex neural machinery and greater variety in media through technology. There's a hidden lineage in creatures' expressive capabilities that may not be evident when we look at a Caravaggio today, sure.
There's nothing terribly chicken-and-egg about these issues (though the resolution is the same - "chicken" and "egg" co-evolved through iterative selection pressures, one didn't precede the other).
All that said, I concur that its a fantastic and mysterious topic to unpack, but it doesn't follow for me that humans are anointed as uniquely "creative" or that there's necessarily something metaphysical at play.
Truly new ideas are, say, 0.01% of all 'novel' ideas, with the rest being rehashing or recombinations. And even then, the truly new kernel of the truly new idea is some small fraction of the whole thing. Like Einstein's theories of relativity, they were truly fresh, but mostly had to recombine existing work in 'novel' ways.
So it's not there are zero new ideas, but so few compared to the whole of the intellectual domain, that we can say 'there are no new ideas under the sun' and be basically correct to within a small rounding error. And AI is sufficiently ingenious to serve the needs of the vast majority of humanity, even though we'll still need to find/cultivate/inspire some new genius humans if we want to have more truly new ideas in the future.
>> A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results.
Actually, we do. We do dismiss calculator results. We are all generally aware of a calculator's abilities. We know it has limits. So when a calculator outputs 9999e999999 we know to distrust that result. It was something out of the calculators concept of understanding. So too a calculator's version of Pi. "3.14159265359" is not Pi and every user knows it is an approximation not be used for anything too important or precise. The same approach should apply to LLMs... but their limitations remain nebulous and ever-changing.
Most calculators use 14-15 digits of pi even if they only show 10. It would be hard to come up with a case where for single calculations this was not “sufficient”.
I agree with your opinion. The definition of novelty largely depends on subjectivity.
Results produced by combinations of LLMs can also be novel.
Combining multiple insights, imitating them, and then organizing them into one can also be considered sufficiently novel.
LLMs have a kind of bias due to RLHF, and that may make them uninteresting to humans, but when a "human" uses them, they can enable the human to do more, and that greater amount of work can create new novelty.
It's like reading a news paper, when it's about your field of expertise you find lots of errors, but when it's about something you don't know you take it at face value.
Strong agree. My 30 years of experience building computers and breaking them has made me better because I don’t trust my knowledge or myself as a first principle. If I didn’t have that and I could easily see myself in an alternate reality, coming in late to the game to leverage ai to build a car out of seatbelts; harnessing all the power of 1000 suns seeing that idea to fruition.
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. —Michael Crichton
Edit: sibling comment explains it much more plainly.
I have noticed that several times with a newspaper I had at the time. It seemed to be a generally good and reliable newspaper, and one of the biggest in the country, but every time an article was about something that I actually knew about, it was wrong, which made me doubt the validity of everything else.
I now have a newspaper that doesn't give me that feeling.
I've come to the conclusion that they know a lot, but they're not very smart. Not even the best ones.
They know far more than any human ever could: they've read all the books. Their knowledge is enormous. And that often includes how to apply it in standard ways, and sometimes in slightly non-standard ways. But whenever something gets complex, detailed and nuanced, they can't keep up with a human. It becomes a shallow illusion that's easily punctured by just a bit of critical thought.
You mention system architecture, but even for writing code, the code it writes is often horrible. It makes it work because it has a feedback loop that rapidly tests it and ensures it will work, but the actual code it writes down is often not something I would feel comfortable showing to another programmer.
Strong agree, but with a major caveat: First since we humans don't have a memory the size of the internet, new TO YOU is extremely easy to find. And second: a lot of discoveries can be made by trying to match two different domains and generate new insights at the seams.
A lot of math theorems are discovered that way by humans btw: find a difficult problem in a domain, map it to another domain, apply existing theorems there, go back to the original domain. The only difference is an LLM would have to search for an interesting problem, but that can be done by brute-forcing search at the seams and validating the problem does not previously exists. And there is no new TO YOU for the LLM: it can get help from a vast amount of helper domains.
IMHO there's a shot at having category theory really unifying a large chunk of maths in the next decade. It'll be a big ugly ball of Lean, but a queryable one at that.
This is not novel. Computers have been generating mathematical information that would be resource-prohibitive to produce by hand since the dawn of computing.
The ingenious aspect seems to be that we've found simpler ways to apply computing power to a broader range of problems, which it's important to note is as much a testament to the developers of proof assistants like Lean as it is to LLM developers, to say nothing of the countless mathematicians over many centuries that contributed training data.
Whether this will lead to many revolutions, or a handful of impressive results followed by a long dark spell remains to be seen.
LLMs are like an athletic child - they can do some basic moves, but in general they're uncoordinated.
Occasionally they do something that surprises you. If you put them inside a search loop and spend huge sums of money you'll get some nuggets of value, but for most hard problems the cost isn't worth it.
And they work best with a lot of steering.
From that POV they're exactly like human culture. Most scientific papers are trash, a few are gamechangers. Same with the arts, etc.
But we've gone from 'Can write sentences' to 'Can do frontier math problems' in maybe five years.
We absolutely do have AGI now. It's not very good AGI, but it's probably better than a lot of people were expecting.
Being better than humans at everything isn't AGI, it's ASI. A lot of the comments here seem to be lamenting that we don't have that yet, which is an interesting data point about how far the goal posts have moved.
Edit: also it makes things like this possible. So...
if you search papers with it, and from there search related papers and have it do literature reviews, it comes up with very useful white areas uncovered, and from there problem statements and experimental designs to uncover the white spots.
it is nothing but combining known knowledge that begets new.
Unless LLMs drastically shift incredibly quickly, said declaration is likely to hold, because of how LLMs work. I mean, they've worked the same way at their core for several years now.
They predict what comes next and are very good at it. That's all intelligence is. What comes next in a pattern? What will my senses perceive next when I drop this ball? What is the next hypothesis to test after I made these measurements in the previous experiment? What is the best next move in this game after the current trajectory?
> by its nature it doesn’t have a “vision” of how the whole system fits together
I think a harness for building hierarchical summaries with tooling for expanding and searching history is what's missing for this. We're getting there with Recursive Language Models and continuous compaction systems like the one NVidia just published:
This stokes my suspicion that effective LLM input for software development will ultimately resemble more of a higher level programming language than the natural language in common use today, and that the design of such languages and related tooling will play at least as large a role as the models themselves.
which is actually great, because LLM so far does not show true AGI ability, they are not innovative. Humans are still needed so everyone still have a job. And you can delegate the stuff that you need to be careful with, or attention to detail stuff, to LLMs
On the other hand, there's no evidence that work will dry up merely because it becomes more efficient, or even if it does, that this cannot lead to a better world. The trick is finding a way to prevent rapid changes in the nature of work from destroying too many lives, and that's a very human problem machines alone can't solve.
I'd argue it is the other way around - LLMs are great at pure creativity, creating something that never existed before, but not always at "pulling in information that’s new TO YOU". I use them for entertainment - jokes, short flash fiction etc. and they are excellent at it.
There's no such thing as "pure creativity," which I presume you mean "creative output not directly derived by prior inputs." The very concept is utter nonsense.
They can be quite good at taking knowledge from one domain and applying it to another. This can give new perspectives on things, but it isn't explicitly new information.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
I wonder how much different that is from humans, even professional comedians. You don't usually get to see all the jokes that comedians try that don't work, unless you're an avid attendee at comedy clubs.
For comedians, perhaps. But there are plenty of humans who aren’t very good at comedy.
In these LLM comparisons, people tend to compare them to the best humans, not average or below average. It’s a kind of motivated reasoning, with the goal of deprecating the abilities of LLMs.
Well, you did ask for a comparison to professionals.
But I think humor is pretty easy to rate for a human. Did it make you laugh? Yes? Then it might be a good one. This simple human filter radically improves your efficiency as a joke validator, leaving any LLM far behind.
For pictures I'm not sure if humans are much better. For music, yes, after a few listens (recognizing a hit usually takes time). For games, definitely yes.
Exactly. If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming. Even for 15 line snippets, if you're very familiar with the programming language, you'd have written it differently.
An LLM is like a low IQ person with vast knowledge.
I have a PhD and Opus 5.5 came up with a novel solution to solve a problem in my area of research that I, and other experts in the field, had been unable to. This was from very general prompting it to find improvements over published methods.
It came up with a way to efficiently evaluate the reliability of certain types of complex system using generating functions (https://en.wikipedia.org/wiki/Generating_function) to split the computation, which was far to large to solve with existing methods, into many embarrassingly parallel smaller problems and then reduce it back to a final result. The solution involved several distinct new ideas for it to work, really mind-blowing it was able to come up with it. The other cool thing was that, in the process and unprompted, it implemented code to validate and benchmark against existing approaches and known results just from the pdfs of the published papers (none of which gave reference implementations). In my research it would often take weeks just to replicate a single result from a paper.
Humans test higher on IQ tests if they had previously taken IQ tests of similar types too tho. And LLMs are trained on IQ tests. IQ tests only really hold their value when the test is fairly novel to everyone taking it.
Said tests are often quite inflated, quite frankly. If you can give me a source that disproves this claim, I'll readily eat my words, but I readily believe that those tests are just as inflated as all other claims about AI.
They're solving them in a way that isn't particularly interesting because it's near impossible to read and understand, and follow, what the LLM did. It's this way not because it's so smart, but because it's so stupid and still reached the goal by pure force and effort.
I don't know how to communicate this, but what you are saying is almost entirely wrong.
People can follow what they did. The documents OpenAi produced in their big drop are hard to read because they are poorly written in weird ways, not because they are beyond human intelligence. They didn't brute force in the normal sense. AI doesn't get tired, so it can try 1000 promising ideas, but it's like 1000 Ph.D.-level mathematicians trying different ideas, not 1000 calculators trying all combinations.
The biggest results -- quasi-Riemann, no Siegel zeros, and Unique Games Conjecture -- have already been digested by experts. No Siegel zeros in particular already has like five different write-ups, and Tao has written a exposition on his blog.
So did we get any new mathematics out of this, or just solutions to problems without new techniques or insights? I haven’t been able to keep up with this or read the writeups you mention, genuinely wondering since this seems to be a central concern of the math community
It's only been a month and people aren't that fast, but I think yes. Analytic number theory and complexity theory were stuck on the lack of progress on certain questions, that is now unstuck.
I am sorry but your argument looks to contradict itself: They solve hard mathematical problems because they are stupid and use brute force? Then why a human skilled in mathematics guided them for many turns in their conversation?
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
Can't you kind of think of them as a genetic algorithm that has a very good hypothesis generator? Humans could, in principle, help guide that (already very impressive) hypothesis generator to make better guesses to find the proof more easily.
I think it does gel. You're right that the comment you're replying to is weakened by that fact.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
and I'm guessing it was with "tool calling" ie. they can write a bunch of Lean proofs and hit compile and basically brute-force a whole bunch of hypotheses quickly?
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
In my experience they write an informal proof first, check it without Lean, and only then turn it into Lean code. That last step sometimes takes them much longer than checking their proof without Lean, which is a problem humans have as well. What they're truly bad at, however, is explaining their proofs. Until they improve at that, we'll have to keep resorting to Lean to check whether they solved something correctly.
I'm less sure about the following, however: I think that while current LLMs are bad at explaining their own proofs, some models are great for discussing older, human-written proofs. For this use case, from best to worst, Gemini > Claude > GPT-Astra >>>>>>>>>>>> GPT-Sol.
I definitely don't know but I got the impression it's the opposite. Based on the observations about Navier Stokes and the natural language proof not matching up with the lean proof and other things. They generate lean and convert it into natural language at the end. But really I could be wrong just giving anyone interested a lead to check out.
Look cmon, this is pure cope. The problems it has solved in TCS are far from trivial, with plenty of smart people making their careers in trying to solve them. The techniques these LLMs are using are non-trivial, and not just brute force.
I'm more interested in the techniques used to generate these proofs, and how they can be improved to be as reliable at solving a broader class of problem than already formalized mathematical conjectures.
The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
yeah chain of thought reasoning isnt really a think besides running the model for longer and hide most of the output (but still charging you for it). it literally does not work any differently than "non-reasoning" models (except for marketing)
The “conversation” is just one between some redditors or book publishers.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
All novel insights expressible as text can be assembled from snippets of existing text. The challenge is "only" that you need to find the right snippets. Hence being a stochastic parrot does not represent an upper bound on capabilities, but a suggestion that if you want your parrot to give you a specific answer, you should arrange for a context where it's statistically likely to be correct.
No. In a large enough state space, you can of course easily find a state that has never existed before. For example, if you pick a 1024-bit integer uniformly at random, it will certainly be different from any previous 1024-bit integer. However, it's much more difficult to find a new integer that cannot be assembled out of bits taken from previous integers. Composition from a handful of base atoms can cover very large state spaces, and unbounded composition can cover infinite state spaces. Which is why only being able to combine things that already exist is not much of a limitation.
>It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
ignoring the fundamentally dishonest representation of LLMs in this subthread..
if humanity was given a tool that could restructure already existing and known knowledge and facts into novel results with a statistically better answer automatically it wouldn't be as a minor a win as you make it sound; it would be tremendous.
Firstly, many of them are still being checked, so hold your horses.
What I’ve found LLMs to be OK at when it comes to novel insight is being given a goal to work towards (“prove or disprove this”), with access to a large body of sources and it can then chase down 1000s of options no human would bother with.
This is why security and maths proofs seem to be an area where it can accelerate things. However I know from my own work in these areas it can still hallucinate and so the output needs careful checking.
Is that novel ideation? Or is it self-organising grid search? It’s more efficient than getting 1000 mathematicians to do the work, but it’s not doing something “new”, it’s building on existing work.
And yes, I know what “attention” means, I was making a pun. And it still stands up: they’re not in some deep thought about this and reflecting on new ideas, they’re just hammering away at everything in the literature.
> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
I work in a niche where it is utterly hopeless at this, and everyone in the niche agrees: it produces things that look convincing to non-experts but that is fundamentally derivative dross.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
> It’s great at bread and butter stuff, but not at specialised ideation.
This is fundamentally different from your earlier, stronger claim that - "The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago."
I use a less popular java framework quite a lot, Micronaut (I want to move away from it, but that's a different discussion).
When trying to solve problems with its behaviour, I can watch as Claude digests the framework source to come up with suggestions, identify bugs that are causing behaviour regressions, and generally unlock stuff that I've previously found very frustrating. Frustrating because the documentation covers a tiny amount of what's in there, and frustrating because there's little online discourse about any of it to refer to.
Is this earth shattering in itself? Probably not. But it's certainly better than you can find from "an online forum like Reddit or a book/paper published more than a year ago".
It's also far easier to find information through LLMs for another niche I play in, certificate authorities, than it is on the open web. Try to look up anything to do with certificates and you'll just get guided to endless "Why my certificate not work? Browser says no?" discussions, because actually creating certificate authorites is again very niche , not widely discussed and a lot of what you need to leverage in any given language isn't well documented because "you'll never need to do this". Well actually...
You're the one making the claim, and also claiming that you have specific examples to back up your point. So maybe you should share them, and we can see if your assertions hold up.
The discussion here is talking about helping individual research with proposals a human will read. That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically. There is no "automatic verification" of a system design proposal, and no way to figure out from 10s of millions of proposals which is best automatically, and also assuming people in this discussion don't have 10s of millions of dollars to throw at every design problem.
I guarantee that you could not find a proof using brute force, even with literally trillions of dollars of compute. You're betraying a lack of understanding of the complexities involved, unless you're using a definition of "brute force" which is essentially equivalent to "intelligent search."
Brute force is certainly the wrong way to put it, but that doesn't mean it's easy to distinguish superhuman effort from superhuman intelligence, or, in the words of Thomas Edison, perspiration from inspiration.
Especially as the pure gruntwork involved in acquiring the knowledge required to merely understand the details of the mathematics involved represent at least a significant fraction of a human lifetime.
And unless this human process of knowledge acquisition continues, eventually no one will be left to understand and attach any significance to the results machines produce.
> That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically
It’s kinda like the project where someone generated a lot of midi files for all possible pop riffs to copyright them. If you have a start and endpoint, rules for navigation, and unlimited computes, you can brute force anything. The nice stuff is when you made exploration efficient.
Other than for maybe Navier-Stokes, they just didn't brute force it in the way you guys are thinking. Most unsolved math problems are not susceptible to brute force, which is why the problems remained unsolved. AI is so quick it can of course try more ideas than a person, but each idea is executed at genius IQ level. Even if you knew the broad outlines of the proof of quasi-Riemann, there are probably only 20 people in the world who could have executed it.
This is trivially true for any theorem with a proof that can be automatically verified in finite time, but does not preclude the universe lacking a suitable combination of monkeys, typewriters, and time to do so.
>That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically.
That is not remotely how the vast majority of the proofs have worked. You don't know what you're talking about, but that seems par the course for these discussions.
>The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
The LLMs are vastly superior for any subject matter with even the slightest hint of danger or otherwise high stakes because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.
Like I'm asking you about sheetmetal thickness vs joist spacing for re-skinning a goddamn carport with roofing steel, read the room, this is not the time to whine about permits and snow loads and whatnot because the project is happening with or without your advice.
And it's not just mundane stuff. Some guy who's retrofitting new chinesium controls onto 50yo industrial equipment that had a prior generation of controls cobbled on 20yr ago doesn't need to get talked down at about stainless on stainless corrosion when he's asking a question about a solenoid or something. If the LLM talks down to him he can brush it aside. If he tries that on reddit they'll dig in their heels and fling poo.
Cutting out the "negative value" middle man of having to painstakingly make the case about the validity of your big picture goal before getting answers to simple questions is of substantial value. Stuff that people would've painstakingly researched themselves rather than ask Reddit or HN or Stack Overflow because dealing with those people is so onerous they now just punch into their LLM of choice and get the best of both worlds.
This matches my instincts too. When trying to debug ML learning pipelines or research projects, LLM suggestions I've received are almost always techically correct, but they point out small, tedious details which don't really matter. In my experience, they miss the forest for the trees.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
As a fellow machine learning practitioner, I don't think the OP's issues are specific to machine learning as much as they are more generally about system design and context. Much of my job as a data scientist is gathering more context about what business value a model is actually trying to provide, and it's difficult if not impossible to encode all of that context into a text format that's digestible by LLMs.
For example, there's a model we have in production whose sensitivity has to be balanced based on which of our customers / partners is consuming its inference. It's a cat-and-mouse game where we are increasingly taking into account additional features to use in the model based on feedback directly from these customers and partners. Adjusting the model based on this feedback is much more art than science because there's really no "correct" adjustment to make, just an adjustment that produces inferences that convince our customers and partners that their concerns have been addressed.
Sure, we could encode every conversation that we've had both internally and across the internal<->external interface, both formal and informal, as text that an LLM could consume. But we find it much easier to just have a human (me, or one of my teammates) do the same thing and then guide an LLM to design a system that satisfies the requirements collected from those conversations.
--
That being said, I'm equally amazed at the quality of suggestions that LLMs sometimes come up with for improvements to systems as I am dismayed by the complete dead-ends it sometimes hits when it doesn't have a good solution in its knowledge base. It's not limited to just ML topics, but also very esoteric things that have very little documentation or even contradictory documentation!
So yes, you're right that if the frontier model providers aim their reinforcement learning laser beams at improving a specific topic they will probably succeed. But, no, I don't think there's a panacea.
It's very possible I'm wrong. I think that what we're seeing in maths is that RL is training LLMs to solved well-posed problems, especially when they can recruit existing ideas from across the literature. I don't want to discount that - it's clearly powerful as we've seen in maths. But in fuzzier research fields, the right question is often more important than the solution - I'm not convinced LLMs are there yet, and I don't think this problem is as amenable to RL as problem-solving.
This recent work which tried to get LLMs to reinvent methods they haven't seen before (given the same problem setting) also raised some of the same issues I've noticed in my own usage: https://epoch.ai/publications/innovationeval
If not ”missing the forest for the trees” is a requirement for debugging an ML pipeline then that offers a path to building such a large machine-verifiable dataset, does it not? That’s what labs are currently doing as I understand it.
I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
That's a kind of bitter lesson thinking, that we take this generalist model, which already mastered everything from English to Javascript with unsupervised training, and stuff it inside an expert system so it can do long division for it. That's much similar to the last AI winter, when the researchers realized their logics couldn't cope with certain logical scenarios, but still thought they could salvage their flawed reasoners as dispatchers to specific algorithms.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
You can try adapting to this reality in your methodology though. Divide up your project into clear cut and ambitious modules, with well-defined objectives to evaluate each part.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
Exactly, I agree. I have been working on a ux harness that creates a deep product model with personas, user journeys, etc as well as designs. Originally I was hoping it that it could work just some skills and a data export format. It was burning 300M tokens just to get nowhere.
Now I have this DAG with something like 40 specialized subagents. It's sort of a monstrosity, but it works much better for 100x fewer tokens. I've found that basically every single judgment call (which is characteristic of ux) really wants an agent solely focused on it with plenty of concrete examples.
I don't think its the frontier models, my opus 5.5 has no problem working on a project fairly autonomously, for example if I prompt it in the evening it will work the whole night without issues...
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
The thing is, without doing the grunt work you usually don't have enough understanding to answer the big questions.
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
The proper term is: "On the shoulders of a very large pyramid of hobbits." The concept is the very core of human intelligence, also known as cultural evolution. I can't resist to recommend Joseph Henrich's book "The Secret of Our Success" every few years here, to anyone interested in human intelligence, and how it is different from other primates.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
True. History of Science isnt part of most curiculums. So most people dont see the path and time it took to work out why the magnet points north. But LLMs can actually show you the whole path and a map of different paths people have taken.
Using Fabel to improve some non-AI ML models a while back, I had a similar result - it could implement perfectly any suggestion I made, but it could not work autonomously due to making very poor suggestions itself, or getting distracted by the wrong things, making dumb assumptions, failing to see the big picture, etc.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
I feel like at this point in time only agent swarms can have truly innovative behavior. We'll see if this behavior can be distilled into something smaller and more cost effective.
I'm surprised I'm not seeing anyone bring up the fact that the internal models used by the frontier labs are light years ahead of the public models we have - in the latest METR report, we're three generations behind - there's still an unreleased model from July!
This is easy for us to empirically verify with some back of the envelope math - OpenAI's internal model that solved Navier-Stokes ~2 months ago required 10,000 agents running for 88 hours, or 880,000 total hours, to solve it. A month later, when they dropped the 722 math proofs with some truly stunning results like UGC in there, it took just a single agent an average of 3 hours to get those results. Granted, the results aren't on the same caliber, but some of them genuinely are insane - and even if you knock off multiple digits off the 300,000x speedup there, you're still getting some insane acceleration in capabilities.
And then of course ETH was disproved, and Anthropic claimed it took just 14 million output tokens to do this with their internal research model.
Put simply, the models the frontier labs are using to speed up R&D autonomously that are making them believe RSI is imminent look nothing like the outdated models we have publicly available to us. Saying that the models today don't look like they would do autonomous R&D that well says nothing about how the internal models would perform on similar tasks - in the same way that someone looking at older LLMs and saying they don't code very well autonomously would be mind-blown to see how autonomous coding has turned out today.
No they are not. If they were really getting good at that kind of speed Ai of Meta, Google and Grok etc would have also gotten very good at least matching the current frontier as all the ai dev engineers have moved from oen of these companies to another or build their own companies.
Aside from everything else, you can't just add together the hours from an agent swarm, because swarms become less cost-effective at scale. There's a nice study of that here: https://www.tobyord.com/writing/swarm-scaling
With the most optimistic scaling law from that source, a swarm of 10,000 only gives you a ~500x improvement in performance, and with the most pessimistic only a ~90x improvement. The point of a swarm is to get the same result faster, but more expensively (which OpenAI wanted because of the race dynamics around Navier-Stokes).
This alone knocks 1-2 OOMs off your supposed improvement, and the rest may easily be the different inherent difficulty of the constructions, which is all over the place -- in the big dump, one result (Quasi-RH) took more effort than the rest, and already with Astra people are finding crazy results. Right now, it's easier to judge problem difficulty in terms of the AI time needed to solve them (And whether they can be solved! Most still can't!) than to judge AIs based on which problems they can solve.
> I'm surprised I'm not seeing anyone bring up the fact that the internal models used by the frontier labs are light years ahead of the public models we have
It's really easy to understand, what they have are the same model with less restrictions and gaurdrails, they also have infinite amount of usage and no worry about cost, they can burn a million dollars for an article in the news.
Because we see the products they release even after the breathless "we do 30000 PRs per day" marketing posts.
No, the labs' models are not light-years ahead. That's what labs claim with literally no independent verification of these claims. They want you (and investors, and politicians) to believe these models are light-years ahead.
What they do have is unlimited and unrestricted access to their own models.
OpenAI sat on GPT-4 for 8 months before actually releasing it and that was a major stride over 3. It's kind of silly not to realize they do indeed have access to models months ahead of the public at minimum. Nothing you have access to is the true frontier. It only takes some common sense to acknowledge that. It's not unbelieveable, you just don't want to believe it.
They sat in it. We had concrete details ('rumors' that turned out surprisingly accurate) months before the public release.
>Yes, they do. Nothing in this sentence implies that what they have access to is "light years ahead of the public models".
"Light years" is subjective, but the accomplishments of those internal models we're priven to should be enough to give you some indication. You can fire up as many Astra agents as you like you're not going to get the performance of Open AI's internal model in math. Just as you can fire as many GPT-4 instances and you're not going to get anything resembling Opus 5.5.
> They sat in it. We had concrete details ('rumors' that turned out surprisingly accurate) months before the public release.
So all you had was rumours then "confirmed" by OpenAI marketing
> "Light years" is subjective
Yes. And this subjective description invokes a certain image that I objected to.
> but the accomplishments of those internal models we're proven to should be enough to give you some indication.
Yes. The indication is: there are some improvements, with diminishing returns [1]. A lot of perceived improvements now come from tooling, harnesses etc.
And the people working at the labs have access to unrestricted models and unlimited compute that will not flag you for "bio research" and will not stop running after 5 hours, which adds to the marketing myths.
> You can fire up as many Astra agents as you like you're not going to get the performance of Open AI's internal model in math.
Of course I won't. I don't have the same access.
Doesn't mean any of their models are light years ahead. And as for maths, again, you literally have no idea what they were actually doing besides their marketing materials. But even their marketing materials let slip some details that amount to "commandeered a full cluster for 20 000 hours and let it rip".
Edit: Moreover. We know they are not light years ahead, or substantially better than what users want. While they let some of their researchers do cool shit with unlimited resources, you can see the rest of what these labs produce: from their websites, to their harnesses, to their apps. Buggy incoherent mess. Doesn't scream "light years ahead" to me.
[1] Subjectively the models haven't moved an inch since February this year, and regressed in quite a few areas (code duplication is worse, Claude-ish permeates everything, context compaction is even more random and training quickly overrides context etc.)
> In all of these cases, the model's judgment is just very poor in a basic way.
This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.
The ceiling on LLM intelligence might be in sight. Sure, they are squeezing out the last juice now with RL, for example on design Opus 5.5 is clearly an upgrade. But overall, true intelligence (to think big picture, small detail, ingenious, innovative, novel, or even to just use short simple language to get the fundaments across) seems lacking just as much now as 2 years go.
How does this comment fare with the recent drop of 700 or so novel math proofs? Is math intelligence possibly the outlier because math is the most self-contained knowledge where absolute truth is the easiest to establish?
I wonder, now that we see how far "dumb" approaches to math have gotten, if we just had a massive DB of every math statement that has been proven so far, and we had a program that had 10s of thousands of GPUs running vectorized proof solvers, and we just started doing a search from the final proof statement to underlying axioms, how far we would get...
It seems like LLMs are just guiding an extreme brute-force search (there's a reason they didn't use a single Astra agent, but 10s of thousands).
I'm curious if there are maybe a few million mathematical statements, and only a certain subset will connect at the search nodes you're on. If we had such a DB (which is currently the LLM), how effective such a search would be? This is a bit like program induction but on a math theory. Perhaps the limit so far has been lack of such a database, no one has put every math paper into a formal LEAN proof so it can be leveraged.
A lot of key breakthroughs so far seem to just be "recombination" of existing things (not a massive quantum leap in new math), so perhaps we just need to automate this via search (and not with a very expensive intermediate database of LLMs).
Of course, I've read that just compiling the core math lib of LEAN takes 10+ minutes, so its not exactly a fast process at the moment, so quality of guess is probably a limiting factor.
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
You are correct, but application of that knowledge might require understanding of why the proof is correct. Maybe not in all cases. The proof is the start of the process, not the whole thing.
Why would this be the last juice? We’ll probably see 10x more scale and 10x more / better RL in the coming year. I’m very bullish that that will improve LLMs a lot.
The scaling laws say deminishing returns. 10x scale is not a 10x improvement, maybe more like 1.05x at this point.
I have a gut feeling this is the last juice because for 2 years now they have been "beating benchmarks" yet relatively simple tasks they still can't do sometimes
It would have been a lot more informative if the author told us more about what Astra had suggested and why it didn’t work. I have a strong feeling that neither of the first two tasks require an “ingenious” solution. Just applying techniques from similar games. I don’t know enough about the third task to comment.
Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.
Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.
Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.
Having a good memory is an underrated part of what we generally consider as intelligence. It's the foundation of pattern matching and abstractions. It's speed up for things you already solved before.
Memory + abstraction capability is a more important factor in human intelligence than thinking speed in my opinion.
When facing a novel situation or problem if you can recall a structurally similar one you can better predict how to solve it or how to find new interesting questions to ask.
Part of the problem here is how people use these models and expectations are formed.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
It does seem like they are much better in the small as opposed to the large. I agree on the judgement being questionable, and every report on making large swarms seems to suggest it's relying on spectacularly diminishing returns on the performance of a single agent, especially a single agent with a human in the loop.
It does not look like the majority here realize that your coding AI + harness is very different from a standard LLM's output and capability. The majority of the comments here are referencing their coding harness and attributing that class of application behaviors and nature to a standard LLM. They are very different. Your coding agent and your desktop chatgpt app have more harness enabled capabilities unlocking aspects of the LLM that create interactions that a standard LLM is not capable, and would only engage with considered prompting and context tuning.
I think it shows that much of coding is a lot easier than we think. Even very challenging programming problems involve a lot of just boilerplate and rote implementation of learned patterns.
Arcane is not the same as actually intellectually hard. Programming is arcane.
Same with circuit layout, which is also amenable to things like genetic algorithms, and yes I believe math proofs might be easier than we thought.
It seems to be that you can mix some intelligence and a learned bag of motifs with iterative gradient descent and solve math proofs like a hard jagged optimization problem. I feel like this is what has been demonstrated. You get a huge blob of Lean but it works.
We might learn there are domains like, say, nursing or writing (good writing) or blue collar work or being a secretary for a large company that require a lot more fluid intelligence than fields that have historically been smug about their high IQ requirements.
Coding itself is really quite easy, for an individual that can properly apply logic. However, it's like learning a language (Because you quite literally are.) in that much of learning to code is remembering what words do what.
The main reason it's considered so hard is that, as I've learned over the years, a large majority of the general population struggles when it comes to formal logic. It's the same reason math is considered challenging.
And then I try explaining to people that writing software and using AI and that combination of using AI to write software are all communications problems. There is an entire formal school called effective communications that zeros in on the communications aspects of, well, doing anything which include by yourself as well as collaboration teams as well as AI in any of these and that is a communications situation, and by focusing on the quality of those communications a majority of the stress and confusion evaporates. Then they argue because they argue at anything, and that ends any forward progress.
I think a good harness hides a ton of the abstraction work that makes solving these problems easier.
I work in construction in drafting for electrical and the models go off the rails all the time. Just adding grounds “to be safe” (and it sometimes makes it less safe by doing so)
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.
Honestly at first I was really impressed by AI progress because most of what I do day to day falls squarely into the training distribution - and the models seem to excel at that kind of work, literally in certain areas they are more competent than me.
But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.
For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.
They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.
I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".
After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:
-- It's possible for code in a bundle to execute without the bundle being activated.
We also have a new state (another kind of fact)
-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.
So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:
"which is definitely ACTIVE (since your code is running from it),"
Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.
Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.
For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.
It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.
Not sure why you didn't get more engagement, this is the best take in this thread. LLMs are super impressive, but they differ qualitatively from biological intelligence in ways that make it unlikely for them to replace human labor in places where true human adaptability (and not just language generation) is required.
It blows my mind how bad the standard of basic logical arguments are on lesswrong, which is supposed to be a rationalist website, and has the sort of smug name that would challenge me to hold myself to a higher standard than this.
Here we have an article which amounts to “I have used AI to do some unimpressive research. I have read some unimpressive AI-assisted research. So AIs are unimpressive at research.”
Extrapolate from a tiny sample and establish a universal conclusion. Then look yourself in the mirror and congratulate yourself for how much “less wrong” you are than all the irrational schmucks out there.
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)
I agree that it probably isn't ingenious, but this article does nothing to actually prove that point. "It didn't do stuff, the stuff it did was low value, I will provide no actual details about any of it."
This a twitter thread, not an article arguing a point in a salient manner.
The way I look at AI R&D is it's a really clever brute force search of the problem and solution space. The amazing bit to me, is how rarely, these models and methods seems to get stuck in local maxima. i.e. ex ante, I'd expect these models and methods to get stuck in local maxima 100% of the time.
I'm old enough to have realized that I've done plenty of R&D in my life that wasn't formally named that. R&D is about the substance, the mode, the impact, not the names or titles.
Silent Bob guy just "rebuts" the original poster by spouting how many loc Claude spits out without bugs and unread tests. Which doesn't truly counter anything op said. I'm sure silent on guy can be smarter than me but come on. And yes ai report writing is often verbose and full of crap. Shouldn't need special prompts for it to be focused get to the point
LLMs are great at doing work quickly and they don’t say something is impossible or discourage you, so you can really tackle problems that previously would have taken years of work.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
I think the reason models' default creative output is so bland is sort of like why if you combine all color crayons it just turn out brown.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
> Because it works by generating probable answers, it can only generate probable names.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.
LLMs are skill multipliers, so of course someone without any education and working knowledge of philosophy will get his own results multiplied and be disappointed.
This reads like someone who tried Codex a few times with a blank system prompt and no context and came to the conclusion LLMs are dumb. All of those four examples are handled well by Opus 5.5. Especially with context and competent direction.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.
It reads to me like he’s using Astra. This is based on me reading the comment, and seeing him say he’s using Astra several times, eg:
For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every method in the literature that seems kind of related," etc.
Perhaps you should instead say something like “This seems amenable to Stock Dismissal #4”.
Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together
> The main way they can surprise you today is pulling in information that’s new TO YOU
Many comments here are speaking in absolutes and being myopic about what's "novel". The truth is yes, LLMs output is essentially derivative, but it's also true that a lot of what we consider novel is also derivative. The two aren't mutually exclusive at all.
Had a novel idea? You didn't get it from thin air, everyone is standing on the shoulders of everyone who came before. LLMs most of all, but that doesn't mean it can't bring disparate bits of word-encoded knowledge together and burp out something we'd consider "novel" if a person had made the same connections.
That was actually the basis of some of the mathematics proofs. It made connections from existing material. Nothing a human couldn't do looking at the same material, but we're talking speed and scale here - same as all computational processes.
A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results. The only difference is LLMs are intentionally nondeterministic, as that behaviour can be usefully harnessed, pun intended.
So there's no need to argue about what's novel or not. LLMs can make novel connections via raw speed and breadth of disparate data. The human brain is more complex and "imaginative" for whatever your definition of that is - I mean fundamentally one either believes we are just physical neurons, or that there's a spiritual aspect on top, the end result is the same.
A six sided dice can also be novel if it happens to unexpectedly roll your phone number. It's a novel occurrence, same as an idea coming to mind that, TO YOU, seems novel... but we can't ask the other 8 billion people on the planet so who are we to judge what's a novel idea anyway?
This is a common rebuttal. But it falls flat, in my opinion, because LLMs appear to be a compression of language-based knowledge, and that compression is lossy. The result is that "knowledge" retrieved from the language-based storage model is biased towards the "average" idea.
As a concrete example, when you ask Claude Sonnet 5.5 to generate a small-to-medium-sized one-shot Python project, it will always fall into the same code patterns. Same structures of functions. Same file layout. Same package layout. They're not even good patterns, because if you throw a few common linters at it (e.g. flake8, isort, mypy) there are a ton of linting errors to clean up.
The only way that it escapes this mediocrity is through context. If you give it enough guardrails or sample data (e.g. Python code to emulate), it will start to go off in a "novel" direction that is defined by that context.
Now, what's the most likely way that "novelty" will be generated in an automated system like this? Randomly pulling in context and going for a random walk through the compressed language-based knowledge storage. It remains to be seen whether that's an effective way of producing more than just parlor tricks for investors. Human+LLM is overwhelmingly more novel and effective than only LLM.
This doesn't even address the direct point. The question is how any of this is fundamentally different (difference in type and not just of "scale") from how human cognition works.
Well, I've addressed it implicitly. Every human collects different context by living one of 8.3 billion+ unique lives. We don't have 8.3 billion+ different LLMs. The only way to simulate that is to perform a random walk through the compressed language-based knowledge storage by injecting random context.
Also, those 8.3 billion+ unique lives already produce contexts that are highly optimized for solving human problems that modern humans experience in the physical universe most relevant to humans.
One opinionated answer to this question: LLMs can’t jump [1].
[1]: https://openreview.net/forum?id=klU4737opt
All knowledge storage and retrieval is lossy. Oral histories are lossy. Lectures are lossy. On on one mentorships are lossy, writing or reading a textbook is lossy.
What's your point?
Agree with this as language use is inherently an attempt to compress observations into a shared semantics
The syntax of math is overloaded so you need less syntax to write out something that would be verbose in English ...no one wants to make sense of arithmetic expressed in German syntax. May as well write all software in brainfuck.
Written literacy on a core language like we focus on today is not the historical norm. Creoles and local dialects have vanished in the US as we hyper-normalize everyone into Children of the Corn like carbon copy zombies of character traits
If you want to argue, as I think you do, that there are no new ideas other than new combinations of old ideas, then you have to explain where the old ideas came from. It can't just be combinations all the way down. At some point, some initial set of ideas must have been created where none existed before otherwise we would have nothing to combine. After all the combinatoric closue of ∅ is ∅.
Ultimately any claim that there exist only combinations of old ideas must tackle this chicken-and-egg problem convincingly. For example- explain art. There was no art until it appeared fully formed and out of nowhere in early anatomically modern humans' dwellings (Neanderthals also had art but it was simpler and it is subject to the same chicken-and-eggness). So where did art come from? What pre-existing ideas were combined to bring it into existence from non-existence?
Btw, since I've had this conversation before- if the answer is that we looked around us in the world and saw things; yes, but we didn't see art. Where did art come from? Where did the idea come from of abstracting shapes and colours so that they resemble the shapes and colours of existing things, without actually being those things, or being composed of pieces of those things? How about numbers? How about writing? How about all our technology and science? Where did those things come from? What did we combine to create them?
Obviously I think that the argument that there are only new combinations is wrong: we are a creative species. We can create new things. Somehow. It would be very interesting to understand how. We have not yet created machines that can match us in that and we can't understand the "how" by studying the machines we have now. By denying the mystery of this our ability, by denying that it is a mystery, by dismissing it and accepting any facile explanation that presents itself in our current era -it's god, it's nature, we're like LLMs- we are making ourselves a great epistemic disservice, denying ourselves the chance to find out what it all means, to understand what kind of creature we truly are.
> then you have to explain where the old ideas came from
Its fairly plausible that the original ideas could be sensory gradient-triggered impulses in simpler creatures' proto-minds that (physical and cultural) evolution has gradually built up through more complex neural machinery supporting more complex representational language.
> where did art come from
Similarly, its not a stretch to see art as built from earlier attempts at representation in physical space - starting with mundane attempts at information exchange by colony insects or pack creatures. The breadth and depth of expression has evolved alongside more complex neural machinery and greater variety in media through technology. There's a hidden lineage in creatures' expressive capabilities that may not be evident when we look at a Caravaggio today, sure.
There's nothing terribly chicken-and-egg about these issues (though the resolution is the same - "chicken" and "egg" co-evolved through iterative selection pressures, one didn't precede the other).
All that said, I concur that its a fantastic and mysterious topic to unpack, but it doesn't follow for me that humans are anointed as uniquely "creative" or that there's necessarily something metaphysical at play.
Truly new ideas are, say, 0.01% of all 'novel' ideas, with the rest being rehashing or recombinations. And even then, the truly new kernel of the truly new idea is some small fraction of the whole thing. Like Einstein's theories of relativity, they were truly fresh, but mostly had to recombine existing work in 'novel' ways.
So it's not there are zero new ideas, but so few compared to the whole of the intellectual domain, that we can say 'there are no new ideas under the sun' and be basically correct to within a small rounding error. And AI is sufficiently ingenious to serve the needs of the vast majority of humanity, even though we'll still need to find/cultivate/inspire some new genius humans if we want to have more truly new ideas in the future.
>> A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results.
Actually, we do. We do dismiss calculator results. We are all generally aware of a calculator's abilities. We know it has limits. So when a calculator outputs 9999e999999 we know to distrust that result. It was something out of the calculators concept of understanding. So too a calculator's version of Pi. "3.14159265359" is not Pi and every user knows it is an approximation not be used for anything too important or precise. The same approach should apply to LLMs... but their limitations remain nebulous and ever-changing.
Most calculators use 14-15 digits of pi even if they only show 10. It would be hard to come up with a case where for single calculations this was not “sufficient”.
Not hard: Number theory.
Radians? Do any math where Pi is a base number and the errors can cumulate quickly, depending on where you insert the pi approximation.
Single calculation.
I agree with your opinion. The definition of novelty largely depends on subjectivity. Results produced by combinations of LLMs can also be novel. Combining multiple insights, imitating them, and then organizing them into one can also be considered sufficiently novel. LLMs have a kind of bias due to RLHF, and that may make them uninteresting to humans, but when a "human" uses them, they can enable the human to do more, and that greater amount of work can create new novelty.
It's like reading a news paper, when it's about your field of expertise you find lots of errors, but when it's about something you don't know you take it at face value.
The long name for this is Man Gelatin Ambrosia.
Known as the Gell-Mann Amnesia Effect.
The short name for that is Gell-Mann amnesia.
Strong agree. My 30 years of experience building computers and breaking them has made me better because I don’t trust my knowledge or myself as a first principle. If I didn’t have that and I could easily see myself in an alternate reality, coming in late to the game to leverage ai to build a car out of seatbelts; harnessing all the power of 1000 suns seeing that idea to fruition.
> build a car out of seatbelts
Such a good turn of phrase.
Gell-Mann Amnesia Effect:
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. —Michael Crichton
Edit: sibling comment explains it much more plainly.
https://news.ycombinator.com/item?id=50390237
I have noticed that several times with a newspaper I had at the time. It seemed to be a generally good and reliable newspaper, and one of the biggest in the country, but every time an article was about something that I actually knew about, it was wrong, which made me doubt the validity of everything else.
I now have a newspaper that doesn't give me that feeling.
I've come to the conclusion that they know a lot, but they're not very smart. Not even the best ones.
They know far more than any human ever could: they've read all the books. Their knowledge is enormous. And that often includes how to apply it in standard ways, and sometimes in slightly non-standard ways. But whenever something gets complex, detailed and nuanced, they can't keep up with a human. It becomes a shallow illusion that's easily punctured by just a bit of critical thought.
You mention system architecture, but even for writing code, the code it writes is often horrible. It makes it work because it has a feedback loop that rapidly tests it and ensures it will work, but the actual code it writes down is often not something I would feel comfortable showing to another programmer.
Strong agree, but with a major caveat: First since we humans don't have a memory the size of the internet, new TO YOU is extremely easy to find. And second: a lot of discoveries can be made by trying to match two different domains and generate new insights at the seams.
A lot of math theorems are discovered that way by humans btw: find a difficult problem in a domain, map it to another domain, apply existing theorems there, go back to the original domain. The only difference is an LLM would have to search for an interesting problem, but that can be done by brute-forcing search at the seams and validating the problem does not previously exists. And there is no new TO YOU for the LLM: it can get help from a vast amount of helper domains.
IMHO there's a shot at having category theory really unifying a large chunk of maths in the next decade. It'll be a big ugly ball of Lean, but a queryable one at that.
And then there's all the math problems if solved where the information is new to every human on earth.
This is not novel. Computers have been generating mathematical information that would be resource-prohibitive to produce by hand since the dawn of computing.
The ingenious aspect seems to be that we've found simpler ways to apply computing power to a broader range of problems, which it's important to note is as much a testament to the developers of proof assistants like Lean as it is to LLM developers, to say nothing of the countless mathematicians over many centuries that contributed training data.
Whether this will lead to many revolutions, or a handful of impressive results followed by a long dark spell remains to be seen.
I, for one, wouldn't place large bets on either.
LLMs are like an athletic child - they can do some basic moves, but in general they're uncoordinated.
Occasionally they do something that surprises you. If you put them inside a search loop and spend huge sums of money you'll get some nuggets of value, but for most hard problems the cost isn't worth it.
And they work best with a lot of steering.
From that POV they're exactly like human culture. Most scientific papers are trash, a few are gamechangers. Same with the arts, etc.
But we've gone from 'Can write sentences' to 'Can do frontier math problems' in maybe five years.
We absolutely do have AGI now. It's not very good AGI, but it's probably better than a lot of people were expecting.
Being better than humans at everything isn't AGI, it's ASI. A lot of the comments here seem to be lamenting that we don't have that yet, which is an interesting data point about how far the goal posts have moved.
Edit: also it makes things like this possible. So...
https://x.com/alifarhat79/status/2118478127202713778
if you search papers with it, and from there search related papers and have it do literature reviews, it comes up with very useful white areas uncovered, and from there problem statements and experimental designs to uncover the white spots.
it is nothing but combining known knowledge that begets new.
Yeah, they can only impress the bottom 99% of people. Who needs junk like that.
> by its nature
Declarations about what LLMs can't do "by their nature" have a terrible shelf life, historically.
Unless LLMs drastically shift incredibly quickly, said declaration is likely to hold, because of how LLMs work. I mean, they've worked the same way at their core for several years now.
Their nature hasn't changed but their capabilities have. What we thought was impossible due to their "nature" keeps shifting.
They predict what comes next and are very good at it. That's all intelligence is. What comes next in a pattern? What will my senses perceive next when I drop this ball? What is the next hypothesis to test after I made these measurements in the previous experiment? What is the best next move in this game after the current trajectory?
Maybe. And while this may be a useful superpower, it's hard to predict whether it would be an enjoyable one.
> by its nature it doesn’t have a “vision” of how the whole system fits together
I think a harness for building hierarchical summaries with tooling for expanding and searching history is what's missing for this. We're getting there with Recursive Language Models and continuous compaction systems like the one NVidia just published:
https://arxiv.org/abs/2611.05306
This stokes my suspicion that effective LLM input for software development will ultimately resemble more of a higher level programming language than the natural language in common use today, and that the design of such languages and related tooling will play at least as large a role as the models themselves.
which is actually great, because LLM so far does not show true AGI ability, they are not innovative. Humans are still needed so everyone still have a job. And you can delegate the stuff that you need to be careful with, or attention to detail stuff, to LLMs
Most human jobs don't require innovation.
On the other hand, there's no evidence that work will dry up merely because it becomes more efficient, or even if it does, that this cannot lead to a better world. The trick is finding a way to prevent rapid changes in the nature of work from destroying too many lives, and that's a very human problem machines alone can't solve.
I'd argue it is the other way around - LLMs are great at pure creativity, creating something that never existed before, but not always at "pulling in information that’s new TO YOU". I use them for entertainment - jokes, short flash fiction etc. and they are excellent at it.
> I'd argue it is the other way around - LLMs are great at pure creativity
Interesting because pure creativity is the most prominent thing they can't do by design.
There's no such thing as "pure creativity," which I presume you mean "creative output not directly derived by prior inputs." The very concept is utter nonsense.
turn up the temperature, the outputs will be very "creative" ;)
They can be quite good at taking knowledge from one domain and applying it to another. This can give new perspectives on things, but it isn't explicitly new information.
I too am entertained by LLMs.
Morbid fascination.
What?
They literally generate stuff that's averages of other stuff.
It's not interesting in the least
I guess you chose a fitting username
They're terrible at jokes. I bet any good joke they've given you is simply new TO YOU.
I tested this. Out of 200 attempts I got 2 novel jokes that got good amount of upvotes on r/jokes.
Usually commentators say that they have heard so and so version of the joke, or that it's a repeat, but they did not
Both from opus. Sol/astra could not be funny. The versions that worked I had fed top posts from r/jokes as references and gave abstract directions.
However, the core premises were not new even if the construction was. And it's a lot of work to get something useful out.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
It matches your experience. Its judgement on which joke will get laughs is only slightly better than random with lots of false positives.
I wonder how much different that is from humans, even professional comedians. You don't usually get to see all the jokes that comedians try that don't work, unless you're an avid attendee at comedy clubs.
I'd guess the difference is around a factor of 100 or so.
For comedians, perhaps. But there are plenty of humans who aren’t very good at comedy.
In these LLM comparisons, people tend to compare them to the best humans, not average or below average. It’s a kind of motivated reasoning, with the goal of deprecating the abilities of LLMs.
Well, you did ask for a comparison to professionals.
But I think humor is pretty easy to rate for a human. Did it make you laugh? Yes? Then it might be a good one. This simple human filter radically improves your efficiency as a joke validator, leaving any LLM far behind.
For pictures I'm not sure if humans are much better. For music, yes, after a few listens (recognizing a hit usually takes time). For games, definitely yes.
They're not all bad! Comedian Don McMillan has done the user testing to see if AI will put him out of a job.
https://youtu.be/wBdkmI03V6g
Exactly. If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming. Even for 15 line snippets, if you're very familiar with the programming language, you'd have written it differently.
An LLM is like a low IQ person with vast knowledge.
I have a PhD and Opus 5.5 came up with a novel solution to solve a problem in my area of research that I, and other experts in the field, had been unable to. This was from very general prompting it to find improvements over published methods.
What was it?
It came up with a way to efficiently evaluate the reliability of certain types of complex system using generating functions (https://en.wikipedia.org/wiki/Generating_function) to split the computation, which was far to large to solve with existing methods, into many embarrassingly parallel smaller problems and then reduce it back to a final result. The solution involved several distinct new ideas for it to work, really mind-blowing it was able to come up with it. The other cool thing was that, in the process and unprompted, it implemented code to validate and benchmark against existing approaches and known results just from the pdfs of the published papers (none of which gave reference implementations). In my research it would often take weeks just to replicate a single result from a paper.
Sounds interesting but very vague!
Perhaps but I have no incentive to write all the details here.
I mean breaking a big problem into many small ones is hardly novel. It's how every business works.
By that logic modern cryptography is hardly novel because encryption is just mixing things up which is how every salad works.
>If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming.
How many people on this planet are experts on every topic?
>An LLM is like a low IQ person with vast knowledge.
Are you aware that frontier models have scored in the 135-195 IQ range when they have been tested? How smart are you?
Humans test higher on IQ tests if they had previously taken IQ tests of similar types too tho. And LLMs are trained on IQ tests. IQ tests only really hold their value when the test is fairly novel to everyone taking it.
Okay. So what does it mean when somebody says that LLMs are "low IQ"?
That if you gave them a test far outside their trained materials that it will do terrible compared to a human.
Said tests are often quite inflated, quite frankly. If you can give me a source that disproves this claim, I'll readily eat my words, but I readily believe that those tests are just as inflated as all other claims about AI.
Does this gel with top mathematicans basically begging OpenAI to stop solving their open problems?
They're solving them in a way that isn't particularly interesting because it's near impossible to read and understand, and follow, what the LLM did. It's this way not because it's so smart, but because it's so stupid and still reached the goal by pure force and effort.
I don't know how to communicate this, but what you are saying is almost entirely wrong.
People can follow what they did. The documents OpenAi produced in their big drop are hard to read because they are poorly written in weird ways, not because they are beyond human intelligence. They didn't brute force in the normal sense. AI doesn't get tired, so it can try 1000 promising ideas, but it's like 1000 Ph.D.-level mathematicians trying different ideas, not 1000 calculators trying all combinations.
The biggest results -- quasi-Riemann, no Siegel zeros, and Unique Games Conjecture -- have already been digested by experts. No Siegel zeros in particular already has like five different write-ups, and Tao has written a exposition on his blog.
So did we get any new mathematics out of this, or just solutions to problems without new techniques or insights? I haven’t been able to keep up with this or read the writeups you mention, genuinely wondering since this seems to be a central concern of the math community
It's only been a month and people aren't that fast, but I think yes. Analytic number theory and complexity theory were stuck on the lack of progress on certain questions, that is now unstuck.
I am sorry but your argument looks to contradict itself: They solve hard mathematical problems because they are stupid and use brute force? Then why a human skilled in mathematics guided them for many turns in their conversation?
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
Can't you kind of think of them as a genetic algorithm that has a very good hypothesis generator? Humans could, in principle, help guide that (already very impressive) hypothesis generator to make better guesses to find the proof more easily.
>> If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
"Clearly" as in "somebody tried that and failed" or "clearly" as in nobody ever thought to try it and now nobody will because there's no point?
I think it does gel. You're right that the comment you're replying to is weakened by that fact.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
and I'm guessing it was with "tool calling" ie. they can write a bunch of Lean proofs and hit compile and basically brute-force a whole bunch of hypotheses quickly?
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
You cannot brute force lean proofs of any complexity. The space is infinitely large.
In my experience they write an informal proof first, check it without Lean, and only then turn it into Lean code. That last step sometimes takes them much longer than checking their proof without Lean, which is a problem humans have as well. What they're truly bad at, however, is explaining their proofs. Until they improve at that, we'll have to keep resorting to Lean to check whether they solved something correctly.
I'm less sure about the following, however: I think that while current LLMs are bad at explaining their own proofs, some models are great for discussing older, human-written proofs. For this use case, from best to worst, Gemini > Claude > GPT-Astra >>>>>>>>>>>> GPT-Sol.
I definitely don't know but I got the impression it's the opposite. Based on the observations about Navier Stokes and the natural language proof not matching up with the lean proof and other things. They generate lean and convert it into natural language at the end. But really I could be wrong just giving anyone interested a lead to check out.
A relentless low IQ person with broad textbook knowledge and no need to sleep, I guess?
Look cmon, this is pure cope. The problems it has solved in TCS are far from trivial, with plenty of smart people making their careers in trying to solve them. The techniques these LLMs are using are non-trivial, and not just brute force.
You mean techniques like using the prompt of mathematicians in the training set and brute force a solution before they publish?
No, I mean the techniques in the proofs produced by these LLMs.
I'm more interested in the techniques used to generate these proofs, and how they can be improved to be as reliable at solving a broader class of problem than already formalized mathematical conjectures.
The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
yeah chain of thought reasoning isnt really a think besides running the model for longer and hide most of the output (but still charging you for it). it literally does not work any differently than "non-reasoning" models (except for marketing)
The “conversation” is just one between some redditors or book publishers.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
All novel insights expressible as text can be assembled from snippets of existing text. The challenge is "only" that you need to find the right snippets. Hence being a stochastic parrot does not represent an upper bound on capabilities, but a suggestion that if you want your parrot to give you a specific answer, you should arrange for a context where it's statistically likely to be correct.
Are you saying that a complex stateful system cannot generate a state that is not already recorded in text somewhere?
No. In a large enough state space, you can of course easily find a state that has never existed before. For example, if you pick a 1024-bit integer uniformly at random, it will certainly be different from any previous 1024-bit integer. However, it's much more difficult to find a new integer that cannot be assembled out of bits taken from previous integers. Composition from a handful of base atoms can cover very large state spaces, and unbounded composition can cover infinite state spaces. Which is why only being able to combine things that already exist is not much of a limitation.
To demonstrate:
"Are you saying that a" https://news.ycombinator.com/item?id=49642931
"complex stateful system" https://news.ycombinator.com/item?id=14881439
"cannot generate a" https://news.ycombinator.com/item?id=45956162
"state that is not already" https://news.ycombinator.com/item?id=43013637
"recorded in text" https://news.ycombinator.com/item?id=44731577
"somewhere?" https://news.ycombinator.com/item?id=50391679
A novel state, obtainable by combining six existing ones.
>It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
ignoring the fundamentally dishonest representation of LLMs in this subthread..
if humanity was given a tool that could restructure already existing and known knowledge and facts into novel results with a statistically better answer automatically it wouldn't be as a minor a win as you make it sound; it would be tremendous.
That's not what "attention" means.
> it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge
So all of the maths breakthroughs were not novel?
Firstly, many of them are still being checked, so hold your horses.
What I’ve found LLMs to be OK at when it comes to novel insight is being given a goal to work towards (“prove or disprove this”), with access to a large body of sources and it can then chase down 1000s of options no human would bother with.
This is why security and maths proofs seem to be an area where it can accelerate things. However I know from my own work in these areas it can still hallucinate and so the output needs careful checking.
Is that novel ideation? Or is it self-organising grid search? It’s more efficient than getting 1000 mathematicians to do the work, but it’s not doing something “new”, it’s building on existing work.
And yes, I know what “attention” means, I was making a pun. And it still stands up: they’re not in some deep thought about this and reflecting on new ideas, they’re just hammering away at everything in the literature.
And it still might be bullshit.
> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
I work in a niche where it is utterly hopeless at this, and everyone in the niche agrees: it produces things that look convincing to non-experts but that is fundamentally derivative dross.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
> It’s great at bread and butter stuff, but not at specialised ideation.
This is fundamentally different from your earlier, stronger claim that - "The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago."
I use a less popular java framework quite a lot, Micronaut (I want to move away from it, but that's a different discussion).
When trying to solve problems with its behaviour, I can watch as Claude digests the framework source to come up with suggestions, identify bugs that are causing behaviour regressions, and generally unlock stuff that I've previously found very frustrating. Frustrating because the documentation covers a tiny amount of what's in there, and frustrating because there's little online discourse about any of it to refer to.
Is this earth shattering in itself? Probably not. But it's certainly better than you can find from "an online forum like Reddit or a book/paper published more than a year ago".
It's also far easier to find information through LLMs for another niche I play in, certificate authorities, than it is on the open web. Try to look up anything to do with certificates and you'll just get guided to endless "Why my certificate not work? Browser says no?" discussions, because actually creating certificate authorites is again very niche , not widely discussed and a lot of what you need to leverage in any given language isn't well documented because "you'll never need to do this". Well actually...
You're the one making the claim, and also claiming that you have specific examples to back up your point. So maybe you should share them, and we can see if your assertions hold up.
You’re wrong on literally every point.
Nobody solves the Quasi-Riemann Hypothesis by looking up online forums.
The discussion here is talking about helping individual research with proposals a human will read. That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically. There is no "automatic verification" of a system design proposal, and no way to figure out from 10s of millions of proposals which is best automatically, and also assuming people in this discussion don't have 10s of millions of dollars to throw at every design problem.
I guarantee that you could not find a proof using brute force, even with literally trillions of dollars of compute. You're betraying a lack of understanding of the complexities involved, unless you're using a definition of "brute force" which is essentially equivalent to "intelligent search."
Brute force is certainly the wrong way to put it, but that doesn't mean it's easy to distinguish superhuman effort from superhuman intelligence, or, in the words of Thomas Edison, perspiration from inspiration.
Especially as the pure gruntwork involved in acquiring the knowledge required to merely understand the details of the mathematics involved represent at least a significant fraction of a human lifetime.
And unless this human process of knowledge acquisition continues, eventually no one will be left to understand and attach any significance to the results machines produce.
> That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically
It’s kinda like the project where someone generated a lot of midi files for all possible pop riffs to copyright them. If you have a start and endpoint, rules for navigation, and unlimited computes, you can brute force anything. The nice stuff is when you made exploration efficient.
Other than for maybe Navier-Stokes, they just didn't brute force it in the way you guys are thinking. Most unsolved math problems are not susceptible to brute force, which is why the problems remained unsolved. AI is so quick it can of course try more ideas than a person, but each idea is executed at genius IQ level. Even if you knew the broad outlines of the proof of quasi-Riemann, there are probably only 20 people in the world who could have executed it.
This is trivially true for any theorem with a proof that can be automatically verified in finite time, but does not preclude the universe lacking a suitable combination of monkeys, typewriters, and time to do so.
>That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically.
That is not remotely how the vast majority of the proofs have worked. You don't know what you're talking about, but that seems par the course for these discussions.
They do by reading papers and connecting dots.
Which is what they claim to have done.
>The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
The LLMs are vastly superior for any subject matter with even the slightest hint of danger or otherwise high stakes because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.
Like I'm asking you about sheetmetal thickness vs joist spacing for re-skinning a goddamn carport with roofing steel, read the room, this is not the time to whine about permits and snow loads and whatnot because the project is happening with or without your advice.
And it's not just mundane stuff. Some guy who's retrofitting new chinesium controls onto 50yo industrial equipment that had a prior generation of controls cobbled on 20yr ago doesn't need to get talked down at about stainless on stainless corrosion when he's asking a question about a solenoid or something. If the LLM talks down to him he can brush it aside. If he tries that on reddit they'll dig in their heels and fling poo.
Cutting out the "negative value" middle man of having to painstakingly make the case about the validity of your big picture goal before getting answers to simple questions is of substantial value. Stuff that people would've painstakingly researched themselves rather than ask Reddit or HN or Stack Overflow because dealing with those people is so onerous they now just punch into their LLM of choice and get the best of both worlds.
This matches my instincts too. When trying to debug ML learning pipelines or research projects, LLM suggestions I've received are almost always techically correct, but they point out small, tedious details which don't really matter. In my experience, they miss the forest for the trees.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
Why wouldn’t scale and better RL solve this though? It’s working for mathematics.
As a fellow machine learning practitioner, I don't think the OP's issues are specific to machine learning as much as they are more generally about system design and context. Much of my job as a data scientist is gathering more context about what business value a model is actually trying to provide, and it's difficult if not impossible to encode all of that context into a text format that's digestible by LLMs.
For example, there's a model we have in production whose sensitivity has to be balanced based on which of our customers / partners is consuming its inference. It's a cat-and-mouse game where we are increasingly taking into account additional features to use in the model based on feedback directly from these customers and partners. Adjusting the model based on this feedback is much more art than science because there's really no "correct" adjustment to make, just an adjustment that produces inferences that convince our customers and partners that their concerns have been addressed.
Sure, we could encode every conversation that we've had both internally and across the internal<->external interface, both formal and informal, as text that an LLM could consume. But we find it much easier to just have a human (me, or one of my teammates) do the same thing and then guide an LLM to design a system that satisfies the requirements collected from those conversations.
--
That being said, I'm equally amazed at the quality of suggestions that LLMs sometimes come up with for improvements to systems as I am dismayed by the complete dead-ends it sometimes hits when it doesn't have a good solution in its knowledge base. It's not limited to just ML topics, but also very esoteric things that have very little documentation or even contradictory documentation!
So yes, you're right that if the frontier model providers aim their reinforcement learning laser beams at improving a specific topic they will probably succeed. But, no, I don't think there's a panacea.
It's very possible I'm wrong. I think that what we're seeing in maths is that RL is training LLMs to solved well-posed problems, especially when they can recruit existing ideas from across the literature. I don't want to discount that - it's clearly powerful as we've seen in maths. But in fuzzier research fields, the right question is often more important than the solution - I'm not convinced LLMs are there yet, and I don't think this problem is as amenable to RL as problem-solving.
This recent work which tried to get LLMs to reinvent methods they haven't seen before (given the same problem setting) also raised some of the same issues I've noticed in my own usage: https://epoch.ai/publications/innovationeval
Because "miss the forest for the trees" does not have a large and easily machine-verifiable dataset.
If not ”missing the forest for the trees” is a requirement for debugging an ML pipeline then that offers a path to building such a large machine-verifiable dataset, does it not? That’s what labs are currently doing as I understand it.
As someone who doesn't work in ML, I can only guess what drives the recent improvements.
This kind of post-training and improvements to the training pipeline is exactly what my guess would be.
Curated tasks where the efficient approach is incentivized over getting bogged down in details. This behavior should then generalize.
It seems like a much more likely explanation than the fantasies about illegally training on user data that I often read on HN.
I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
That's a kind of bitter lesson thinking, that we take this generalist model, which already mastered everything from English to Javascript with unsupervised training, and stuff it inside an expert system so it can do long division for it. That's much similar to the last AI winter, when the researchers realized their logics couldn't cope with certain logical scenarios, but still thought they could salvage their flawed reasoners as dispatchers to specific algorithms.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
You can try adapting to this reality in your methodology though. Divide up your project into clear cut and ambitious modules, with well-defined objectives to evaluate each part.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
Exactly, I agree. I have been working on a ux harness that creates a deep product model with personas, user journeys, etc as well as designs. Originally I was hoping it that it could work just some skills and a data export format. It was burning 300M tokens just to get nowhere.
Now I have this DAG with something like 40 specialized subagents. It's sort of a monstrosity, but it works much better for 100x fewer tokens. I've found that basically every single judgment call (which is characteristic of ux) really wants an agent solely focused on it with plenty of concrete examples.
I don't think its the frontier models, my opus 5.5 has no problem working on a project fairly autonomously, for example if I prompt it in the evening it will work the whole night without issues...
On one hand, yeah, I agree.
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
The thing is, without doing the grunt work you usually don't have enough understanding to answer the big questions.
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
The proper term is: "On the shoulders of a very large pyramid of hobbits." The concept is the very core of human intelligence, also known as cultural evolution. I can't resist to recommend Joseph Henrich's book "The Secret of Our Success" every few years here, to anyone interested in human intelligence, and how it is different from other primates.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
That's not AI R&D though, that's you using AI to help your r&d
True. History of Science isnt part of most curiculums. So most people dont see the path and time it took to work out why the magnet points north. But LLMs can actually show you the whole path and a map of different paths people have taken.
Using Fabel to improve some non-AI ML models a while back, I had a similar result - it could implement perfectly any suggestion I made, but it could not work autonomously due to making very poor suggestions itself, or getting distracted by the wrong things, making dumb assumptions, failing to see the big picture, etc.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
I feel like at this point in time only agent swarms can have truly innovative behavior. We'll see if this behavior can be distilled into something smaller and more cost effective.
isn't that them just stumbling on yhe best ideas by chance?
And what do people do? Do they engineer the best ideas one predictably chosen step after another?
I'm surprised I'm not seeing anyone bring up the fact that the internal models used by the frontier labs are light years ahead of the public models we have - in the latest METR report, we're three generations behind - there's still an unreleased model from July!
And the latest model that they measured, which is already behind the newest model they're now working on, has an insane jump in capabilities: https://www.reddit.com/r/singularity/comments/1xorf5c/metr_r...
This is easy for us to empirically verify with some back of the envelope math - OpenAI's internal model that solved Navier-Stokes ~2 months ago required 10,000 agents running for 88 hours, or 880,000 total hours, to solve it. A month later, when they dropped the 722 math proofs with some truly stunning results like UGC in there, it took just a single agent an average of 3 hours to get those results. Granted, the results aren't on the same caliber, but some of them genuinely are insane - and even if you knock off multiple digits off the 300,000x speedup there, you're still getting some insane acceleration in capabilities.
And then of course ETH was disproved, and Anthropic claimed it took just 14 million output tokens to do this with their internal research model.
Put simply, the models the frontier labs are using to speed up R&D autonomously that are making them believe RSI is imminent look nothing like the outdated models we have publicly available to us. Saying that the models today don't look like they would do autonomous R&D that well says nothing about how the internal models would perform on similar tasks - in the same way that someone looking at older LLMs and saying they don't code very well autonomously would be mind-blown to see how autonomous coding has turned out today.
No they are not. If they were really getting good at that kind of speed Ai of Meta, Google and Grok etc would have also gotten very good at least matching the current frontier as all the ai dev engineers have moved from oen of these companies to another or build their own companies.
Aside from everything else, you can't just add together the hours from an agent swarm, because swarms become less cost-effective at scale. There's a nice study of that here: https://www.tobyord.com/writing/swarm-scaling
With the most optimistic scaling law from that source, a swarm of 10,000 only gives you a ~500x improvement in performance, and with the most pessimistic only a ~90x improvement. The point of a swarm is to get the same result faster, but more expensively (which OpenAI wanted because of the race dynamics around Navier-Stokes).
This alone knocks 1-2 OOMs off your supposed improvement, and the rest may easily be the different inherent difficulty of the constructions, which is all over the place -- in the big dump, one result (Quasi-RH) took more effort than the rest, and already with Astra people are finding crazy results. Right now, it's easier to judge problem difficulty in terms of the AI time needed to solve them (And whether they can be solved! Most still can't!) than to judge AIs based on which problems they can solve.
If it’s not public, it’s PR.
I’m totally willing to believe the people making these claims are honest, but until it can be replicated, it’s as good as a lie.
Then why haven't they built an open source replacement for MacOS and iOS yet?
What would be the point of that?
Show-off to investors.
I think solving Millenium problems is more compelling, and arguably less work for OpenAI than a feature parity iOS clone.
Except everyone can see and feel an operating system, unlike the exotic math 'proofs' that may or may not be verified at some point.
> I'm surprised I'm not seeing anyone bring up the fact that the internal models used by the frontier labs are light years ahead of the public models we have
I'm surprised anyone actually believes it.
Why?
It's really easy to understand, what they have are the same model with less restrictions and gaurdrails, they also have infinite amount of usage and no worry about cost, they can burn a million dollars for an article in the news.
Because we see the products they release even after the breathless "we do 30000 PRs per day" marketing posts.
No, the labs' models are not light-years ahead. That's what labs claim with literally no independent verification of these claims. They want you (and investors, and politicians) to believe these models are light-years ahead.
What they do have is unlimited and unrestricted access to their own models.
OpenAI sat on GPT-4 for 8 months before actually releasing it and that was a major stride over 3. It's kind of silly not to realize they do indeed have access to models months ahead of the public at minimum. Nothing you have access to is the true frontier. It only takes some common sense to acknowledge that. It's not unbelieveable, you just don't want to believe it.
No it wasn't. GPT-4 was indistinguishable from 3.5.
> OpenAI sat on GPT-4 for 8 months before actually releasing it and that was a major stride over 3.
OpenAI said they sat on it for 8 months
> It's kind of silly not to realize they do indeed have access to models months ahead of the public at minimum
Yes, they do. Nothing in this sentence implies that what they have access to is "light years ahead of the public models".
>OpenAI said they sat on it for 8 months
They sat in it. We had concrete details ('rumors' that turned out surprisingly accurate) months before the public release.
>Yes, they do. Nothing in this sentence implies that what they have access to is "light years ahead of the public models".
"Light years" is subjective, but the accomplishments of those internal models we're priven to should be enough to give you some indication. You can fire up as many Astra agents as you like you're not going to get the performance of Open AI's internal model in math. Just as you can fire as many GPT-4 instances and you're not going to get anything resembling Opus 5.5.
> They sat in it. We had concrete details ('rumors' that turned out surprisingly accurate) months before the public release.
So all you had was rumours then "confirmed" by OpenAI marketing
> "Light years" is subjective
Yes. And this subjective description invokes a certain image that I objected to.
> but the accomplishments of those internal models we're proven to should be enough to give you some indication.
Yes. The indication is: there are some improvements, with diminishing returns [1]. A lot of perceived improvements now come from tooling, harnesses etc.
And the people working at the labs have access to unrestricted models and unlimited compute that will not flag you for "bio research" and will not stop running after 5 hours, which adds to the marketing myths.
> You can fire up as many Astra agents as you like you're not going to get the performance of Open AI's internal model in math.
Of course I won't. I don't have the same access.
Doesn't mean any of their models are light years ahead. And as for maths, again, you literally have no idea what they were actually doing besides their marketing materials. But even their marketing materials let slip some details that amount to "commandeered a full cluster for 20 000 hours and let it rip".
Edit: Moreover. We know they are not light years ahead, or substantially better than what users want. While they let some of their researchers do cool shit with unlimited resources, you can see the rest of what these labs produce: from their websites, to their harnesses, to their apps. Buggy incoherent mess. Doesn't scream "light years ahead" to me.
[1] Subjectively the models haven't moved an inch since February this year, and regressed in quite a few areas (code duplication is worse, Claude-ish permeates everything, context compaction is even more random and training quickly overrides context etc.)
> In all of these cases, the model's judgment is just very poor in a basic way.
This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.
The ceiling on LLM intelligence might be in sight. Sure, they are squeezing out the last juice now with RL, for example on design Opus 5.5 is clearly an upgrade. But overall, true intelligence (to think big picture, small detail, ingenious, innovative, novel, or even to just use short simple language to get the fundaments across) seems lacking just as much now as 2 years go.
Did you see the chart on model performance over time on LifeVal? Ceiling in there. https://openai.com/index/lifeval/
How does this comment fare with the recent drop of 700 or so novel math proofs? Is math intelligence possibly the outlier because math is the most self-contained knowledge where absolute truth is the easiest to establish?
I wonder, now that we see how far "dumb" approaches to math have gotten, if we just had a massive DB of every math statement that has been proven so far, and we had a program that had 10s of thousands of GPUs running vectorized proof solvers, and we just started doing a search from the final proof statement to underlying axioms, how far we would get...
It seems like LLMs are just guiding an extreme brute-force search (there's a reason they didn't use a single Astra agent, but 10s of thousands).
I'm curious if there are maybe a few million mathematical statements, and only a certain subset will connect at the search nodes you're on. If we had such a DB (which is currently the LLM), how effective such a search would be? This is a bit like program induction but on a math theory. Perhaps the limit so far has been lack of such a database, no one has put every math paper into a formal LEAN proof so it can be leveraged.
A lot of key breakthroughs so far seem to just be "recombination" of existing things (not a massive quantum leap in new math), so perhaps we just need to automate this via search (and not with a very expensive intermediate database of LLMs).
Of course, I've read that just compiling the core math lib of LEAN takes 10+ minutes, so its not exactly a fast process at the moment, so quality of guess is probably a limiting factor.
Mathematics is machine verifiable, meaning you can automate RL at high volume.
Those proofs have not yet been verified by any human. It's quite likely those will turn out to be complete slop and gibberish on closer inspection.
> quite likely
Citation needed.
A few have been retracted, although I agree the above comment is overstating how sloppy they are.
A lot of those proofs are slop and lack the elegance of a human solution.
A proof is a proof though. It's either true or it isn't.
I only have a limited math background, but-
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
Edit: for readability.
I wasn't making a value judgement. I was being scientific.
You are correct, but application of that knowledge might require understanding of why the proof is correct. Maybe not in all cases. The proof is the start of the process, not the whole thing.
You can proof a lot of trivial things that nobody has bothered to proof. Then what?
Why would this be the last juice? We’ll probably see 10x more scale and 10x more / better RL in the coming year. I’m very bullish that that will improve LLMs a lot.
The scaling laws say deminishing returns. 10x scale is not a 10x improvement, maybe more like 1.05x at this point.
I have a gut feeling this is the last juice because for 2 years now they have been "beating benchmarks" yet relatively simple tasks they still can't do sometimes
It would have been a lot more informative if the author told us more about what Astra had suggested and why it didn’t work. I have a strong feeling that neither of the first two tasks require an “ingenious” solution. Just applying techniques from similar games. I don’t know enough about the third task to comment.
Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.
Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.
Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.
I've often said that the term should be "Artificial Eidetic Memory" and not "Artificial Intelligence."
Having a good memory is an underrated part of what we generally consider as intelligence. It's the foundation of pattern matching and abstractions. It's speed up for things you already solved before.
Memory + abstraction capability is a more important factor in human intelligence than thinking speed in my opinion.
When facing a novel situation or problem if you can recall a structurally similar one you can better predict how to solve it or how to find new interesting questions to ask.
We've had hard drives for a while now
Part of the problem here is how people use these models and expectations are formed.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
It does seem like they are much better in the small as opposed to the large. I agree on the judgement being questionable, and every report on making large swarms seems to suggest it's relying on spectacularly diminishing returns on the performance of a single agent, especially a single agent with a human in the loop.
Innovation comes out of the human experimenting with solutions and learning new things that the llm knows.
It’s a dance, not an orchestration.
It does not look like the majority here realize that your coding AI + harness is very different from a standard LLM's output and capability. The majority of the comments here are referencing their coding harness and attributing that class of application behaviors and nature to a standard LLM. They are very different. Your coding agent and your desktop chatgpt app have more harness enabled capabilities unlocking aspects of the LLM that create interactions that a standard LLM is not capable, and would only engage with considered prompting and context tuning.
I always go insane trying to tell my software friends this and they wax poetic about how capable the models are.
Trying to insert your own harness into another field makes it very clear and the friends approach AI psychosis levels of overconfidence
I think it shows that much of coding is a lot easier than we think. Even very challenging programming problems involve a lot of just boilerplate and rote implementation of learned patterns.
Arcane is not the same as actually intellectually hard. Programming is arcane.
Same with circuit layout, which is also amenable to things like genetic algorithms, and yes I believe math proofs might be easier than we thought.
It seems to be that you can mix some intelligence and a learned bag of motifs with iterative gradient descent and solve math proofs like a hard jagged optimization problem. I feel like this is what has been demonstrated. You get a huge blob of Lean but it works.
We might learn there are domains like, say, nursing or writing (good writing) or blue collar work or being a secretary for a large company that require a lot more fluid intelligence than fields that have historically been smug about their high IQ requirements.
Coding itself is really quite easy, for an individual that can properly apply logic. However, it's like learning a language (Because you quite literally are.) in that much of learning to code is remembering what words do what.
The main reason it's considered so hard is that, as I've learned over the years, a large majority of the general population struggles when it comes to formal logic. It's the same reason math is considered challenging.
And then I try explaining to people that writing software and using AI and that combination of using AI to write software are all communications problems. There is an entire formal school called effective communications that zeros in on the communications aspects of, well, doing anything which include by yourself as well as collaboration teams as well as AI in any of these and that is a communications situation, and by focusing on the quality of those communications a majority of the stress and confusion evaporates. Then they argue because they argue at anything, and that ends any forward progress.
I think a good harness hides a ton of the abstraction work that makes solving these problems easier.
I work in construction in drafting for electrical and the models go off the rails all the time. Just adding grounds “to be safe” (and it sometimes makes it less safe by doing so)
Models go off the rails in programming all the time too. Despite the hype, they still very much suck at programming.
That's because you're not communicating to them what you want and need, not enough, clearly, or they would not "go off".
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.
Honestly at first I was really impressed by AI progress because most of what I do day to day falls squarely into the training distribution - and the models seem to excel at that kind of work, literally in certain areas they are more competent than me.
But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.
For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.
They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.
I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".
After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:
-- It's possible for code in a bundle to execute without the bundle being activated.
We also have a new state (another kind of fact)
-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.
So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:
"which is definitely ACTIVE (since your code is running from it),"
Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.
Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.
For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.
It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.
Not sure why you didn't get more engagement, this is the best take in this thread. LLMs are super impressive, but they differ qualitatively from biological intelligence in ways that make it unlikely for them to replace human labor in places where true human adaptability (and not just language generation) is required.
It blows my mind how bad the standard of basic logical arguments are on lesswrong, which is supposed to be a rationalist website, and has the sort of smug name that would challenge me to hold myself to a higher standard than this.
Here we have an article which amounts to “I have used AI to do some unimpressive research. I have read some unimpressive AI-assisted research. So AIs are unimpressive at research.”
Extrapolate from a tiny sample and establish a universal conclusion. Then look yourself in the mirror and congratulate yourself for how much “less wrong” you are than all the irrational schmucks out there.
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)
AGI right around the corner!
I agree that it probably isn't ingenious, but this article does nothing to actually prove that point. "It didn't do stuff, the stuff it did was low value, I will provide no actual details about any of it."
This a twitter thread, not an article arguing a point in a salient manner.
It only appears to be not ingenious because it does not know what you want to achieve.
this really hit me with Jev. it never occurred to me that something like scoring rules weren't applied before.
The way I look at AI R&D is it's a really clever brute force search of the problem and solution space. The amazing bit to me, is how rarely, these models and methods seems to get stuck in local maxima. i.e. ex ante, I'd expect these models and methods to get stuck in local maxima 100% of the time.
I'm old enough to have realized that I've done plenty of R&D in my life that wasn't formally named that. R&D is about the substance, the mode, the impact, not the names or titles.
Silent Bob guy just "rebuts" the original poster by spouting how many loc Claude spits out without bugs and unread tests. Which doesn't truly counter anything op said. I'm sure silent on guy can be smarter than me but come on. And yes ai report writing is often verbose and full of crap. Shouldn't need special prompts for it to be focused get to the point
They are useful if properly guided. Just as every other tool.
The bitter lesson is that ingenuity is not really necessary in this realm; more benefits will accrue to you if you just scale harder.
Ingenuity was last decade's moat. But human ingenuity didn't crack those millennium math problems; throwing more money at compute did.
> I struggle to believe at an intuitive level that autonomous R&D is so ingenious.
Why would anyone go to the effort of struggling?
LLMs are great at doing work quickly and they don’t say something is impossible or discourage you, so you can really tackle problems that previously would have taken years of work.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
I think the reason models' default creative output is so bland is sort of like why if you combine all color crayons it just turn out brown.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
> Because it works by generating probable answers, it can only generate probable names.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.
Could you give some examples? Because I doubt that they sound properly novel.
LLMs are skill multipliers, so of course someone without any education and working knowledge of philosophy will get his own results multiplied and be disappointed.
anything you prompt is such that instruction fine-tuning means you're actually the idiot
This reads like someone who tried Codex a few times with a blank system prompt and no context and came to the conclusion LLMs are dumb. All of those four examples are handled well by Opus 5.5. Especially with context and competent direction.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.
It reads to me like he’s using Astra. This is based on me reading the comment, and seeing him say he’s using Astra several times, eg:
Perhaps you should instead say something like “This seems amenable to Stock Dismissal #4”.