Why don't machine learning research agents overfit?

(amazon.science)

129 points | by Betelbuddy 16 hours ago

12 comments

  • diddid 15 hours ago

    I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.

    It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.

    • srean 14 hours ago

      I think you should get less annoyed.

      > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.

      I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct

      https://web.archive.org/web/20170428225156/http://www.cse.bu...

      https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

      • arijun 7 hours ago

        “You should prefer it, because it’s simple” is just restating Occam’s razor, not giving any explanation. “The simplest is more likely to be accurate” is a much better interpretation than yours.

        I think the most accessible example of Occam’s razor is fitting a line to some points; you can always use a high enough order polynomial to fit the seen points exactly, but a straight line is likely closer to representing the underlying distribution.

        • sillyfluke 15 hours ago

          >It’s just like the Hopper quote.

          Not sure about Hopper, as I recall biographers of Lawrence of Arabia certainly made it seem like he was using the fog of war to do things he knew his superiors may object to.

          Regardless, even if its misinterpreted it still has a kernal of truth and separate utility than your version, that is: the people in the field closest to the action have an operational awareness that may result in better decisions in times of urgency.

          • gowld 15 hours ago

            That's not true. It's pretty clear that she meant "do something you knew they were going to say no to and now you are trying to get away with something."

            https://youtu.be/wHdHCoeUbU4?t=861s

            > So I want to tell something to all the young people here on many many occasions you'll find it is much easier to apologize than it is to get permission. You do it then when somebody comes after you and say are you supposed to do that, "oh gee I didn't know I wasn't supposed to do that" ... so just remember it's frequently much easier to apologize than it is to get permission do it

            She goes on further, explaining how to deceive your superiors to manipulate them to get what you want.

            • bnmik2 13 hours ago

              Marcus Hutter formalized this in his AIXI work.

              • beckhamc 14 hours ago

                And sadly, in academia, complexity (opposite of Occam's razor) is what gets you published.

              • demibabs 15 hours ago

                Even tech giants are putting out articles seemingly fully written by Claude.

                • ks2048 15 hours ago

                  The animated graphic labeled "Occam's razor, formalized" is bizarre. Is that really visualizing "Occam's razor, formalized"?

                  • serial_dev 15 hours ago

                    Time to first detected slop in this article is <1s. Claudisms per paragraph is also very high.

                    Is it too much to ask from people to read their own article anymore?

                    If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...

                    • percentcer 14 hours ago

                      Nobody wants to work anymore!

                    • signalbright 14 hours ago

                      > Why don't machine learning research agents overfit?

                      they do.

                      • dguest 15 hours ago
                      • jsrozner 12 hours ago

                        Why is this being published as a blog post and not as a peer-reviewed submission? If it's going to be a blog post, why isn't there a corresponding scientific version for me to look at?

                        Someone else already found it. I don't understand why the link isn't in the blog post. https://arxiv.org/abs/2606.11045

                        Use of claude for writing it should be disclosed.

                        • llflw 4 hours ago

                          Do you know what the scaling law actually is? Overfit everything as much as you can.

                          • ubutler 10 hours ago

                            If anything, the latest generation of AI models, Astra and Fable, are prime example of overfitting—whereas benchmarks suggest they’re AGI-tier, users (including myself) report the same old gaslighting, hallucination, context rot, cheating, incomprehensibility patterns as with prior models, sometimes even more pronounced.

                            Fable and Opus 5, I suspect, will become textbook examples of RL collapse.

                            • nyeah 15 hours ago

                              They tend not to overfit ... when there are way more data points than parameters.

                            • 32df179 15 hours ago

                              Wherein Claude gives an honest assessment that it genuinely does not overfit. I also had Grok telling me that it isn't quantized.

                              Do the submitters really not notice that this is AI slop? Do they like this? It is a complete pain to read.

                              • sigbottle 14 hours ago

                                Compression in this modern day and age is so slop.

                                Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can always hide the "heavy work" in your mappings and descriptions. It's ironic that a kolomogorov complexity minimizer is so loaded that it's vacuous.

                                This is pretty much why I roll my eyes at this point at all the compression is intelligence memes.

                                I wonder when intervention and causality will hit the mainstream. These tools were designed specifically to counteract purely predictive theories. But your average compression dude will hold tight to their paradigms and slogans, not realize their internal contradictions (that their own field has brought up), and then whenever a new paradigm suddenly becomes visible and mainstream, they'll latch onto that. It's not principled at all.

                                And to be clear - I do think intelligence is some amount of compression, and I am well aware of formal results such as the arithmetic decoding theoretical and empricial result. Just annoyed. It's literally no different than the whole Bayesianism meme. If you're not actually practicing that type of intelligence as a basis, then you don't get to go around beating the drum about how it's the ultimate reality. You're just spouting dogma to feel like part of an in-group.

                                • benlivengood 9 hours ago

                                  I think that the ratio of work done by priors and search is the interesting question at this point. We aren't quite at the point where reading off the most likely hypothesis decompressed from a transformer is sufficient, but it's a lot closer than I originally suspected when we were ~solving chess. I think AlphaGo was kind of the watershed moment that prior-guided search was so much better than either alone.

                                  I think the adversarial policies against Go AIs directly show the gap between intelligence and compression/priors.

                                  • srean 12 hours ago

                                    > I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them

                                    Then why don't you ? and did you mean all or those that halt ? I presume you have a way of separating those.

                                    • Xcelerate 12 hours ago

                                      I’m guessing your downvotes were for tone? You’re correct though regarding Solomonoff induction, as the choice of reference universal partial recursive function gives drastically different results for predictions based on finite data (even with access to a halting oracle). Asymptotically, any choice eventually converges to the same predictions, but that’s no help when there are infinitely many choices for U and no obvious natural prior over universal functions. And I don’t find the argument that our natural environment “implements some choice of U” particularly convincing. There’s definitely an open mystery there.

                                    • vatsachak 14 hours ago

                                      No point in reading anything AI related anymore. It's all slop.

                                      We need to retvrn to rss feeds

                                      • jimmaswell 4 hours ago

                                        Is there a study demonstrating that AI-assisted articles are less substantive or is it purely a kneejerk bias towards humanslop?

                                        • exit 14 hours ago

                                          what would returning to rss feeds achieve?

                                        • dominotw 16 hours ago

                                          > Machine learning, at its core, is about generalization, not memorization.

                                          Well they memorize the patterns.

                                          memorization doesnt mean rote learning.

                                          • tomrod 15 hours ago

                                            I don't take issue with that. Attempting memorized pattern generalization through holdout / validation strategies is a big part of ML that you would not typically see with econometrics / psychometrics / possibly sabermetrics / most other -metrics. Philosophically the explain versus predict divide. https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf

                                            • red75prime 13 hours ago

                                              Does '110001111000000011111111111' contain n>1∧∀d(d|n→(d=1∨d=n)) as well as infinite number of other generalizations?

                                              • porridgeraisin 15 hours ago

                                                That's a bit pedantic no. Memorization in ML refers to the model having the wrong level of capacity such that it's too hard to optimise it such that it doesn't memorize the _training examples_ themselves.