I don't think being low level is the main innovation, really. There are several things Rust did right over traditional ML. Explicitly caring about learnability and the "weirdness budget". Having great error messages that don't require a course in category theory (many ML) or 800kB of scrollback buffer (C++) to understand.
Having great tools. Excellent documentation. Being friendly to new users.
Yes, it's also a systems language without a runtime. But that's not the novel part. You could write horrors in C++ that approximate ML even without language support. There are eldritch libraries where some kind of pattern matching is done via generic lambdas.
The main difference is developper UX. Good tools, good error messages, quality of life. The novelty is making ML not painful.
> Yes, it's also a systems language without a runtime. But that's not the novel part.
Low level strong correctness was absolutely a novel part. In fact it’s exactly why many people glommed onto early rust, and why it was lowered on the stack.
Although learnability and weirdness budgets were also extremely novel in low level contexts which had been subsumed by C and C++.
> horrors in C++
Yes, horrors in C++. Half baked jerry-rigged and barely usable nonsense. Not an industrial strength langage with a reliable type system and a strong focus on safety through types.
50 years of computing have proved pretty conclusively that less than that is wishful thinking at best. Large C++ programs, even with massive amounts of resources and tooling, can’t even get memory management correct.
Moving a complex system of muscles so that they can just stand upright is already very very complex, let alone intercepting a prey's movement mid-flight by just controlling all those muscles.
People way overestimate the actually intelligent part of LLMs vs simply being good at recalling context-related stuff from the training data.
Complexity does not require intelligence. Modern computers (even without AI) and technological systems do incredibly complex things and I'm quite sure you would not call those systems (again, without AI) intelligent.
There is a difference between a problem being complex and you try to find a solution to it (hard), vs a program being complex. The latter is trivial to execute, but that is entirely different from analysing it.
Sure, but now we've strayed far from the starting point which was that cats are bad because they torture small animals. In fact most animals are bad. The question becomes which animals do we want around the place.
That's oversimplifying the topic to some catchy lyrics' lines level.
Birds burn a ton of energy flying (at least the birds in question here, other birds can glide for long times), it's not something they would willingly do to no ends.
Well, is slightly modified regurgitated code a copy or not? We have yet to have it answered in the age of AI, but e.g. I could not be selling Mickey Mouse merch with a simple color filter on for long.
Agree it will be interesting to see how things play out. There's enough permissive open-source licensed code available that using that only could be an option.
As for Mickey, is the difference from Oswald enough today?
Yeah, the funny thing that Linux being open-source is absolutely in line with capitalism. Just look at the list of maintainers - they are almost all paid employees of gigacorps.
It is just an optimization that makes sense -- writing an OS that is compatible with all sorts of hardware is hard, let alone one that is performant, checked for vulnerabilities, etc.
Why would each gigacorp waste a bunch of money on developing their own, when they could just spend a tiny bit to improve a specific area they deeply care about, and benefit from all the other changes financed by other companies.
And the GPL makes it all work - as no single gigacorp can just take the whole and legally run with it for their gain, like they could if it was say MIT or BSD licensed.
So you have direct competitors all contributing to a common project in harmony.
Well, GPL is good but I think this setup would still be a local optimum for gigacorps, were it MIT or so. They are using plenty of MIT libraries, e.g. Harfbuzz.
It would just simply not make sense for them to let other companies' improvements go out of the window, unless they can directly monetize it. So it doesn't apply to every project, but especially these low-lying ones would be safe even without any sensible license.
No company at the moment has enough money operate with 10x the reasoning tokens of their competitors because they're bottlenecked by GPU capacity (or other physical constraints). Maybe in lab experiments but not for generally available products.
And I sense you would have to throw orders of magnitude more tokens to get meaningfully better results (If anyone has access to experiments with GPT 5 class models geared up to use marginally more tokens with good results please call me out though).
Well, how many more dogs would you need to help you write your university thesis? It's a logical fallacy to assume that more tokens would somehow help - especially that even with cursory use you would see that LLMs, once they go off the road, they are pretty much lost, and the best thing you can do with them is to give them a clear context.
Humans are "multi-modal". Sure we get plenty of non-textual information, but LLMs were trained on basically every human-written world ever. They definitely see many orders of magnitude more language than any human has ever seen. And yet humans get fluent based after 3+ years.
If you treat the human brain as a model, and account for the full complexity of neurons (one neuron != one parameter!) it has several orders of magnitude more parameters than any LLM we've made to date, so it shouldn't come as a surprise.
What is surprising is that our brain, as complex as it is, can train so fast on such a meager energy budget.
You are right, but at the same time the human brain does way more stuff (muscle coordination, smell, touch sensing) and all those others take up at least some budget.
So interesting question, but I'm not convinced it's only a scale issue. Like finished models don't really learn the same way as humans do - we actually change the parameters "at runtime", basically updating the model and learning is not only for the current context.
It goes both ways though. All that extra stuff is also a part of our "training set" when growing up. And we have already seen that training models on vision etc improves their text outputs as well, even in tasks that aren't directly connected to visual things. That might account for a lot of our advantages.
But yes, of course it's not just a scale issue. Note though that a "finished model" can still be fine-tuned, and you can in fact allow it to fine-tune itself even. It's just that this is prohibitively expensive in practice (once again, the hardware is lagging behind the wetware here).
For sure, it seems like there's something there primed to pick up human language quickly, clearly evolutionarily driven.
Not necessarily so for the dynamics of magnetic fields, or nonhuman animal communications, or dark energy/matter.
We are bombarded nonstop by magnetic fields, nonhuman animal communications, and live in a universe which seems to be majority dominated by dark energy and matter, and yet understand little to none of it all.
reply