What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.
I need help understanding this.
I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models.
How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?
I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem.
Anyone care to explain this poor silly fellow some of those points?
Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.
Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this.
Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Quite weird that heavy quantization method on a dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.
At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.
After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.
I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.
How do you find the overall experience? And do you have any special sauce or recommendations for going this route?
The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.
What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.
Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.
It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.
Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.
It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
Of course not, personally almost all of my code these days is generated.
The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.
27B is way more than you need for a phone. Doesn't matter how much you try to compress it, it's the wrong application of the wrong tool. There are already useful tiny models that fit on phones and do basic things really well. Dumb down a big model too much and it becomes worse than a small fine-tuned model.
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.
I was trying Ornith 9B locally (it's up on Ollama) which claims:
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol
Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.
I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.
What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.
I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models.
How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?
I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem. Anyone care to explain this poor silly fellow some of those points?
Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...
Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.
Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.
[1] https://jackson.dev/post/dont-sleep-on-bitnet/
Preliminary analysis via lm-evaluation-harness + vllm
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this.Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Depending on which model you're running, you might need to use the custom forks.
Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....
I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac
If you can share details on where it is failing, we'd love to help fix.
You can also join Discord to communicate with us directly http://discord.gg/prismml
I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
Quite weird that heavy quantization method on a dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.
At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.
After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.
What model? And what hardware do you run it on?
I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.
How do you find the overall experience? And do you have any special sauce or recommendations for going this route?
The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.
What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.
Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.
Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.
It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
packing multiple trits together
e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer.
The way they do it is packing like the other comment says.
Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
I'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom:
https://github.com/Liquid4All/antidoom
Tried it on Android and got "!!!!!!!!!!!!!" for answers.
The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.
When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.
For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
start saving your money.
I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
Most probably not optimized yet for this model...
Entire blog post seems to be AI-generated :/
Do you think people who work on AI for a living are not going to use it?
Of course not, personally almost all of my code these days is generated.
The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.
The text is mostly content-free. Headline + charts are enough for most HN stories.
27B is way more than you need for a phone. Doesn't matter how much you try to compress it, it's the wrong application of the wrong tool. There are already useful tiny models that fit on phones and do basic things really well. Dumb down a big model too much and it becomes worse than a small fine-tuned model.
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.
I was trying Ornith 9B locally (it's up on Ollama) which claims:
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
https://deep-reinforce.com/ornith_1_0.html
Only tried it so much so far; it did a little better than Qwen 9B
Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).
3.5 9B can do thinking. Its just disabled by default in its gguf chat template.
It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.
Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals
Is that a 1-bit LLM? I don’t understand the connection with this article.
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol
This must be some sort of unpublished app?
I can just see their image tool on the app store
It's a LLM model, not a phone app.
Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b
This is useful research, but this particular model itself is likely absolutely useless.
Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.
I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.