PrismML hopes its tiny LLM may change how all of us use AI

If AI lab PrismML isn’t in your radar but, it ought to be — not as a result of it’s raised gobs of cash (it hasn’t but, only a $22.25 million seed spherical), however due to the technical minds concerned and the doubtless industry-changing tech it’s creating.

PrismML is betting that succesful, high-performing, reasoning massive language fashions don’t, actually, must be massive.

It’s making reasoning fashions so small they will match on PCs and smartphones. (It’s even rumored to be in talks with Apple, although CEO Babak Hassibi declined to touch upon that to TechCrunch.)

On Thursday, PrismML released Bonsai 2 27B, its newest in a household of fashions, which compresses Qwen3.8 27B, a broadly used open-source mannequin from Alibaba, down to five.9 GB. That’s sufficiently small to suit on a PC and, probably, a high-end smartphone. It’s a 9x to 10x discount in reminiscence versus the unique.

PrismML was based by a bunch of Caltech researchers and is led by Hassibi, a Caltech professor and an skilled in compression applied sciences. The startup additionally counts Ion Stoica as an advisor. Stoica is a co-founder of Databricks (and different corporations) and the director of Berkeley’s famed Sky Computing Lab, which has birthed many applied sciences and startups, from Letta to SGLang.

PrismML can be backed by traders Khosla Ventures, Cerberus Capital, and Caltech.

This startup is definitely not the one firm engaged on LLM compression tech. Multiverse Computing, based by a well known professor from Spain’s Donostia Worldwide Physics Heart, is one other. (And Multiverse Computing has raised gobs of cash.)

However Hassibi says that PrismML’s compression tech is exclusive as a result of its LLMs have misplaced nearly no efficiency in contrast with the originals. Bonsai 2 matches 98% of Qwen’s mixture benchmark scores. That’s up from the primary Bonsai, launched a few months in the past in March, that matched 95%. That unique mannequin has already been downloaded over 11 million occasions, and PrismML’s even smaller fashions have been downloaded one other 2.6 million occasions, the corporate says.

So this exhibits that PrismML’s compression outcomes have improved from one launch to the following. Whether or not it may ever get to 100% benchmark efficiency parity is a query that continues to be to be seen. Compression will seemingly at all times have some impression, Hassibi says.

Nonetheless, excellent benchmark parity is pretty educational anyway. LLMs usually are not so correct of their uncompressed kind, and benchmarks not so completely reflective of precise duties, {that a} 2% degradation would seemingly meaningfully have an effect on how a mannequin performs in precise use. (Plus, the encircling software program — the harness a mannequin runs inside — matters a lot when it comes to accuracy, too.)

PrismML says it achieves this by shrinking the “weights” that make up a mannequin — weights are, primarily, the data a mannequin learns and shops throughout coaching. Usually, every weight requires 16 bits. PrismML’s method, known as “ternary” weights, simplifies that down to a few: +1, −1, or 0. With far smaller values to retailer for every weight, the mannequin takes up dramatically much less house. (For a deeper dive on the compression method, right here’s the challenge’s GitHub page.)

The startup’s subsequent purpose is to use this compression method to even larger fashions. “The subsequent fashions that we’ll launch, hopefully within the subsequent couple of months, shall be within the several-hundred-billion-parameter vary, and I count on will probably be simpler to retain the intelligence there,” Hassibi advised TechCrunch.

As mannequin dimension grows, he added, “There’s extra room to have the ability to compress them with out dropping the intelligence. So I might simply say, as a basic development, for bigger fashions, it’s simpler to get to 100%.”

Stoica tells us that he’s excited for this tech as a result of it’s making it doable for superior fashions to run on customers’ gadgets. “You’ll have intelligence at your fingertips, and it’s going to be free as a result of it’s going to run on the machine you already purchased. It’s additionally going to be personal, since you’re not going to ship it to the cloud.”