For the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
I agree with you, and I'm using claude and other models heavily, the idea is not in the soneshot - because actually as you see in the logs I posted on github, it took around 107 + 50-ish experiments to figure out which models to run which and push the speed.
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.
So true, people think they can prompt AI, tell it to create something amazing and "Novel" so they can feel cool and bragg about it. Shame on such people.
lol sure
"claudia slop me up some 600m, commit, push, make 500 pages of slopped README.md, github stars+++"
great, you should definitely do this, there will be absolutely no problems whatsoever and it'll also run at peak speeds while of course being completely unusable because who wants to run a slopped 600m
t. at 1200+ commits on my "oneshot"
Recently I started this open source project at my AI company, where I'm trying to make LLMs working
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
> a more challenging device
A Playdate
A Speak and Spell that replies using the voice processor
perhaps the pi 1st/2nd gen?
I'd thought of the ESP32 devices but their RAM is too small
A PlayStation 2 dev kit (with 128 MiB of RAM instead of the standard 32 MiB).
Between the VUs, the R5900's SIMD (MMI), the GS and the I/O buses (FireWire, Ethernet, IDE, Memory Cards...) to stream model weights, the smallest Bonsai models (or aggressively quantized Qwen3.5-0.8B) and some really cursed programming might just eke out something the PlayStation 2 has no business running.
> I want a more challenging device, what do you think?
One of those vapes with a screen on it.
This is very cool. Love the trend of optimizing small models
I’ve been working on small local models and it’s amazing how fast the tech has progressed
A year ago, I was creating conversational characters by finetuning a qwen 2.5 0.5b param model, and it worked pretty well. But no reasoning or math capabilities, and still required some decent specs to run; could maybe do well enough in a beefy raspberry pi
But now, I have a locally trained 6.5M param model that does reasoning, solves multi-step math problems, does simple physics problem simulation, tool calling, RAG and conversational chat. Total size about 30MB packaged for ONNX, and runs with <10ms latency in the browser
A year ago 1B parameter models couldn't do much, today with the correct harness and/or fine-tuning they can do more or less serious work (not frontier tho).
I believe in a 6-8 months it will get even better and my TV will be happy.
I didn't try fine tuning so small models honestly but looks interesitng to try.
It's technically sweet regardless, so don't take this as even a little bit of negative pushback, but I am curious what the utility of these small models is? Are they smart enough to be useful on practical tasks? What are people doing with them?
Not op but my opinion is that It doesn't matter if it's not currently useful. I think it's important that the community focuses on this line of research because big companies will not. It's in their best interest to make LLMs resource hungry
If you work in a highly regulated space subject to e.g. HIPAA, and you work with ancient on-prem systems within a slow moving and risk intolerant organization -- for instance, virtually every state run healthcare system in the US -- then these are all you can hope for.
In these settings, data is insanely messy beyond your wildest nightmares and hidden away in free-text fields that small models like this are great at cleaning up, normalizing, parsing, categorizing, etc
This is an interesting question, so the idea is, based on experience, fine tuned small models can do narrow tasks very well (my experience running models fine-tuned on customer data as low as 1-2B is very promising) also the goal is to find a breakthrough were instead of the line of work (which I respect) that tries to squeeze much bigger models into smaller hardware like strata and maya, I'm working backward to discover/search for custom kernels and modifications till we exhaust the limit of what a smaller machine is capable of, so instead of people doing 64gb machines and old gpus to run a deepseek, I want to see what is the biggest usable qwen small model can run with reasonable speed on smaller devices, and by scaling as we discover more tricks, normal laptops can unlock more compute - for example a qwen 3.8 27B can do pretty serious work when coupled with a specialized harness - but we are yet to figure out what is the smallest machine that can run it with usable output and usable speed, and this can unlock interesting business.
Speech recognition, speech generation.
This alone is quite useful in small/limited devices/contexts.
Realtime log parsing and summarization is another application I've seen for little LLM's..
Basically, anywhere you'd use an LLM, but at much, much smaller scales of ontology.
Also curious on this. They are too dumb to use with tools and if you ask it how to bake a pancake you only get rubbish.
The AI botnet is going to fill every crevice of civilization, do you want AI ants? This is how we get AI ants.
Or maybe AI bees :(
in a similar vein, somebody got tinystories LLM running on the nintendo wii, including gpu acceleration [0].
this looks very interesting, it can't run a real (read : working on some real tasks) LLM, yet, but it's actually very interesting device.
Thanks for sharing! I'm exploring a similar direction, but optimising for extreme model size. I have gotten an LLM down to 970 KB, with some interesting results compared to GPT-2 (small).
Super interesting, not a CS/AI guy here, but I'm curious as to how you can train such a small-sized model give you as accurate an answer or good hypotheses to get you going
This is extremely interesting, do you have a public repo or write up to share or not yet? if not yet please sign me up when you release it, Cheers!
This is really cool man. I like this on going trend of resurrecting old tech with AI and this is yet another avenue to explore.
I'd be interested in seeing some older flag ship mobile devices
I have a few phones and tablets laying around, will try to play with..
I'm doing https://github.com/rcarmo/go-pherence for the same reason. We should swap notes.
Is this the new version of "Can it run Doom"?. Maybe, "can it run Qwen?"
What can these small models do? Are they just for novelty?
as of now, it's just for novelty and fun, without serious fine tuning on domain data, you can't really do much with the smaller models.
So does it make sense to try and run models on older 4gb GPUs? I assume the energy efficiency isn't worth it?
well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.
It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.
There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.
Unless you have electric resistance heat, and it's cold.
> Device #1 is a Xiaomi TV Box S (3rd Gen): Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable)
Whoah!! A tv has almost 2GB of ram?! Does it need it to run Chrome or Electron? /s Ouch
no electrons were hurt in the process of making this :(
This is great. I've recently been playing with repurposing some old thin clients as homelab Linux machines
Not my site, but same idea: https://www.parkytowers.me.uk/thin/
All this old/obsolete gear is plenty capable when used with the right software.
Funny, I use that exact website to find passively cooled hardware to use as a server.
I did get Qwen3.5 0.8B working at 5 tokens per second on the iGPU of a Dell Wyse 7020. I mostly use it to complain to; I have not yet found an agentic task for it. But also the CPU does plenty of small services and terraria servers and such.
https://nortyspock.github.io/blog/running-a-language-model-o...
Out of interest, what have you been doing with them?
but why that matters in the age of scaling llm ??
machine spirits are coming for my calculator
This is the lord’s work right here. Models at the the intersection of small and useful — forthcoming models, admittedly— will be in the toolset that the next generation of poor kids will use to build a more egalitarian world.
Thanks for your kind words :) hope to do something impactful one day with it
It appears that it's not you doing that but claude doing that.
That's programming in 2026. Right now, Claude it doing my job for me, but if it goes well enough I'm the one who's going to be promoted.
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)
Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.
ESP P4 or ESP S31?
challenge accepted, I have one lying around, will see.
AI skynet is winning. The spread of the slop is unstoppable now.