Last week I wrote that physical AI asks where it runs. The argument ended with a router: something that decides, task by task, whether inference runs on the device, at the edge (servers in many cities, close to the device), or in the cloud. I said I thought Cloudflare should build it.
Then I spent time at a conference with people building on these models, and I came back with a different view. This is not a someday product. The models that make it possible shipped this month, people are already building demos that need it, and nobody sells the layer in between. So I want to build it.
This post covers what I want to build, why, and what’s changing to make it necessary.
Inference Used to Be a Question
For the last three years, inference has had one shape:
Prompt → reason → output.
You ask a question. The model thinks. It answers. Then it waits until someone asks again. Every AI gateway on the market is built for that shape. It takes a request, picks a model and a provider, and returns a response.
Embodied AI has a different shape. A robot, a car, or a pair of camera glasses doesn’t ask questions. It watches. Its environment updates sixty times a minute, or sixty times a second, and the inference has to keep up. The question the model answers is no longer “what is the answer to this prompt?” It is:
“Has anything in my environment changed that updates what I believe, or that I need to react to?”
Most of the time, the answer is no. The room looks the same as it did a frame ago. The person is still talking. The road is still clear. The hard part isn’t generating a brilliant answer. It’s noticing, cheaply and constantly, the rare moment when something changes, and then calling in the expensive thinking only for that moment.
That is a different workload:
| Shape | Query inference | Stream inference |
|---|---|---|
| Input | One prompt | A constant flow of frames, audio, sensor data |
| Main question | ”What is the answer?" | "Has anything changed?” |
| How often | When someone asks | Every frame, all day |
| Most results | Useful | ”No change” |
| What matters | Quality of one answer | Cost and speed of the 99% of checks that find nothing, and catching the 1% that matter |
| Memory | Usually none | The current picture of the world, updated every frame |
You can’t serve this by sending every frame to a large language model. A camera at 30 frames per second produces 108,000 frames an hour. No one can afford an LLM call on each, and no LLM answers fast enough to keep up.
A New Kind of Model for the Stream
This month Cloudflare shipped two models that fit this shape better than anything before them.
Clef is a decision model. You give it a state (text, plus up to four images) and a list of questions with fixed options. It doesn’t write text. It returns a probability for every option, for up to 64 questions, in one pass. Cloudflare reports a median latency of about 209 milliseconds for Clef and about 39 milliseconds for its smaller sibling, Clef-flash. The weights are open, so anyone can download the model and run it.
Clairic is an extraction model, which Cloudflare calls a “System One” model, after the fast, automatic kind of thinking. It reads unstructured input and returns typed values: numbers, dates, spans of the original text, estimates with confidence intervals. It typically responds in under a second. In Cloudflare’s own example pipeline, splitting a job across Clairic, Clef, and an LLM took 19.3 seconds, against 61.7 seconds for the same job done by an LLM alone.
Put these next to the stream problem and the pattern is clear:
- Decision models ask the standing questions. “Is someone in front of me? Are they talking? Is anything in my path?” Every frame, fast and cheap, with a probability for each answer.
- Plain code decides when something changed. A threshold, a few frames in a row, a comparison with the last state.
- The large model is called only when it matters. Reasoning, conversation, planning: the slow, expensive thinking, triggered by a change, not by a clock.
This is how people work too. You don’t consciously reason about every frame of what you see. You notice when something is different, and then you think.
The Demo That Made It Concrete
Jordan Coeyman (@acoyfellow) built a demo on Clef that shows the pattern better than any description. A webcam feeds frames to Clef, and the agent answers live questions about the scene: is the person wearing a jacket, smiling, talking.
One mode is charades. The screen shows a word to act out, with no talking. Clef watches the camera and guesses from eight possible words, frame by frame. In one round, it was 94% sure of “Sleeping” by the sixth frame, about 4.2 seconds in, with each frame taking about 294 milliseconds.
The detail I keep coming back to is the rule printed at the bottom of the panel: the code calls the answer “when your word is top and over 60% for 2 frames in a row.”
That one line is the whole pattern:
- A fast model gives probabilities on every frame.
- Simple code holds a short memory and a threshold.
- An action fires only when the belief is strong and stable.
Now imagine that demo on a robot, or on glasses with a battery, or on a car moving through a city. Every one of those three steps has a best place to run, and the best place changes from second to second.
What I Want to Build
An AI gateway that chooses where inference runs, as well as which model and provider, and splits a stream into the parts that run in each place.
Today’s gateways treat a request as one unit: pick a model, pick a provider, send it. The gateway I want treats a stream as a pipeline, and places each stage:
| Stage | What it does | Where it usually runs |
|---|---|---|
| 1. Filter | Drop frames where nothing changed. Pixel difference, motion, a tiny model | On the device |
| 2. Decide | Ask the standing questions with a decision model (Clef-flash, Clef) | On the device or at the edge |
| 3. Hold state | Keep the current picture of the world. Apply thresholds and “N frames in a row” rules | At the edge, next to the decision model |
| 4. Escalate | When something changed, call an extraction model or an LLM with the context | At the edge or in the cloud |
| 5. Act | Send the result back to the device | On the device |
| 6. Learn | Log changes and outcomes for the fleet | In the cloud |
And the placement isn’t fixed. The gateway decides it live, from:
- The time budget. A robot avoiding an obstacle has milliseconds. A wearable suggesting what to say has a second or two.
- The device. How much compute it has, and how much battery is left.
- The network right now. Strong signal, weak signal, or none.
- Cost. Cheap decisions everywhere; expensive models only when needed.
- Privacy. Some frames should never leave the device. Only the decision should.
When the network drops, the gateway pulls the decision stage onto the device and keeps running. When the device is hot or low on battery, it pushes more up. When a stage on the device is unsure, it asks the edge for a second opinion.
What makes this possible now is that the same open-weight models can run at both ends. Clef-flash already runs locally: a community project runs it on a Mac’s webcam at about 250 milliseconds a frame. The same model family runs on Workers AI, Cloudflare’s service for running models across its network. A gateway can move a stage between the device and the edge without changing what the developer wrote.
What a developer would write:
- A stream, not a request: a WebSocket or WebRTC connection from the device.
- Standing questions: the schema of what to watch for, as Clef already takes it.
- Rules for change: thresholds and how many frames make a decision.
- Escalation: what to call, with what context, when a rule fires.
- A placement policy: time budgets and privacy rules per stage. The gateway handles the rest.
And what they would get back: per-stage latency, cost, and where each stage ran, for every stream.
Why Now
Four changes make this necessary, and they are all happening at once.
1. Much more inference on the device. Chips in robots, cars, and glasses are getting good enough to run small models, and open-weight decision models are small enough to fit. That doesn’t remove the cloud. It means every system is now split, and someone has to manage the split.
2. Inputs are streams. Cameras, microphones, and sensors don’t send prompts. The share of inference that is “watch and notice” will grow much faster than the share that is “ask and answer.”
3. The outputs are atoms. When a model’s output moves a robot arm or tells someone what to do mid-conversation, a slow answer is a wrong answer. Latency stops being a user-experience detail and becomes the product.
4. Models are splitting by job. Decision models, extraction models, and generative models now do different parts of one task. Cloudflare’s own example cut a pipeline from 61.7 seconds to 19.3 by giving each part to the right model. The next step is giving each part to the right place.
In my last post I estimated that robot fleets alone could need about 100x today’s off-device inference within five years. Most of that will be stream inference. Without a gateway that filters and splits the stream, that number is unaffordable. With one, most frames never leave the device, and the cloud sees only what changed.
Why the Gateways We Have Don’t Do This
Current AI gateways, including Cloudflare’s AI Gateway, are built for requests. They are very good at what they do: routing a prompt to a model, caching, retries, rate limits, logs, failover between providers.
But they have no idea of:
- a stream that lasts hours,
- a device on the other end with its own compute and battery,
- state that carries from one frame to the next,
- a change that should trigger a bigger model,
- or a stage that should move from the cloud to the device when the network drops.
Device platforms have the opposite gap. They deploy models to devices, but don’t decide, frame by frame, where each piece of inference should run.
The missing layer sits between them.
What I’m Building First
Here is where I’m starting:
- A working demo. A webcam or wearable stream, filtered on the device, decided by Clef-flash locally or Clef on Workers AI, with state in a Durable Object (a small Cloudflare server that holds one session’s memory) and an LLM called only on change. Measure frames in, frames escalated, latency per stage, and cost per hour.
- An SDK for the device end, so the same pipeline runs with or without a network.
- The placement policy, first as simple rules, later as something that learns from the logs.
- The numbers in public. How many frames out of 108,000 an hour actually need a large model? My guess is a tiny fraction. I want to measure it.
I’m building it as Callozum, at a company I’m calling System3 Inference. System3 is the part that decides between fast System 1 checks and slow System 2 thinking, and where each one runs. Callozum is named after the corpus callosum, which connects the two halves of the brain. It plugs into your app or your existing AI router, splits each request into parts, and places each part: dropped, on the device, on a local GPU, at the cloud edge, or in the cloud, with your data rules built in. If you are building on robots, wearables, cameras, or anything with rules about where data may go, join the Callozum waitlist. I want to talk to you.
Inference Is Becoming a Nervous System
The first wave of AI was a question and an answer. The next wave is a nervous system: reflexes on the device, quick judgments close by, deep thinking far away, and a constant flow between them.
Nobody designs a nervous system by asking “where should the brain run?” The answer is: everywhere, in parts, all the time. Inference is going the same way. The gateway that splits the stream and places each part is the piece I want to build.
Sources