The New Bottleneck in AI is Delivery

A 2019 Mac Pro is not supposed to run a 744-billion-parameter language model. It has a powerful processor and 256 gigabytes of memory, but the model proposed for this machine is far larger than that memory can comfortably hold. So the experiment begins with a move that sounds almost absurd: place most of the model on a bank of solid-state drives, then ask the computer to answer questions while continually fetching pieces of the model from storage. 6

At first, this sounds like trying to produce a live television broadcast using an archive that is mostly located across town. Surely the solution is simply to buy a larger, more expensive control room that can hold everything at once. That is how the AI industry has usually approached the problem: more memory, more accelerators, larger data centers. But the experiment asks a more interesting question. What if the control room does not need to hold the entire archive? What if it needs only the right piece of footage, patched into the switcher at the exact moment the director calls the next cut?

That question is possible because of a particular kind of artificial intelligence called a Mixture-of-Experts, or MoE model. An MoE contains many specialized computational components—called experts—but typically activates only a small subset for a given piece of text. A small routing system makes that selection. In the broadcast analogy, the model has thousands of specialized camera feeds, graphics, and audio sources available, but a director selects only the few relevant to the next moment of airtime. The total archive can be enormous even when the active production is small. 1

This is the source of both MoE’s promise and its problem. Selective activation reduces the amount of calculation required for a single word, but it does not make the inactive components disappear. Their learned parameters still have to exist somewhere. Early research into MoE offloading showed that this property could make large sparse models more practical on hardware with limited accelerator memory: the parts not needed immediately could be held on a slower tier and brought closer only when required. 2

The important word is when.

A computer has something like a live switcher, a near-play media server, and a deep archive. The switcher is the processor and its fast memory—often called RAM—where information is immediately available to be used. The archive is persistent storage, such as an NVMe solid-state drive, which can hold far more information at a lower cost but takes longer to deliver a particular item. A cache is the near-play server: a limited area where recently or predictably useful material is kept close at hand. An expert is not literally a piece of video footage. It is a large collection of learned numerical rules. But the broadcast analogy is useful because it preserves the part that matters: the switcher cannot cut to a source that has not arrived.

Imagine the director in the control room. The broadcast is live. The director has already loaded the main anchors and the studio graphics, which represent the model’s always-used components. Then the next piece of the broadcast requires a highly specialized piece of historical footage. If that footage is already on the near-play server, the cut happens instantly. If it is still in the deep archive, the director must wait while it is retrieved. The broadcast does not become more complicated. The production has simply been interrupted by logistics.

The analogy has an important limit. An ordinary television producer often has a rundown: a written sequence of planned cuts. An MoE has no fully written rundown. As it transforms each small piece of output through many layers, its router dynamically selects the expert components needed for the next calculation. The system must therefore predict a future source without knowing with certainty that it will be called. That is why this is not simply a scheduling problem. It is a measured problem of probabilistic staging under a limited memory and I/O budget.

The technical name for that interruption is foreground stall: the time the model is forced to wait for the expert data it needs after useful preparation has already done everything it can. It is the pause a user can actually feel, much like dead air or a frozen frame on television. The proposed experiment makes foreground stall per completed output token its main scorecard—not cache-hit rate, not raw drive bandwidth, not a prediction system’s accuracy in isolation. 6

That distinction may sound fussy until one considers a familiar mistake. Suppose a media archive boasts that 95 percent of requests are filled quickly. That figure is encouraging, but it does not tell the producer whether the one missing item is a background graphic or the only camera angle that shows the winning goal. A cache hit is valuable only when it prevents a meaningful wait. A fast drive is valuable only when the required data arrives before the processor reaches the point where it must stop. A prediction is valuable only when its early delivery removes a pause rather than clogging the network with footage no one uses.

This is why the proposal’s central idea is more important than any single optimization. It changes the question from “How do we make the drive faster?” to “How do we remove the wait the user can see?” The difference is subtle. It is also the difference between measuring a system and measuring the experience of using it.

The first proposed lever is prediction. In a normal archive, a producer asks for footage only after discovering that they need it. In a better control room, an assistant notices what the director is doing and cues up likely sources before the request is made. In an MoE system, the prefetcher can use several clues: the model’s current routing signals, patterns that tend to recur across layers, similarities to prior requests, recent activity, and longer-term usage history. The proposal treats these clues not as a crystal ball, but as weather forecasts. One forecast can be wrong. Several independent clues, calibrated against what actually happens, can tell the system how broadly it is safe to prepare.

That last qualification matters. A predictive system can be accurate and still make the computer slower. If it floods the storage queue with low-value guesses, it may delay the expert that the model truly needs. If it evicts a useful expert from the cache to make room for a speculative one, it can turn a correct idea into a worse outcome. FineMoE, a prior expert-offloading system, similarly uses fine-grained selection patterns and hints from prompts to guide prefetching, caching, and offloading; its published results are encouraging, but they were obtained on a different testbed and cannot be treated as a forecast for this machine. 3 The proposed experiment therefore places an explicit budget on guesses. A prediction must earn the space, bytes, and queue time it consumes.

The second lever is placement. Most caches answer a simple question: which source is used most often? The proposal asks a more useful one: which byte sitting near the switcher prevents the greatest amount of visible waiting? Those are not always the same source. A frequently requested expert that can be reached quickly may be less important than a moderately likely expert that would otherwise require a late, expensive storage read.

This is where the control-room analogy becomes more precise. There are at least three ways to make a delivery less painful. One is to keep the most consequential material on the near-play server. Another is to divide a large file into useful sections, leaving the early frames on the server so the broadcast can begin while the rest arrives. A third is to pull one source once and share it across multiple planned cuts. Research on fine-grained expert offloading and expert-splitting provides evidence that the boundaries of an expert need not be treated as fixed blocks for every deployment. 3 The new experiment does not claim to have invented this idea. Its narrower question is whether placement should be ranked by expected stall avoided per byte, under the specific realities of a large NVMe-backed model and a fixed memory budget. 6

The third lever is sharing. Here the explanation requires one more definition. Speculative decoding is a technique in which a model prepares several possible next words, then verifies them together. It is not permission to publish an unverified answer. Think of it as a producer lightly sketching the next few moments of the rundown before checking them against the authoritative script. The main broadcast system still verifies which cuts actually go to air.

For a storage-backed MoE, that draft is useful because it serves as an advance rundown. If several likely future words will need the same expert, one delivery from the archive may support several verified positions rather than just one. But this only helps when the draft is accepted often enough and when the set of experts required by the future words overlaps. If the draft is rejected, or every proposed word needs completely different sources, the supposed shortcut can create more work, not less.

This is another place where the proposal is unusually candid about prior art. Systems such as SP-MoE have already shown that speculative decoding and expert prefetching can be combined, using draft-model information and bounded prefetching to hide loading delay. 4 The experiment described here is not claiming that draft-guided expert prefetching is new. Its narrower hypothesis is that the target model’s own native multi-token-prediction capability may provide useful advance notice for a storage tier, without adding a separate draft model, and that this claim must be tested before it receives access to scarce I/O bandwidth. 6

The fourth lever concerns the pauses that averages hide. A broadcast may look smooth most of the time and still fail if, once in a while, it goes to black for several seconds. In a control room, that is the moment five urgently needed sources all turn out to be missing from the near-play server, while the network is already filled with speculative deliveries. In an MoE system, several experts can be absent together, and a prefetching system can compete with the real request it was intended to help.

That is why the experiment does not stop at average speed. It proposes measuring the long pauses—the p95 and p99 tail of the waiting-time distribution—as well as the number of late and unused prefetched bytes. It also separates an optional quality trade-off from the main, lossless path. In the optional mode, the system could choose an already-resident expert when the model’s own router indicates two choices are nearly tied. In television, this is like using a B-roll shot that is already loaded because the slightly better shot is stuck in the archive. But that could alter the model’s behavior. The proposal therefore keeps it switched off by default, demands separate approval for testing it, and refuses to hide a potential quality cost inside a performance claim. 6

This restraint is part of what makes the experiment significant. The stated default path is designed to preserve the model’s router decisions, weights, precision, and deterministic token output. In plain language, the team is not trying to make the AI faster by quietly changing what it says. They are trying to change the location and timing of the sources, not the contents of the sources or the director’s choice of shot. The open-source colibrì runtime, proposed as the experimental control, publicly describes the same principle: placement should alter speed without silently changing routing semantics or weight precision. 5

The broader lesson is not that every giant AI model can now run perfectly on any computer. It cannot. The public colibrì repository describes a storage-streamed 744-billion-parameter MoE runtime, but its own documentation shows how sharply performance changes when a host moves between disk, RAM, and accelerator-resident regimes. Those are software-artifact measurements, not guarantees for the Mac Pro experiment. 5 Nor is the proposal a promise of a particular speedup. It is a carefully bounded test with fixed memory budgets, cold and warm storage conditions, interleaved comparisons, exactness checks, and explicit conditions under which a mechanism should be abandoned. 6

That is the actual technical importance. The experiment refuses the easy story in which a collection of optimizations simply stacks into a dramatic number. A better predictor, a smarter cache, a partial-residency scheme, and speculative decoding all draw from the same pools of memory and storage bandwidth. Their gains can interfere. One can starve another. The right result may be a measured negative: perhaps the model’s routing is not predictable enough, perhaps the storage system is already fast enough, perhaps the processor becomes the bottleneck once the waiting is removed. A negative answer is still an answer about where investment should go next.

At first glance, this is a narrow systems-engineering question. It is about storage queues, memory tiers, and the timing of expert parameters. But it exposes a much larger shift in the AI economy. For years, the usual measure of computational possibility has been capacity: How many parameters can the model contain? How much data can the cloud hold? How much hardware can be assembled in one place?

Capacity matters. But it is no longer sufficient.

A system can possess extraordinary resources and still fail at the moment of use if the necessary component is unavailable, delayed, or controlled by a separate intermediary. In the new AI economy, value increasingly depends on whether the right capability can be held close enough to act. The real unit of performance is not the total size of the archive. It is the arrival of the right thing before the broadcast goes to black.

This is where the experiment connects naturally to Onli. Onli does not replace an AI runtime, choose an MoE expert, or make an NVMe drive faster. The relevance is architectural. Actual possession means directly holding a digital asset rather than merely holding an account-based claim that a distant system will make the asset available. A directly held asset can be transferred, governed, and put to work from the owner’s own environment; a custodial claim remains dependent on the continuing availability, rules, and permission of an intermediary.

The parallel is not perfect, and it should not be forced. An AI expert is not a financial instrument. But both problems reveal the same design principle: control has a timing dimension. Permission that arrives after an action is required is not the same as capability. Access that depends on a distant system remaining available is not the same as possession. In both cases, the question is not merely whether the thing exists somewhere. It is whether it is held, governed, and ready where the next decision must be made.

That is why Onli’s emphasis on direct digital possession matters beyond the usual language of ownership. As AI agents, private instruments, and programmable assets become more active, the architecture beneath them must answer a practical question: who can make the asset available, under what rules, and without turning every action into a request to a central archive? The answer will shape not only speed, but resilience, governance, privacy, and the ability to form private digital markets around assets that remain genuinely controllable by their owners.

The assumption becoming obsolete is that scale requires centralization, or that larger systems are inevitably less local, less direct, and less governable. Sparse AI suggests another possibility. A system can be immense without calling everything into action at once—provided it can accurately place, deliver, and verify the small part that matters now.

The defining question is not how large the archive becomes. It is whether the right source is already patched into the switcher when the broadcast must go live.

References

Source note

This article synthesizes three unpublished technical papers supplied to The Onli Corporation in July 2026: Expected-Stall Minimization for Storage-Tier Mixture-of-Experts Inference (v0.6), Making Giant Sparse AI Models Practical on Existing Hardware (v1.0), and the MoE Experiment Executive Summary. Descriptions of the proposed testbed, controls, hypotheses, and criteria are drawn from those materials. They describe a planned, falsifiable experiment—not completed benchmark results.

---

About The Onli Corporation

The Onli Corporation develops infrastructure for creating, financing, delivering, and governing entirely new classes of digital assets. Through actual possession, private digital markets, and programmable digital property, Onli enables organizations to build secure, scalable digital economies beyond the limitations of traditional blockchain architectures.

To learn more, contact hello@theonlicorporation.com.

Download OnliYou on the Apple App Store.

Develop on the Onli platform at https://onli.cloud.