Expected Foreground Stall

A Measurement and Control Framework for Storage-Tier Mixture-of-Experts Inference

Prior-art-audited research proposal and experimental protocol

July 26, 2026

Status. This manuscript proposes a measurement framework, a control principle, and a pre-registered experimental program. It does not report new end-to-end performance results. All claimed performance numbers are attributed to prior work or to public software artifacts. Any result from the proposed Mac Pro testbed will be reported only after the instrumentation, exactness checks, and protocol described here are complete.

---

Abstract

Sparse Mixture-of-Experts (MoE) models activate only a small subset of their expert parameters for each token. This makes it possible, in principle, to serve models larger than fast memory by retaining a dense shared trunk in RAM and streaming routed experts from slower tiers such as NVMe. The central systems problem is not simply storage bandwidth or cache hit rate. It is whether the required expert data becomes available before the decode loop reaches the point at which no useful computation can continue.

Existing work attacks many parts of this problem: expert offloading and caching, residual-stream and spatio-temporal prediction, distributional expert-map retrieval, learned replacement, mixed precision, vertical expert splitting, CPU–GPU co-execution, speculative decoding, draft-guided prefetching, co-activation-aware placement, and bounded-loss expert substitution. The literature is therefore richer—and the remaining novelty narrower—than earlier versions of this manuscript recognized. In particular, recent work already covers multi-signal prediction, speculative-activation utility scheduling, co-activation-based placement, and uncertainty-gated substitution. The unresolved gap is not the absence of another predictor. It is the absence of a common, directly measurable objective that distinguishes a correct prediction from a useful and timely one across heterogeneous storage tiers and speculative execution.

We propose expected foreground stall per committed token: the measured duration for which the main forward pass cannot perform useful model computation because required expert bytes are unavailable, divided by the number of output tokens actually accepted and committed. The metric explicitly accounts for asynchronous overlap, prefetch deadlines, batch-union deduplication, rejected speculative positions, correlated miss bursts, and heterogeneous service costs. We then formulate a stall-valued control principle: prefetch, placement, and eviction actions should be ranked by their estimated reduction in residual foreground stall per constrained byte or queue slot—not by activation probability, hit rate, or prediction recall alone.

The paper makes three deliberately bounded contributions. First, it defines an implementation-oriented stall metric and trace schema that can causally separate predictor quality from actuator timeliness. Second, it derives a deadline- and congestion-aware value function applicable to any established prediction source. Third, it proposes a longitudinal, user-specific extension of distributional expert-map retrieval and a protocol for testing whether cross-session stationarity has practical stall value on a personal machine. A prior-art ledger distinguishes these claims from inherited mechanisms.

The proposed evaluation uses a pinned colibrì runtime and a 2019-generation Intel Mac Pro with 256 GB RAM and a four-NVMe RAID-0 cold tier. Fixed memory regimes, macOS page-cache controls, token-exact validation, paired interleaved runs, and explicit falsifiers are used to test whether foreground stall predicts end-to-end latency better than conventional metrics and whether stall-valued decisions outperform probability-only policies at equal resource budgets.

---

1. Introduction

Mixture-of-Experts models decouple total parameter count from the amount of computation used for one token. A router selects a small set of expert feed-forward networks at each sparse layer, allowing the full model to be very large while the active parameter set remains much smaller. This sparsity creates a deployment option that dense models do not provide: inactive experts can reside outside fast memory and be transferred only when needed.

That option changes the inference problem from one of permanent residency to one of timed delivery.

A storage-tier MoE runtime typically keeps embeddings, attention, normalization, routers, shared experts, and other frequently used components in RAM. Routed expert weights occupy most of the model and may live in an application-managed RAM cache, the operating system's page cache, or an SSD/NVMe backing store. At decode time, the router reveals which experts are required. If a required expert is already resident—or if a prefetch completed early enough—its storage location affects speed but not semantics. If it is not ready, the forward pass eventually reaches a barrier and waits.

The distinction between data-transfer time and visible waiting time is fundamental. A 20 ms read that overlaps 25 ms of useful computation adds no foreground delay. A 5 ms read submitted too late may add the full 5 ms. Raw bandwidth cannot express this difference. Neither can conventional cache hit rate, which assigns equal value to hits that avoid very different costs. Prediction recall also cannot express it: a forecast can be correct but redundant, late, or harmful to the shared I/O queue.

A public colibrì experiment illustrates the problem. A merged cross-layer coupling predictor improved held-out expert-recall relative to a marginal baseline, yet produced neutral performance at a modest prefetch budget and a slowdown at a larger budget on a warm-cache M3 Max. The predictor learned real routing structure; the actuator had little foreground stall left to remove, and aggressive readahead created additional traffic. This is not a contradiction. It is evidence that prediction accuracy and latency value are different quantities.

Earlier versions of this paper responded by proposing four mechanisms—prediction, placement, speculative amortization, and tail control—as if the novelty lay primarily in combining them. A July 2026 prior-art audit changes that position. FineMoE already uses semantic and trajectory retrieval over probability-distribution expert maps. ST-MoE combines cross-layer and cross-token signals. DALI combines residual prediction with workload-aware caching on local PCs. MoE-SpAc uses speculative decoding as a memory-management sensor and unifies prefetch and eviction under a utility framework. CAP and public software artifacts use co-activation for placement. BuddyMoE and related systems use uncertainty and expert similarity to gate substitutions. Draft-guided expert prefetching is established by SP-MoE and MoE-SpeQ.

The academic contribution must therefore be narrower and more precise.

This paper asks a different question:

What measurement and control objective should determine whether any prediction, placement, or speculative action is useful on a storage-tier MoE runtime?

We answer with expected foreground stall per committed token and a corresponding stall-valued control plane.

The paper makes the following contributions:

1. An implementation-oriented objective. We define foreground stall as the empirically measured time during which the model's critical forward path cannot execute useful work because required expert bytes are unavailable. We normalize by committed—not merely drafted or evaluated—tokens.

2. A causal decomposition and trace schema. The instrumentation separates prediction, admission, issue time, completion time, need time, queue state, service bytes, cache tier, speculative acceptance, and actual critical-path waiting. This makes it possible to determine why a high-recall predictor did or did not create a speedup.

3. A stall-valued control principle. Any candidate action is valued by its expected reduction in residual foreground stall after accounting for its deadline, byte cost, queue occupancy, cache pollution, and competing demand traffic. Established predictors become inputs to the controller rather than contributions claimed as new.

4. A bounded longitudinal hypothesis. We propose evaluating a persistent, local, user-specific store of routing distributions across sessions. FineMoE establishes expert-map retrieval, and MoE-Infinity establishes the personal-machine setting; the proposed delta is a longitudinal user-specific evaluation tied directly to avoided stall rather than hit rate alone.

5. A falsifiable experimental protocol. We specify a pinned open runtime, a machine with a model larger than RAM, fixed memory regimes, cold-cache controls, exactness validation, statistical treatment, and observations that would falsify each central hypothesis.

The remainder of the paper first reviews the current literature and explicitly narrows the novelty claims. It then defines the system model and foreground-stall objective, derives the control framework, states claims and non-claims, and presents the measurement program.

---

2. Background and related work

2.1 Expert offloading and local inference

Early MoE offloading systems established that sparse activation can be exploited on memory-constrained hardware. Eliseev and Mazur showed that expert weights can be offloaded and prefetched on consumer systems. MoE-Infinity explicitly targets single-user personal machines and uses sparsity-aware activation traces to guide an expert cache. Fiddler reduces CPU–GPU data movement by choosing whether to transfer or compute experts on the CPU. Pre-gated MoE changes the model to reveal expert demand earlier. EdgeMoE places expert weights on external storage for mobile inference and combines preloading with expert-specific precision. PowerInfer-2 decomposes weight movement and computation into fine-grained neuron clusters to coordinate mobile storage and heterogeneous compute.

Recent systems move closer to the setting considered here. FlashMoE targets SSD-resident experts under limited RAM and learns a cache policy from recency and frequency. DALI targets local PCs and combines dynamic CPU/GPU assignment, residual-based prefetching, and workload-aware replacement. The colibrì software artifact streams routed experts for a 744B-parameter GLM-5.2 container from disk, maintains a learning cache, supports asynchronous I/O and native multi-token prediction, and publishes token-exact validation checks.

These systems establish that local and storage-tier inference are not new problem settings. The present work uses them as foundations and baselines.

2.2 Predicting expert demand

Expert demand has structure across several timescales.

Residual and intermediate-state prediction. Offloading work by Eliseev and Mazur exploits residual similarity to predict the next layer's experts. ProMoE uses intermediate results to proactively cache experts. ExpertFlow trains a routing-path predictor and combines it with token scheduling and predictive caching. DALI uses residual information to identify high-workload experts.

Distributional retrieval. FineMoE introduces an iteration-level expert map containing the full routing-probability distribution across layers. It retrieves historical maps using prompt-semantic and partial-trajectory similarity, uses similarity to adapt prefetch breadth, and manages a bounded map store. This is materially richer than request-level activation counts.

Spatio-temporal correlations. ST-MoE combines adjacent-layer and previous-token correlations in a lightweight table-based predictor. Public colibrì tooling likewise captures cross-layer coupling and can predict one or two layers ahead. These results mean that generic multi-signal or cross-layer-plus-temporal fusion cannot be claimed as novel here.

Model dependence. Liang et al. show that local routing consistency varies substantially across checkpoints and propose metrics for cacheability and future-information value. A prediction policy that works on one checkpoint may have little transferable value on another. Any evaluation must therefore characterize the checkpoint rather than assume universal locality.

The open issue is not whether experts can be predicted. It is how to determine which predictions are worth acting on under a real storage queue and a specific readiness deadline.

2.3 Caching, byte reduction, and placement

Caching policies range from LRU/LFU and hybrid recency-frequency scores to learned replacement. FlashMoE demonstrates that a lightweight learned policy can improve SSD-tier reuse. FineMoE uses retrieved probability maps to guide caching and eviction. MoE-Infinity and colibrì preserve workload history to seed future residency.

Other systems reduce the cost of a miss. HOBBIT substitutes lower-precision versions for less critical cache-miss experts. MoEpic splits each expert vertically so that more experts can be partly resident and remaining bytes can overlap resident computation. PowerInfer-2 applies fine-grained storage and compute scheduling at neuron-cluster granularity.

Placement based on co-activation is also established in adjacent domains. CAP groups co-activated experts to reduce distributed communication. A public mbolt software artifact reports profile-guided co-activation ordering and tensor interleaving for storage layouts. These do not remove the need to study layout on a striped NVMe tier, but they do remove any broad claim that co-activation-aware placement is new.

2.4 Speculative decoding and expert movement

Speculative decoding evaluates multiple candidate positions and accepts a verified prefix. Its interaction with MoE systems has become a substantial literature.

MoESD analyzes when sparse MoE models benefit from speculative decoding and introduces target efficiency to capture effects not explained by acceptance rate alone. SpecMoEOff uses speculation to enlarge per-expert workloads and autotunes the compute–transfer balance. SP-MoE uses the auxiliary draft model's execution to predict target-model experts and prefetches them under a cutoff-layer policy. MoE-SpeQ likewise predicts future expert sequences with a small on-device draft model and uses an amortization roofline to govern the strategy. MoE-SpAc treats speculative decoding as an activation sensor, estimates expert utility, and coordinates prefetch, eviction, and heterogeneous execution.

These works establish both draft-guided expert prefetching and utility-based speculative scheduling. The use of a target model's native multi-token-prediction head may still be operationally attractive because it avoids an auxiliary model, but it is a system adaptation—not a new idea class.

2.5 Bounded-loss responses to misses

Several systems change model execution to reduce miss penalties. Cache-conditional routing prefers resident experts. BuddyMoE identifies similar resident “buddy” experts and gates substitutions using routing uncertainty, residency distribution, co-activation, and optional probability margins. Madan et al. show that executing speculated rather than true selected experts can preserve useful accuracy in some regimes. HOBBIT changes the precision of selected experts. CAP includes communication-aware pruning.

These methods are relevant baselines for tail-latency control, but they violate the strict lossless invariant used for the primary experiments in this paper. Expert substitution and expert reduction are therefore excluded from the default path and evaluated, if at all, in a separately authorized bounded-loss appendix.

2.6 The remaining gap

The reviewed systems optimize different intermediate targets:

• expert hit rate;
• prediction recall or accuracy;
• bytes transferred;
• transfer–compute balance;
• time per output token;
• throughput;
• hardware utilization;
• accuracy–latency trade-offs.

These metrics are all useful, but they do not provide the same causal information. In particular, they do not necessarily reveal whether a correctly predicted expert was already resident, whether it completed before its deadline, whether its speculative load delayed a demand load, whether multiple positions shared the load, or whether a correlated burst determined the tail.

To our knowledge after the July 2026 audit, no published framework uses measured critical-path expert unavailability per committed token as the common objective for prediction admission, placement, speculative amortization, and tail reporting on a storage-tier runtime. The claim is not that earlier work ignores latency. The narrower claim is that the proposed objective joins timeliness, heterogeneous service, accepted-token normalization, batch-union growth, and correlated critical-path waiting in one implementation-level quantity.

---

3. System model

3.1 Model and memory hierarchy

Consider an autoregressive MoE model with:

• $L$ routed MoE layers;
• $J_l$ routed experts at layer $l$;
• router-selected set $R_{j,l}(p)$ for evaluated position $p$ in decode step $j$;
• a dense or always-used trunk that remains resident in fast memory;
• routed expert weights distributed across one or more slower tiers.

The target hierarchy in this study has three observable levels:

1. Application-managed resident RAM, including pinned/hot experts and reusable expert slabs;
2. Operating-system page cache, an implicit tier whose contents are not fully controlled by the runtime;
3. NVMe backing storage, containing the immutable model container.

The page cache must be modeled explicitly in experiments. A predictor that appears useless on a warm host may be valuable on cold storage, while a broad readahead policy may merely warm pages without increasing the runtime's own cache hits.

3.2 Decode steps, evaluated positions, and committed tokens

Without speculation, one forward step evaluates one position and commits one token. With speculative decoding, step $j$ evaluates $A^{\mathrm{pos}}_j \ge 1$ positions and commits an accepted prefix of $A_j$, where:

$$ 1 \le A_j \le A^{\mathrm{pos}}_j. $$

Rejected positions still consume routing, expert loading, and computation. At layer $l$, the runtime must serve the union of experts required by all evaluated positions:

$$ U_{j,l} = \bigcup_{p=1}^{A^{\mathrm{pos}}j} R{j,l}(p). $$

A batch-union implementation loads each unique expert or expert segment once for that layer and reuses it across positions. The quantity $|U_{j,l}|$ therefore grows sublinearly, linearly, or nearly linearly with $A^{\mathrm{pos}}_j$ depending on routing overlap. Speculation is storage-profitable only when committed-token gain outpaces union-byte growth and any additional queue contention.

3.3 Deadlines and service events

For each required expert or segment $e \in U_{j,l}$, define:

• $t^{\mathrm{issue}}_{j,l,e}$: time its load is submitted;
• $t^{\mathrm{ready}}_{j,l,e}$: time the required bytes become usable by the compute path;
• $t^{\mathrm{need}}_{j,l,e}$: latest time at which those bytes can become ready without stalling useful work;
• $b_{l,e}$: bytes transferred from the serving tier;
• $z_{j,l,e}$: serving tier and action source, such as resident, OS-cache hit, demand read, router lookahead, trajectory retrieval, or speculative draft.

A simple lateness variable is:

$$ \ell_{j,l,e} = \max\left(0, t^{\mathrm{ready}}{j,l,e} - t^{\mathrm{need}}{j,l,e}\right). $$

However, summing $\ell_{j,l,e}$ over experts generally overcounts stall because reads can complete concurrently, resident experts can compute while cold experts load, and segmented execution may create multiple opportunities for overlap. The primary metric must therefore be observed at the critical path rather than reconstructed by an independence assumption.

---

4. Expected foreground stall

4.1 Empirical definition

For step $j$ and layer $l$, let $\mathcal{I}_{j,l}$ be the union of time intervals during which:

1. the main forward path has reached work whose required expert data is unavailable;
2. no other useful model computation on that critical path can proceed; and
3. the wait is attributable to expert-data readiness rather than unrelated scheduling, sampling, logging, or operating-system activity.

The foreground expert stall at that layer is:

$$ W_{j,l} = \mu\left(\mathcal{I}_{j,l}\right), $$

where $\mu$ denotes elapsed duration.

In a simple barrier implementation that computes all resident work before joining missing experts, this may reduce approximately to:

$$ W_{j,l} \approx \max\left(0, \max_{e \in U_{j,l}} t^{\mathrm{ready}}{j,l,e} - t^{\mathrm{barrier}}{j,l}\right), $$

but the trace-derived interval definition is preferred because it remains valid under segmented computation, multiple joins, and concurrent I/O.

Across $N$ decode steps, the empirical foreground stall per committed token is:

$$ \widehat{S}N = \frac{\sum{j=1}^{N}\sum_{l=1}^{L} W_{j,l}} {\sum_{j=1}^{N} A_j}. $$

The population objective is the long-run limit or corresponding ratio of expectations when the process is stationary:

$$ S = \frac{\mathbb{E}\left[\sum_{l=1}^{L} W_{j,l}\right]} {\mathbb{E}[A_j]}. $$

The accepted-token denominator prevents speculative execution from receiving credit for positions that are evaluated but rejected.

4.2 Relation to end-to-end latency

Foreground stall is not the whole of decode latency. For a measured interval or step, let:

• $C_j$ be useful model compute;
• $W_j = \sum_l W_{j,l}$ be expert-readiness stall;
• $O_j$ be other overhead, including sampling, synchronization not caused by data readiness, tracing, and runtime management.

Then:

$$ T_j = C_j + W_j + O_j $$

under a non-overlapping accounting partition. At the aggregate level, time per committed output token is approximately:

$$ \mathrm{TPOT} = C_{\mathrm{tok}} + S + O_{\mathrm{tok}}. $$

The framework does not assume that $S$ always dominates. On a warm, high-residency, or compute-limited host, $S$ may approach zero and additional prefetching should have little value. That regime sensitivity is a feature of the metric: it identifies when the optimization target has disappeared.

4.3 Supporting metrics

$S$ is the primary objective, but diagnosis requires additional measurements.

Timely required-byte recall. Among expert bytes that were not already ready when a prediction was made, what fraction completed before their need deadlines?

$$ R^{\mathrm{timely}} = \frac{\sum b_{l,e},\mathbf{1}[e\ \mathrm{required}\ \wedge\ t^{\mathrm{ready}} \le t^{\mathrm{need}}]} {\sum b_{l,e},\mathbf{1}[e\ \mathrm{required}\ \wedge\ e\ \mathrm{not\ already\ ready}]}. $$

This distinguishes useful prefetches from correct-but-late predictions.

Unused speculative bytes. Bytes loaded by a prediction source that are not used before eviction or the end of a defined horizon.

Lead slack. For timely prefetches:

$$ \mathrm{slack}{j,l,e} = t^{\mathrm{need}}{j,l,e} - t^{\mathrm{ready}}_{j,l,e}. $$

Large positive slack may indicate that the load was issued earlier than necessary and occupied scarce resources unnecessarily.

Union bytes per committed token. Let $B(U_{j,l})$ be unique expert bytes required by the evaluated union. Then:

$$ B^{\mathrm{union}}{\mathrm{tok}} = \frac{\sum{j,l} B(U_{j,l})}{\sum_j A_j}. $$

This is the storage-side profitability measure for speculation.

Stall-conversion efficiency. Relative to a control policy with stall $S_0$, a policy using speculative bytes $B^{\mathrm{spec}}_{\mathrm{tok}}$ has:

$$ \eta^{\mathrm{stall}} = \frac{S_0 - S_{\mathrm{policy}}} {B^{\mathrm{spec}}_{\mathrm{tok}}}. $$

A policy can have high prediction recall but near-zero or negative $\eta^{\mathrm{stall}}$.

Tail stall. For each step define $s_j = W_j/A_j$. Report the empirical median, p90, p95, p99, and maximum of $s_j$, together with burst size and queue occupancy. The normalized step measure is not a claim that stall is experienced uniformly across all accepted tokens; it is a reproducible allocation for comparing speculative steps of different accepted lengths.

4.4 Four interacting factors

The metric supports a conceptual decomposition into four interacting factors:

1. Readiness probability. The probability that a required expert is not ready by its deadline, after accounting for residency and prefetch.
2. Residual service cost. The critical-path delay caused when required bytes are late, conditional on tier, byte count, concurrency, and overlap.
3. Committed-token amortization. The amount of accepted output served by the union of expert data loaded for a step.
4. Correlation and tail. The joint structure of expert bursts, queue contention, and barrier completion that determines high quantiles.

These factors are not statistically independent, and $S$ is not generally equal to a simple product of four scalar terms. Product intuition is useful only as a first-order approximation. For example, reducing late-miss probability and miss bytes may compound when the I/O queue is underutilized, but the gains may become sub-additive if both mechanisms compete for the same queue or cache slots.

4.5 Relation to earlier analytical models

FineMoE formulates offloading and caching around on-demand loading latency under its system assumptions. SpecMoEOff, MoE-SpeQ, and MoESD model the interaction between speculative workloads, acceptance, computation, and transfer. FlexGen uses linear-programming search to optimize dense-model offloading schedules for throughput-oriented serving.

The present formulation should not be described as a strict mathematical generalization of all these models. It instead extends the scope of what is measured in a storage-tier interactive runtime by making five quantities explicit at once:

• the actual readiness deadline;
• heterogeneous serving tiers and byte costs;
• accepted versus merely evaluated positions;
• union-based reuse across speculative positions;
• correlated critical-path waiting rather than independent per-expert miss costs.

Its intended value is experimental comparability and causal diagnosis, not a claim of a universally optimal closed-form scheduler.

---

5. Stall-valued control

5.1 Predictions are inputs, not objectives

Let $x_{j,l}$ denote the runtime state available before layer $l$ needs its experts. It may contain any combination of established signals:

• current-layer residual or router-lookahead logits;
• cross-layer coupling tables;
• previous-token activation history;
• FineMoE-style semantic or trajectory retrieval;
• session heat and long-term frequency;
• draft-model or native-MTP information;
• cache residency, OS-cache evidence, and current queue state.

A predictor produces an estimate of future need. The controller decides whether acting on that estimate is worth the cost. Separating these roles prevents a predictor from being evaluated solely by recall.

5.2 Deadline-aware need probability

For candidate expert $e$ at layer $l$, define a deadline-aware probability:

$$ q_{j,l,e} = \Pr\left( e \in U_{j,l}\ \wedge
e\ \mathrm{not\ otherwise\ ready\ by}\ t^{\mathrm{need}}{j,l,e} \mid x{j,l} \right). $$

This differs from ordinary activation probability. An expert that is likely to be selected but already resident has low actionable need. An expert likely to be selected far in the future may have sufficient time for a later load. Calibration should therefore target the event that matters to the controller: late unavailability without the proposed action.

Calibration can be implemented with logistic, isotonic, or other lightweight methods. The experimental requirement is not a particular estimator. It is that predicted probabilities be evaluated with Brier score, expected calibration error, reliability plots, and held-out sessions before they are used to widen prefetch breadth.

5.3 Expected stall benefit

For an action $a$—such as prefetching an expert, retaining it in RAM, or evicting an alternative—define the expected gross benefit:

$$ G(a \mid x) = \mathbb{E}\left[W^{\mathrm{base}} - W^{a} \mid x\right]. $$

This directly asks how much foreground stall the action is expected to remove.

The action also has external costs:

• $C^{\mathrm{queue}}(a)$: added stall imposed on other loads through queue contention;
• $C^{\mathrm{pollution}}(a)$: expected future stall caused by evicting or displacing more valuable data;
• $C^{\mathrm{cpu}}(a)$: submission, prediction, decompression, or bookkeeping overhead;
• $b(a)$: constrained bytes or resident capacity consumed.

A normalized action value is:

$$ V(a \mid x) = \frac{G(a \mid x) - C^{\mathrm{queue}}(a) - C^{\mathrm{pollution}}(a) - C^{\mathrm{cpu}}(a)} {b(a)}. $$

For a prefetch candidate, a practical approximation is:

$$ G(a) \approx q_{j,l,e}, \Pr(t^{\mathrm{ready}} \le t^{\mathrm{need}} \mid \mathrm{queue}, a), \widehat{\Delta W}_{j,l,e}, $$

where $\widehat{\Delta W}_{j,l,e}$ is the estimated stall that would occur without the action. This value automatically falls toward zero when the expert is already warm, the deadline cannot be met, the miss would be cheap, or the host is compute-bound.

5.4 Budgeted prefetch admission

Given candidate actions $\mathcal{A}$, a byte budget $B$, and a queue-slot budget $Q$, the idealized admission problem is:

$$ \max_{y_a \in {0,1}} \sum_{a \in \mathcal{A}} y_a\left(G(a)-C(a)\right) $$

subject to:

$$ \sum_a y_a b(a) \le B, \qquad \sum_a y_a qslots(a) \le Q. $$

Exact additivity rarely holds because experts co-activate, reads may merge, and cache pollution interacts across actions. The implementation can therefore use a greedy approximation, bundle co-activated candidates, or learn marginal values from replay. The contribution is the objective—not a claim that one solver is universally optimal.

Demand reads have strict priority. Speculative loads should be cancelable when their deadlines pass, their probability collapses, or queue pressure crosses a threshold. The controller should expose its rejected candidates so that low recall can be distinguished from deliberate admission control.

5.5 Stall-valued placement and eviction

Over a longer horizon, the same principle applies to residency. The marginal value of keeping expert $e$ resident is the expected foreground stall avoided over the placement horizon, net of the opportunity cost of the bytes it occupies:

$$ V^{\mathrm{resident}}(e) = \frac{\mathbb{E}[\Delta W_e \mid \mathrm{resident}] - \mathbb{E}[C^{\mathrm{displace}}_e]} {b_e}. $$

This can rank experts differently from raw frequency. A highly frequent expert already served reliably by the OS page cache may have lower marginal application-cache value than a moderately frequent expert whose cold miss creates a long barrier. Likewise, a large expert with frequent reuse may be less valuable per byte than several smaller or split segments.

Vertical splitting, mixed precision, and co-activation layout are prior-art mechanisms that can be evaluated under this value function. Their role in the present paper is not novelty; it is to test whether $S$ correctly assigns value across mechanisms with different effects on bytes, overlap, and tail.

5.6 Longitudinal user-specific routing memory

FineMoE establishes a store of iteration-level routing distributions and retrieves maps using semantic and trajectory similarity. MoE-Infinity establishes that personal machines often operate at batch one and exhibit reusable sparse traces. The proposed extension is a long-lived, user-specific, cross-session map store with three properties:

1. It persists across days and sessions rather than being evaluated only as a deployment-wide historical store.
2. It separates short-term session state from long-term user state, allowing online decay without discarding durable patterns.
3. It is judged by calibrated late-need probability and stall saved, not by expert overlap alone.

The hypothesis is that a single user's recurring tasks create greater map self-similarity than a pooled multi-user history and therefore improve deadline-aware decisions at the same speculative-byte budget. This is a testable deployment hypothesis, not an established result.

The store also creates privacy obligations. Full prompts need not be retained. Routing distributions, compressed trajectory signatures, and coarse metadata can remain local. If semantic embeddings are stored, the implementation should document retention, encryption, deletion, and the possibility that embeddings reveal information about prompts.

5.7 Native MTP as a lead-time source

A target model's native multi-token-prediction head can expose candidate future positions without an auxiliary draft model. This may reduce memory overhead and avoid a second checkpoint, but draft-guided expert prefetching itself is established prior art.

In the proposed framework, native MTP is evaluated through two quantities:

• additional lead time made available before expert deadlines;
• change in union bytes per committed token.

The controller should not assume that higher acceptance automatically improves storage performance. A draft policy is profitable only when:

$$ \frac{B^{\mathrm{union}}{\mathrm{with\ draft}}} {A{\mathrm{with\ draft}}} < \frac{B^{\mathrm{union}}{\mathrm{control}}} {A{\mathrm{control}}} $$

or when additional lead time reduces $S$ enough to offset any increase in union bytes and draft computation. Grammar-constrained structured output is a useful high-acceptance condition, but it remains an experimental condition rather than a promised gain.

5.8 Lossless tail control

The primary tail controller remains semantics-preserving:

• bounded speculative-byte and queue-slot budgets;
• demand-read priority;
• stale-prefetch cancellation;
• per-source admission caps;
• queue occupancy and deadline monitoring;
• backoff when $\eta^{\mathrm{stall}}$ becomes non-positive;
• p95/p99 stall as first-class objectives.

Expert substitution, route modification, pruning, and precision changes belong to a separate bounded-loss program. BuddyMoE, cache-conditional routing, HOBBIT, YALIS-style speculative execution, and CAP provide suitable baselines. They are not contributions of this paper and are not mixed into the lossless headline result.

---

6. Claims and non-claims after the prior-art audit

The load-bearing claims are the metric, trace, and control objective. The longitudinal store is explicitly contingent on showing a held-out user-specific advantage.

---

7. Experimental methodology

7.1 Research questions

The program addresses six questions:

1. Does foreground stall explain end-to-end decode latency better than hit rate, predictor recall, or raw I/O bandwidth across storage regimes?
2. Can a controller convert established prediction signals into lower stall more reliably by ranking actions by expected stall avoided rather than probability alone?
3. Does a long-lived user-specific routing-distribution store improve calibrated late-need prediction and stall at equal byte budgets?
4. When do placement, splitting, layout, and native MTP reduce stall, and when do they merely move the bottleneck?
5. Can lossless queue control reduce tail pauses without changing model semantics?
6. How sub-additive are individually successful mechanisms when they share the same cache and I/O queue?

7.2 Testbed

The primary machine is a 2019-generation Mac Pro configured as follows:

The exact CPU brand string, physical/logical core counts, compiler version, OpenMP runtime, APFS/RAID configuration, stripe characteristics where observable, firmware, and NVMe health are captured before the run. The paper does not assume a specific Xeon W SKU until verified on-machine.

The model container is the approximately 372 GB GLM-5.2 colibrì int4 conversion with verified int8 MTP heads. It resides on the RAID. Irreplaceable traces and source changes are backed up outside the RAID because RAID-0 provides no redundancy.

7.3 Reference runtime

Colibrì is pinned to commit:

44e489b196c9b7876b3d37a0570ebf1c6f90f54c

The runtime is a control and implementation substrate, not a contribution of this paper. The unchanged revision is characterized before any research change. Each mechanism and instrumentation change lives on a separate branch or commit, and every result records the full code hash and diff.

The primary path remains CPU-only. The discrete Radeon is excluded because the documented colibrì Metal path targets Apple Silicon's unified-memory architecture; adding an unvalidated discrete-GPU path would introduce another transfer hierarchy and confound the storage study.

7.4 Fixed resource regimes

Automatic memory sizing is disabled for confirmatory runs. Candidate RAM budgets are:

• 64 GB: severe storage pressure;
• 128 GB: moderate pressure;
• 192 GB: high residency with meaningful cold-tier use;
• 208 GB: highest candidate budget, subject to the runtime's safety planner and measured free memory.

The highest budget is not forced if projected peak memory risks compression or swap. The final confirmatory budget set is frozen after a safety-only planning pass, not after performance inspection.

I/O worker count is chosen in a separate pre-experiment queue-depth sweep using the runtime's approximately expert-sized random-read pattern. Candidate widths are 1, 2, 4, 8, 12, and 16. Once selected for each direct/buffered mode, worker count is held fixed across policy comparisons.

7.5 macOS page-cache protocol

macOS does not expose Linux O_DIRECT. Colibrì's compatibility path uses F_NOCACHE for unbuffered-style reads, but this does not retroactively evict pages already present in the unified buffer cache. Therefore:

1. Authoritative cold runs use untouched model ranges or follow a reboot.
2. Cold/direct-like cells run before any buffered access to the same shard ranges.
3. Every result records whether the model range was cold, previously touched, or intentionally warm.
4. The OS page cache is reported as an implicit tier, not silently counted as storage performance.
5. Readahead policies are evaluated separately for runtime-cache hits and OS-cache warming.

A run without a valid cache-state attestation is exploratory and excluded from confirmatory cold results.

7.6 Reproducibility and semantic invariants

Lossless configurations use deterministic greedy decoding, fixed prompts, fixed seeds where applicable, disabled conversation persistence, and explicit settings for every performance mechanism. The default lossless path must preserve:

• router-selected expert identities;
• expert weight precision and values;
• tokenization and sampling policy;
• output token IDs under deterministic decoding.

Validation includes:

• the runtime's existing self-tests;
• teacher-forcing/oracle checks where available;
• token-ID hashes for paired configurations;
• trace-on versus trace-off exactness;
• bit-identical or numerically validated output for any offline layout transformation.

Any configuration that changes routing, reduces experts, substitutes experts, or changes precision is labeled bounded-loss and excluded from the lossless primary analysis.

7.7 Instrumentation before optimization

The first research change is a disabled-by-default STALL_TRACE facility. Each record is keyed by run, request, step, position, layer, and expert or segment. Required fields include:

• evaluated positions $A^{\mathrm{pos}}_j$;
• committed tokens $A_j$;
• expert union $U_{j,l}$ and unique bytes;
• router top-$m$ probabilities or a compressed distribution sufficient for replay;
• source prediction scores and calibration outputs;
• residency tier at prediction and need time;
• issue, ready, need, compute-start, barrier-enter, and barrier-exit timestamps;
• bytes submitted and completed;
• I/O queue depth and worker occupancy;
• demand versus speculative priority;
• eviction and cancellation events;
• measured foreground-idle intervals;
• per-phase CPU time;
• MTP draft count and accepted length.

The trace must support reconstruction of $S$, timely required-byte recall, unused speculative bytes, union bytes per committed token, lead slack, and p95/p99 stall.

Instrumentation overhead is measured by paired trace-on/trace-off runs. If tracing materially changes queue timing, the runtime uses sampled or buffered tracing for confirmatory data, and the residual overhead is reported rather than hidden.

7.8 Workloads

The fixed workload suite contains:

1. General chat: multi-turn explanatory and analytical prompts;
2. Code: generation, debugging, and transformation prompts;
3. Structured output: grammar-constrained JSON or NDJSON;
4. Repeated personal-task trace: recurring domains representative of the intended single-user deployment;
5. Cross-domain holdout: prompts from domains excluded from the user-store training sessions.

Prompt files, order, expected maximum generation length, and stop conditions are versioned. Predictor training and map-store construction use session-based splits to prevent token-adjacent leakage. A recommended split is training sessions, validation sessions for thresholds, and chronologically later test sessions for confirmatory evaluation.

---

8. Experiments and falsifiers

E0. Validate the foreground-stall metric

Purpose. Establish whether $S$ provides information beyond conventional metrics.

Design. Run the unchanged control across fixed RAM budgets, cold/warm cache states, direct-like/buffered I/O, and I/O worker widths selected in the preliminary sweep. Enable one established mechanism at a time: asynchronous load overlap, router lookahead, cross-layer coupling, fixed profile pinning, and MTP.

For each run collect TPOT, tokens/s, raw bandwidth, application-cache hit rate, OS-cache evidence, prediction recall, timely required-byte recall, $S$, $C_{\mathrm{tok}}$, and $O_{\mathrm{tok}}$.

H0a. Across regimes, $S$ explains more within-machine variation in TPOT than application-cache hit rate, raw bandwidth, or prediction recall individually.

H0b. At least one pair of configurations with similar prediction recall exhibits materially different $S$ because of cache temperature, timeliness, or queue contention.

Falsifier. If $S$ does not improve causal interpretation or predictive association relative to simpler metrics after accounting for compute and other overhead, the framework's primary measurement claim is weakened. It may remain a reporting metric but should not be presented as the organizing objective.

E1. Calibrate late-need prediction and test user specificity

Predictor arms. At minimum:

• previous-token persistence or marginal heat;
• one-layer router lookahead;
• cross-layer coupling;
• FineMoE-style trajectory retrieval;
• semantic-plus-trajectory retrieval where embeddings are permitted;
• session-only history;
• longitudinal user-specific store;
• pooled-store control;
• calibrated fusion of established signals.

The fusion is an experimental vehicle, not a novelty claim.

Offline endpoints. Brier score, expected calibration error, reliability plots, top-$k$ selection recall, byte-weighted timely recall at simulated deadlines, and performance versus prefetch distance.

Online endpoints. $S$, unused speculative bytes, queue occupancy, and $\eta^{\mathrm{stall}}$ at equal byte and queue budgets.

H1a. The user-specific store produces better held-out calibration of late unavailability than the pooled store after sufficient genuine cross-session history accumulates.

H1b. Improved calibration converts to lower $S$ at an equal speculative-byte budget.

Falsifier. If user-specific retrieval does not outperform pooled or session-only history on chronologically held-out sessions, the longitudinal user-specific claim is removed. If offline calibration improves but $S$ does not, the result supports the central predictor–actuator distinction but not the store's practical value.

E2. Compare probability-only and stall-valued admission

Design. Hold the underlying prediction probabilities fixed. Compare:

1. highest predicted activation probability first;
2. highest probability per byte;
3. earliest deadline first;
4. stall-valued score $V(a)$;
5. oracle replay upper bound using realized future events, clearly labeled non-deployable.

All policies receive the same speculative-byte and queue-slot budgets. Demand priority is identical.

H2. Stall-valued admission yields lower $S$ and higher $\eta^{\mathrm{stall}}$ than probability-only admission, especially across mixed cache tiers where miss costs differ.

Falsifier. If probability-only or deadline-only policies match the stall-valued controller across all regimes, the additional cost modeling is not justified on this runtime.

E3. Evaluate stall-valued placement, splitting, and layout

Placement arms. LRU, frequency, hybrid recency-frequency, existing learned/pinned policy, probability-per-byte, and stall-value-per-byte.

Splitting. A MoEpic-inspired vertical split is implemented only after exactness and memory accounting are validated. It is treated as a prior-art baseline under the new objective.

Layout. Before rewriting a 372 GB container, a replay simulator uses recorded co-activation, measured read latency, APFS/RAID behavior, and merge rules. A physical rewrite proceeds only if held-out replay predicts meaningful cold-tier stall headroom beyond striping.

H3a. Stall-valued placement lowers $S$ relative to raw frequency at equal RAM, even when hit-rate improvement is small.

H3b. Splitting shifts the $S$-versus-RAM curve toward lower memory without changing output.

H3c. Layout benefit is regime-dependent and may be small on the striped array; any claim is limited to the measured burst and device conditions.

Falsifier. If stall-valued placement cannot outperform simpler policies, or if simulation shows no layout headroom, the corresponding mechanism is not advanced to composition.

E4. Measure native-MTP lead time and union amortization

Arms. MTP disabled; native MTP with draft depths 1–3 or runtime-supported values; grammar-constrained versus free-form output; prefetch disabled versus draft-informed admission.

Endpoints. Accepted tokens per forward, $B^{\mathrm{union}}_{\mathrm{tok}}$, additional lead time, timely required-byte recall, $S$, and total TPOT.

H4a. Grammar-constrained output produces lower union bytes per committed token and more useful lead time than free-form output.

H4b. Draft-informed prefetch reduces $S$ only in regimes where the additional lead time exceeds storage service latency and queue pressure remains bounded.

Falsifier. If union expansion or draft overhead offsets acceptance and timeliness gains, native MTP remains a compute feature rather than a storage optimization on this host.

E5. Lossless tail control

Arms. Unbounded or default speculative admission; fixed byte budget; queue-aware adaptive budget; stale cancellation; per-source caps; demand-priority scheduling.

Endpoints. Median, p95, p99, and maximum $s_j$; demand-read delay; speculative bytes; cancellation rate; throughput.

H5. Queue-aware budgets and cancellation reduce p99 stall without materially increasing median stall or changing output tokens.

Falsifier. If tail reductions merely shift waiting into compute or other overhead, or materially reduce median throughput, the policy is not retained.

A separate, optional bounded-loss appendix may compare BuddyMoE-style substitution, cache-conditional routing, or precision changes at matched quality deltas. Those results are not combined with the lossless headline.

E6. Composition and interference

Only mechanisms that pass their individual gates are composed. Compare the measured joint reduction in $S$ with the product or sum suggested by isolated effects, but do not present multiplication of prior-paper speedups as a forecast.

H6. At least one regime shows sub-additivity due to shared queue, cache, or compute constraints, and the trace identifies the absorbing bottleneck.

Falsifier. This experiment has no required positive speedup. A clean result that all mechanisms become irrelevant once the host is compute-bound is a valid outcome.

---

9. Statistical analysis and reporting

9.1 Primary and secondary endpoints

The pre-registered primary endpoint for lossless experiments is:

$$ \widehat{S}_N = \frac{\mathrm{total\ foreground\ expert\ stall}}{\mathrm{committed\ output\ tokens}}. $$

Secondary endpoints are TPOT, tokens/s, p95/p99 normalized step stall, timely required-byte recall, union bytes per committed token, unused speculative bytes, queue occupancy, cache-tier hit rates, and energy or power where reliable measurements are available.

9.2 Repetition and interleaving

Exploratory screening may use three repetitions. Confirmatory comparisons use at least five paired, interleaved repetitions per workload and regime, subject to the long runtime of the 744B checkpoint. Configuration order is randomized or balanced to reduce thermal and cache-order bias.

Each paired comparison uses the same prompt order and deterministic output. If a run diverges in output under a purportedly lossless policy, it is treated as a correctness failure, not a noisy performance sample.

9.3 Estimation

Report:

• paired absolute and relative differences;
• medians and means where both are informative;
• bootstrap 95% confidence intervals over paired runs or requests;
• full per-run values, not only aggregates;
• effect sizes alongside any significance tests.

Where multiple confirmatory hypotheses are tested, control family-wise error or false-discovery rate, but do not use thresholded $p$-values as a substitute for effect magnitude and reproducibility.

9.4 Regression for E0

To test metric validity, fit within-machine models predicting TPOT or request latency from compute time, other overhead, and candidate storage metrics. Compare out-of-sample error or explained variance for:

• hit rate;
• raw bandwidth;
• prediction recall;
• timely required-byte recall;
• $S$.

Because $T=C+W+O$ is partly definitional, the purpose is not to “discover” that measured wait contributes to time. The meaningful test is whether $S$ provides a stable, mechanism-independent explanation of why policies with similar recall or hit rate differ and whether it transfers across RAM and cache regimes.

9.5 Negative results

A fired falsifier is reported as a result. In particular:

• no user-specific advantage means the longitudinal-store claim is removed;
• no stall-value advantage means the simpler controller is preferred;
• no layout headroom on RAID means the rewrite is skipped;
• compute domination means storage mechanisms are reported as regime-limited;
• high trace overhead means the instrumentation design is revised before claims are made.

---

10. Threats to validity and limitations

10.1 One checkpoint

Routing locality and cacheability vary by model. GLM-5.2's top-8-of-256 routing may behave differently from Mixtral, Qwen, DeepSeek, Phi-MoE, or models with fewer experts. The paper's framework is general, but empirical conclusions are checkpoint-specific unless replicated.

10.2 One unusual machine

A six-channel-memory Intel workstation with a four-drive RAID-0 is not representative of most laptops or servers. It is valuable because it spans storage-bound and potentially compute-bound regimes, but external validity requires replication on single-NVMe machines, Linux O_DIRECT, and other memory sizes.

10.3 Page-cache opacity

macOS's unified buffer cache is only partially controllable. F_NOCACHE prevents new caching behavior in relevant paths but does not guarantee eviction of already warm pages. The protocol reduces but cannot eliminate OS-level uncertainty.

10.4 Rapidly changing runtime

Colibrì evolves quickly. Pinning a commit improves reproducibility but may omit later fixes. Results therefore apply to the pinned revision and must distinguish upstream changes from research changes.

10.5 Measurement perturbation

Fine-grained tracing can alter scheduling and I/O timing. The instrumentation must be benchmarked, buffered, sampled where necessary, and reported with overhead.

10.6 User-store stationarity and privacy

A single user's workload may not be stable enough to produce a useful longitudinal profile. Conversely, a highly predictive store may reveal sensitive task patterns. Local retention is preferable, but locality alone does not eliminate privacy risk.

10.7 Objective scope

Foreground expert stall does not capture all user experience. Time to first token, token pacing, model quality, energy, heat, noise, reliability, and system responsiveness also matter. $S$ is intended as the causal storage objective inside a broader evaluation, not as a universal measure of inference quality.

10.8 No result before execution

The present manuscript is a proposal. It contains no measured Mac Pro speedup and makes no composite performance forecast. Prior-paper speedups come from different models, buses, accelerators, and baselines and must not be multiplied.

---

11. Conclusion

Storage-tier MoE inference is often described in terms of bandwidth, cache hits, or prediction accuracy. Those quantities are insufficient because the decode loop cares about a stricter event: whether the required expert data is ready before useful computation runs out.

This paper proposes expected foreground stall per committed token as the common measurement for that event. The metric counts only critical-path waiting that remains after asynchronous I/O, prefetch, deduplication, and compute overlap; it charges rejected speculative positions through the numerator while crediting only committed tokens in the denominator. It exposes why a predictor can improve recall without improving latency and why the same policy can help on a cold machine, become irrelevant on a warm one, and become harmful when it congests the queue.

The associated control principle is equally simple: predictions should not be acted on because they are probable. They should be acted on when they are expected to prevent enough residual stall to justify their bytes, deadline, queue slot, and pollution risk. This reframes established predictors, caches, layouts, and speculative mechanisms as inputs to a common value function.

The novelty claim is intentionally narrow after a contemporary prior-art audit. The paper does not claim expert offloading, local MoE inference, multi-signal prediction, distributional retrieval, speculative prefetching, utility scheduling, splitting, co-activation placement, or uncertainty-gated substitution. It claims a measurable critical-path objective, a trace that separates forecasting from delivery, a stall-valued control rule, and a conditional longitudinal user-specific experiment.

The proposed Mac Pro study is designed to produce a useful answer even if no optimization wins. If foreground stall dominates, the trace identifies which interventions remove it. If the page cache, RAID, memory system, or CPU moves the bottleneck elsewhere, the metric shows where storage optimization stops mattering. Either outcome replaces a benchmark anecdote with a defensible systems result.

---

Appendix A. Minimum STALL_TRACE schema

A portable JSONL record should include at least:

{
  "run_id": "...",
  "request_id": 0,
  "step": 0,
  "layer": 0,
  "evaluated_positions": 1,
  "committed_tokens": 1,
  "expert_id": 0,
  "segment_id": 0,
  "required": true,
  "router_probability": 0.0,
  "prediction_source": "demand|lookahead|couple|trajectory|session|user|mtp",
  "prediction_score": 0.0,
  "calibrated_late_need_probability": 0.0,
  "tier_at_prediction": "ram|os_cache_unknown|nvme",
  "tier_served": "ram|os_cache|nvme",
  "bytes": 0,
  "issue_ns": 0,
  "ready_ns": 0,
  "need_ns": 0,
  "barrier_enter_ns": 0,
  "barrier_exit_ns": 0,
  "foreground_stall_ns": 0,
  "queue_depth": 0,
  "worker_occupancy": 0,
  "priority": "demand|speculative",
  "used_before_eviction": true,
  "cancelled": false
}

A separate step record should capture draft acceptance, output-token IDs or hashes, compute time, other overhead, and total wall time. Timestamps must use a monotonic clock.

---

Appendix B. Pre-registered hypothesis ledger

---

References

Reference status is stated explicitly. URLs were checked against primary publication pages, arXiv records, official proceedings, or the cited software repositories as of July 26, 2026.

1. Eliseev, A., and Mazur, D. “Fast Inference of Mixture-of-Experts Language Models with Offloading.” arXiv:2312.17238, 2023. [Preprint/technical report] https://arxiv.org/abs/2312.17238

2. Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. “MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache.” arXiv:2401.14361, revised 2025. [Preprint] https://arxiv.org/abs/2401.14361

3. Yu, H., Cui, X., Zhang, H., Wang, H., and Wang, H. “Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading.” Proceedings of EuroSys 2026, pp. 176–191. DOI: 10.1145/3767295.3769319. [Peer-reviewed conference] https://arxiv.org/abs/2502.05370

4. Song, X., Zhong, Z., Chen, R., and Chen, H. “ProMoE: Fast MoE-based LLM Serving using Proactive Caching.” arXiv:2410.22134, revised 2025. [Preprint] https://arxiv.org/abs/2410.22134

5. Kim, B., Lee, J., Han, D., Yoo, H.-J., and Kim, S. “FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices.” arXiv:2601.17063, 2026. [Preprint] https://arxiv.org/abs/2601.17063

6. Zhu, Z., Li, G., Wang, P., Mo, Z., Pei, M., Song, Z., Liang, X., and Cheng, J. “DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs.” arXiv:2602.03495, 2026. [Preprint] https://arxiv.org/abs/2602.03495

7. Zhao, Y., Bunescu, R., Louri, A., Karanth, A., and Wang, K. “A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference.” arXiv:2606.15453, 2026. [Preprint] https://arxiv.org/abs/2606.15453

8. He, X., Zhang, S., Tang, K., et al. “ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling.” DAC 2026. DOI: 10.1145/3770743.3804292. [Accepted/peer-reviewed conference] https://arxiv.org/abs/2410.17954

9. Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., and Yang, M. “Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference.” ISCA 2024, pp. 1018–1031. [Peer-reviewed conference] https://arxiv.org/abs/2308.12066

10. Kamahori, K., Tang, T., Gu, Y., Zhu, K., and Kasikci, B. “Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models.” ICLR 2025. [Peer-reviewed conference] https://proceedings.iclr.cc/paper_files/paper/2025/hash/8cd1ce03ea58b3d7dfd809e4d42f08ea-Abstract-Conference.html

11. Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., and Xu, M. “EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices.” arXiv:2308.14352, revised 2025. [Preprint] https://arxiv.org/abs/2308.14352

12. Xue, Z., Song, Y., Mi, Z., Zheng, X., Xia, Y., and Chen, H. “PowerInfer-2: Fast Large Language Model Inference on a Smartphone.” arXiv:2406.06282, 2024. [Preprint] https://arxiv.org/abs/2406.06282

13. Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.-A., Li, C., and Guo, M. “HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference.” arXiv:2411.01433, 2024. [Preprint] https://arxiv.org/abs/2411.01433

14. Yan, J., Liu, J., Xu, H., and Huang, L. “Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism.” arXiv:2509.08342, 2025. [Preprint; MoEpic] https://arxiv.org/abs/2509.08342

15. Huang, Z., Zhu, L., Zhan, Z., et al. “MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE.” NeurIPS 2025 Spotlight. [Peer-reviewed conference] https://arxiv.org/abs/2505.19645

16. Wang, Z., Zhang, Z., Zhou, Y., et al. “Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding.” arXiv:2508.21706, 2025. [Preprint; SpecMoEOff] https://arxiv.org/abs/2508.21706

17. Chen, L., Wen, Z., Wu, T., Zhang, X., and Wu, C. “SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference.” arXiv:2510.10302, revised 2025. [Preprint] https://arxiv.org/abs/2510.10302

18. Wang, W., Liu, J., Hou, X., Xia, X., Tang, P., Zhang, M., Li, C., and Guo, M. “MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts.” arXiv:2511.14102, 2025. [Preprint] https://arxiv.org/abs/2511.14102

19. Li, S., Lin, J., Ge, D., and Ye, Y. “MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios.” arXiv:2603.09983, 2026. [Preprint] https://arxiv.org/abs/2603.09983

20. Skliar, A., van Rozendaal, T., Lepert, R., et al. “Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference.” Transactions on Machine Learning Research, June 2025. [Peer-reviewed journal] https://arxiv.org/abs/2412.00099

21. Wang, Y., Yang, L., Yu, S., et al. “BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference.” arXiv:2511.10054, 2025. [Preprint; conference status not relied upon here] https://arxiv.org/abs/2511.10054

22. Madan, V., Singhania, P., Bhatele, A., Goldstein, T., and Panda, A. “Speculating Experts Accelerates Inference for Mixture-of-Experts.” arXiv:2603.19289, 2026. [Preprint] https://arxiv.org/abs/2603.19289

23. Shi, X., Sun, Y., Du, J., Chen, Z., and Lu, Y. “Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference.” arXiv:2607.05116, 2026. [Preprint; CAP] https://arxiv.org/abs/2607.05116

24. Liang, J., Wang, S., Tian, M., Li, Y., Tang, D., and Wei, Z. “Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models.” arXiv:2505.16056, revised 2026. [Preprint] https://arxiv.org/abs/2505.16056

25. Sheng, Y., Zheng, L., Yuan, B., et al. “FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU.” ICML 2023. [Peer-reviewed conference] https://arxiv.org/abs/2303.06865

26. JustVugg. “colibrì: Tiny Engine, Immense Model.” Pinned revision 44e489b196c9b7876b3d37a0570ebf1c6f90f54c, accessed July 2026. [Open-source software artifact] https://github.com/JustVugg/colibri

27. JustVugg/colibrì contributors. “Cross-layer routing coupling: trace dump, pair-table builder, and an opt-in coupled prefetch source.” Pull request 176, merged July 14, 2026. [Software experiment/artifact] https://github.com/JustVugg/colibri/pull/176

28. doramirdor. “mbolt: Profile-guided co-activation expert layout.” 2026. [Open-source software artifact; not treated as peer-reviewed evidence] https://github.com/doramirdor/mbolt