Anchored Plug-in MoE for Frozen Language Models

What this part does

Anchored Plug-in MoE extends a capable dense language model without silently changing it. The original model remains an immutable anchor, while specialist MLPs, routers, calibration data, and lifecycle metadata are carried in independently managed packages.

The framework is a research system for comparing modular composition strategies. It is not one fixed Mixture-of-Experts recipe, and a validated standalone primitive is not automatically available through the complete model lifecycle.

Where it fits

The MLP-expert path sits around a Hugging Face causal language model:

1. a backend locates selected MLP layers;
2. each selected MLP is wrapped while preserving its original module as the anchor;
3. installed package metadata and expert payloads are validated;
4. an explicit synchronization step attaches the supported runtime;
5. routing chooses the anchor or compatible specialists for each token;
6. export records a reproducible composition without duplicating the anchor in every package.

In this MLP-expert path, attention, embeddings, normalization, and the language-model head remain part of the frozen shared backbone.

The Needle branches path instead shares a frozen prefix and specializes complete last-layer suffixes, selecting one response branch per request. Its browser deployment uses a generative router with recent conversation context and downloads one base plus router/specialist patches. Both paths preserve an explicit frozen fallback.

Implemented approaches

These labels describe lifecycle integration, not research value or test quality. The Python feature tracker separately records whether each underlying implementation has passed its own acceptance criteria.

How it works

For every converted layer, the dense MLP becomes candidate zero. Specialist scores are calibrated against that fixed baseline rather than against an implicitly learned “default” expert.

In the conservative top-1 path:

1. the anchor begins with score zero;
2. each installed specialist produces a calibrated score;
3. stable anchor-first tie-breaking chooses the largest score;
4. a specialist runs only when it has strictly positive winning evidence;
5. unavailable or overflowing work falls back to the anchor;
6. the selected output is reconstructed in the original token order.

Installing, unloading, upgrading, or rolling back a package changes registry state. Model execution changes only after the caller explicitly synchronizes that state, which keeps lifecycle operations auditable.

Mathematical intuition

For a specialist winner k, the routed layer returns Eₖ(h). For an anchor winner, it returns A(h) with coefficient one. This is why exact fallback is a runtime invariant rather than an approximate regularization goal.

Minimal usage

The smallest offline conversion uses an already loaded Qwen3 model:

from anchored_moe import AnchoredMoEConfig
from anchored_moe.backends import Qwen3Backend

report = Qwen3Backend().convert(
    model,
    AnchoredMoEConfig(converted_layers="upper_half"),
)

The conversion starts in forced-anchor mode. It should preserve every hidden state and output logit before any specialist is explicitly attached.

Verified implementation details

with torch.no_grad():
    dense = model(input_ids=input_ids, output_hidden_states=True)

Qwen3Backend().convert(
    model,
    AnchoredMoEConfig(converted_layers="upper_half"),
)

with torch.no_grad():
    converted = model(input_ids=input_ids, output_hidden_states=True)

assert_tensors_equivalent(converted.logits, dense.logits, name="logits")

model.install(package)
model.synchronize_registry()

routed = model(input_ids=input_ids).logits

model.unload(package.identity)
model.synchronize_registry()
anchor = model(input_ids=input_ids).logits

Current limitations

• Qwen3 is the first implemented backend; the backend contract is broader than the currently validated model-family set.
• Package-driven top-2, adaptive, hierarchical, shortlisted, and delta-expert attachment are not model-level configuration paths.
• Composition checkpoints reference a pinned anchor and do not yet support remote transfer, optional anchor bundling, or inactive rollback history.
• Capacity estimation, fallback beyond an already selected pair, and distributed production validation remain future work.
• Several advanced strategies target experiments, training, or prompt prefill and should not be described as autoregressive serving paths.

References

The project-specific system design and its departures from prior work are defined in the architecture blueprint.

Related pages

• Return to the documentation index.
• Inspect the feature implementation tracker for the acceptance status of individual Python features.
• Read the repository overview for development setup and the full executable example inventory.