Configure Anchored MoE Models and Frozen Backbones

What this part does

Configuration records the identity of the immutable anchor, chooses a model-family backend, and selects which decoder MLPs become routed wrappers. The model façade then preserves the anchor’s native forward and generation behavior while exposing an explicit package lifecycle.

The integrated model-level path is intentionally conservative: conversion starts in exact forced-anchor mode, and packaged anchored top-1 routing becomes active only after an explicit synchronization call.

Where it fits

AnchoredMoEConfig and AnchoredMoEForCausalLM connect the framework’s otherwise independent parts:

1. configuration pins the anchor and resolves converted layers;
2. a backend validates the model family and locates its MLPs;
3. conversion freezes the complete anchor and wraps only the selected MLPs;
4. the façade delegates forward() and generate() to that converted anchor;
5. registry operations update package state without changing execution;
6. synchronization materializes the supported anchored top-1 runtime;
7. local composition export records the synchronized result without copying the anchor weights.

The configuration does not contain live modules, tensors, or package payloads. It is a serializable contract that can be compared with the model and package metadata before runtime mutation occurs.

Implemented approaches

The executable configuration boundary

The resolved foundation configuration contains five independent blocks:

These are strict executable defaults, not a catalogue of every implemented strategy. Passing anchored_top1, top-2, adaptive, or another advanced value through strategy_config currently raises an error instead of silently falling back to the anchor.

Packaged anchored top-1 is nevertheless integrated: it is activated from validated package artifacts by synchronize_registry(). This separate entry point keeps the serialized foundation configuration honest about what it can construct by itself.

How it works

Choose the anchor and layers

AnchoredMoEConfig can record:

• the base-model repository or local path and exact revision;
• an optional SHA-256 digest of the resolved base configuration;
• tokenizer identity and revision;
• the backend name, currently qwen3;
• either upper_half or an explicit list of zero-based decoder layers;
• the five independent strategy blocks.

Explicit layer indices are sorted, deduplicated, and checked against the model depth. Invalid or empty selections fail before conversion. When a live configuration digest is supplied, the façade verifies it before any MLP is wrapped.

Load or wrap the model

Use from_anchor_model() when a supported Hugging Face causal language model is already in memory. Conversion happens in place, and the façade derives a configuration digest when a complete AnchoredMoEConfig is not supplied.

Use from_pretrained() for either:

• a base checkpoint, which is loaded through AutoModelForCausalLM and then converted; or
• a local Mutations composition checkpoint, whose immutable saved configuration, packages, and anchor reference are validated before the anchored top-1 runtime is restored.

In both cases, the native model-family implementation still owns forward and generation behavior. The façade rejects replacement of the frozen input and output embeddings.

Apply registry changes deliberately

install(), unload(), upgrade(), and rollback() change only the in-memory registry. The routed wrappers continue to represent the last successfully synchronized registry until the caller invokes synchronize_registry().

Synchronization validates every candidate artifact and constructs the replacement runtime before changing a wrapper. Synchronizing an empty registry returns every converted layer to exact forced-anchor execution.

This separation means that inspecting or preparing registry state cannot silently alter model outputs.

Mathematical intuition

Layer selection changes where routing can happen, not the initial computation. Immediately after conversion, each selected wrapper still evaluates its original dense MLP once.

Minimal usage

Load and convert a pinned checkpoint through the model façade:

from anchored_moe import AnchoredMoEForCausalLM

model = AnchoredMoEForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    revision="<immutable-commit>",
    converted_layers="upper_half",
)

The returned model is frozen and starts in forced-anchor mode. Loading a remote checkpoint requires network access and any credentials required by its repository; the verified examples below instead use tiny local Qwen3 models.

Verified implementation details

config = AnchoredMoEConfig(
    base_model_name_or_path="Qwen/Qwen3-4B",
    base_model_revision="pinned-revision",
    converted_layers=[3, 1],
)

restored = AnchoredMoEConfig.from_dict(config.to_dict())

assert restored.converted_layers == [1, 3]
assert restored.resolve_converted_layers(4) == (1, 3)
assert restored.resolved_strategies.selection["policy"] == "forced_anchor"
assert restored.base_model_revision == "pinned-revision"

package = _export_package(root / "package", model)
model.install(package)
installed = model.synchronize_registry()

with torch.no_grad():
    routed_logits = model(input_ids=input_ids).logits

model.unload(package.identity)
model.synchronize_registry()

with torch.no_grad():
    removed_logits = model(input_ids=input_ids).logits

assert_tensors_equivalent(
    removed_logits,
    dense_logits,
    name="unloaded anchor logits",
    atol=0.0,
    rtol=0.0,
)

composition_checkpoint = root / "composition"
model.save_pretrained(composition_checkpoint)
reloaded = AnchoredMoEForCausalLM.from_pretrained(
    composition_checkpoint,
    local_files_only=True,
)

with torch.no_grad():
    reloaded_logits = reloaded(input_ids=input_ids).logits

assert_tensors_equivalent(
    reloaded_logits,
    routed_logits,
    name="reloaded composition logits",
    atol=0.0,
    rtol=0.0,
)

Current limitations

• Qwen3 is the only built-in and validated model-family backend.
• strategy_config currently accepts only the forced-anchor foundation values; it cannot declaratively attach anchored top-1 or advanced modes.
• Package synchronization supports full-MLP anchored top-1, not package-driven top-2, adaptive, delta, hierarchical, or dynamic-k execution.
• Composition checkpoints reference an external pinned anchor and do not bundle its weights.
• Composition export and reload are local-only; inactive registry history and rollback records are not included.
• Remote production checkpoints, optimized kernels, multiple devices, and distributed execution require validation in their own declared environments.
• Explicit RoutedMLP experimental modes are lower-level interfaces and must not be presented as automatic model lifecycle behavior.

References

Configuration and modeling are project-specific framework infrastructure, not paper-derived algorithms.

• The Hugging Face configuration documentation defines the PretrainedConfig serialization contract extended by AnchoredMoEConfig.
• The Hugging Face model documentation defines the PreTrainedModel, checkpoint-loading, and generation conventions preserved by the façade.
• The Qwen3 model documentation describes the first supported model family.
• The repository’s architecture blueprint defines anchor immutability, exact fallback, deterministic routing, and auditable installation.

Related pages

• Start with getting started for the smallest offline conversion and exact-equivalence check.
• Read the overview for the full package lifecycle and the integration-label definitions.
• Inspect the feature implementation tracker for acceptance evidence and remaining work for each referenced feature ID.