MoE and LoRA Routing Examples for Frozen LLMs

What this page does

These guides connect the framework’s individual strategies into inspectable workflows. They begin with the conservative dense-equivalence baseline, add the complete packaged anchored top-1 lifecycle, and then introduce explicitly composed weighted top-2 execution, adaptive route selection, always-on-anchor deltas, package-local two-expert training, and hierarchical package routing. Prompt-level shortlisting then shows how an external retrieval stage can bound the columns evaluated by token-level routing. Product-key retrieval closes the page by showing how two smaller key tables can retrieve exact top-(k) micro-expert identifiers from a square package-local index.

The comparison table keeps integration boundaries visible from the start. Exact conversion, packaged top-1, weighted top-2, adaptive selection, routed LoRA deltas, local mini-MoE training, package-first hierarchy, prompt-shortlisted top-1, and compatibility-constrained top-2 are published below. The final two guides vary the number of selected specialists per token under a strict compute cap, then retrieve fine-grained expert identifiers while keeping the unimplemented execution boundaries explicit.

Compare the workflows

For the separate Needle path, see Needle branching and browser deployment below. Its examples cover shared-prefix training, cached decoding, native export, and generative routing through the official browser runtime.

Integration labels are not maturity scores. Integrated means the behavior is available through its intended lifecycle. Composable means the tested parts require explicit assembly. Research primitive identifies a validated experimental building block rather than an automatic model-level path.

Exact anchor conversion

The problem it solves

Before adding specialists, conversion must prove that wrapping selected dense MLPs does not change the model. This workflow creates a tiny local Qwen3 model, records its outputs, converts the upper-half MLPs to routed wrappers, and compares the result with zero tolerance.

Workflow contract

• Expert representation: none; the original dense MLP remains candidate zero.
• Routing and selection: every converted wrapper starts in forced_anchor, so no router score is evaluated.
• Combination and capacity: the anchor output is returned directly; specialist mixing and overflow handling are inactive.
• Training and calibration: neither is required.
• Active work: one anchor MLP per token at every converted layer.
• Packaging and serialization: no expert package is created. The example validates conversion before package lifecycle work begins.
• Expected result: every hidden state and final logit is exactly equal on the deterministic CPU fixture.

How it runs

Tiny local Qwen3 anchor
  → record dense hidden states and logits
  → wrap upper-half MLPs in forced-anchor mode
  → execute the same input
  → require exact tensor equality

The wrapper retains the same original MLP object as anchor_mlp. The integration test also fingerprints its parameters before conversion and requires that fingerprint to remain unchanged.

Expected invariant

Run the verified example

From the repository root, run:

python examples/verify_dense_equivalence.py

The workflow is offline: it constructs a deterministic random model and downloads neither weights nor tokenizer data. A successful run reports layers (2, 3), exact logits, and a maximum absolute difference of 0.0.

with torch.no_grad():
    dense = model(input_ids=input_ids, output_hidden_states=True)

report = Qwen3Backend().convert(
    model,
    AnchoredMoEConfig(converted_layers="upper_half"),
)

with torch.no_grad():
    converted = model(input_ids=input_ids, output_hidden_states=True)

assert_tensors_equivalent(converted.logits, dense.logits, name="logits")

The Getting started guide contains installation instructions and the complete source listing. This page keeps the workflow comparison concise instead of maintaining a second full copy.

Current limitations

• The fixture proves framework equivalence, not language quality.
• Qwen3 is the only currently validated backend.
• Exact equality is established for the deterministic CPU path. Optimized kernels, reduced precision, and other runtimes require separate validation.
• No specialist, router, package, calibration artifact, or composition checkpoint exists yet; those belong to subsequent workflows.

References and related pages

This baseline is project-specific verification infrastructure rather than a paper-derived algorithm.

• Getting started provides the complete local setup.
• Backends explains Qwen3 discovery, replacement, and moved-not-copied anchor identity.
• Evaluation defines exact tensor comparison and its runtime limits.
• Configuration and modeling describes upper_half, forced_anchor, and the model façade.

Packaged anchored top-1

The problem it solves

Exact conversion establishes a safe anchor but does not yet specialize the model. This workflow trains a full-MLP code expert without changing the anchor, fits and calibrates an independent router head, derives an immutable routed package, explicitly synchronizes that package into the model, and saves a composition that reloads with the same routed behavior.

This is the primary integrated workflow because every stage is available through package APIs. The offline verification scripts exercise the stages independently with deterministic temporary fixtures; they do not pass a persistent artifact from one script to the next.

Workflow contract

• Expert representation: one full copied-MLP expert, code:0, for each converted layer.
• Training: forced-path training sends every training token through the new expert while mechanically freezing the anchor and shared model.
• Router: one append-only affine head per (layer, expert) binding, fitted with ridge regression and calibrated with held-out temperature and threshold rows.
• Selection: anchored hard top-1 compares the calibrated specialist score with the anchor’s permanent score of zero.
• Combination and capacity: exactly one complete path runs per token. A positive winning specialist runs alone; otherwise the anchor runs alone.
• Packaging: expert tensors, router heads, calibration provenance, hashes, and explicit layer bindings form an immutable package.
• Lifecycle: install, unload, and reinsert update the registry only. synchronize_registry() is the explicit boundary that changes model execution.
• Serialization: the composition bundles active package bytes and references the external anchor checkpoint instead of duplicating its weights.
• Expected result: reload restores the same registry identity and exactly the same routed logits on the deterministic CPU fixture.

How it runs

Frozen Qwen3 anchor
  → train full-MLP experts through forced paths
  → export an immutable expert-only package
  → fit and calibrate independent affine router heads
  → derive an immutable routed package
  → install, then explicitly synchronize anchored top-1
  → save and reload the composition
  → require exact routed-logit and registry restoration

Installation and synchronization are deliberately separate. This prevents a registry mutation from silently changing live model behavior and keeps router recalibration an explicit operation.

Routing rule

Minimal composition

Once a compatible calibrated package exists, the public lifecycle remains short:

model = AnchoredMoEForCausalLM.from_pretrained(anchor_path, converted_layers=[2, 3])
package = ExpertPackage.from_directory(package_path)
model.install(package)
model.synchronize_registry()
model.save_pretrained(composition_path)
reloaded = AnchoredMoEForCausalLM.from_pretrained(composition_path)

The model will reject incompatible base configuration, converted-layer, feature-space, payload-binding, or package-integrity metadata rather than attaching the package approximately.

Train and export the expert

Run the deterministic training stage:

python examples/train_expert_package.py

It performs one offline training step, proves that both layer experts changed, proves that both anchor MLPs remained exact, and validates the exported package files.

result = trainer.train()

package = export_full_mlp_expert_package(
    root / "code-expert",
    bindings=[
        ExpertPackageExportBinding(
            binding.layer_index,
            binding.expert_id,
            binding.expert,
        )
        for binding in training_bindings
    ],
    config=ExpertPackageExportConfig(
        specialization_name="code",
        specialization_version="1.0.0",
        base_model_repository="local-random-qwen3",
        base_model_revision="synthetic-seed-13",
        base_config_sha256=sha256_json(model.config.to_dict()),
        base_weight_sha256={"in-memory-anchor-state": base_weight_sha256},
        tokenizer_repository="local-random-qwen3",
        tokenizer_revision="synthetic-seed-13",
        base_model_dtype="float32",
        runtime_versions={
            "accelerate": _package_version("accelerate"),
            "kernels": "torch-default-cpu",
            "peft": _package_version("peft"),
            "python": platform.python_version(),
            "safetensors": _package_version("safetensors"),
            "torch": torch.__version__,
            "transformers": _package_version("transformers"),
        },
        training_dataset={
            "dataset_id": "synthetic-local-code",
            "revision": "1",
            "split": "train",
            "filters": [],
        },
        router_feature_specification={
            "feature_type": "identity",
            "input": "pre_mlp_hidden_state",
            "output_size": model.config.hidden_size,
            "version": 1,
        },
        expert_parameterization_config={
            "initialization": "anchor_copy",
            "training_strategy": "forced_path",
            "optimizer": "adamw_torch",
            "max_steps": result.global_step,
        },
    ),
)

assert package.manifest.router_architecture == "none"
package.validate_live_files()

Fit, calibrate, and derive the routed package

Run the deterministic calibration stage:

python examples/fit_router.py

The example fits independent ridge heads, calibrates temperature and activation thresholds on disjoint held-out row IDs, derives a new routed package version, and proves that both anchor and expert routes occur.

fitted = fit_independent_ridge(
    fitting_bank,
    fitting_labels,
    regularization=1e-6,
)
calibrated, report = calibrate_temperature_and_thresholds(
    fitted,
    calibration_bank,
    calibration_labels,
    temperature_row_ids=[10, 11, 12, 13],
    threshold_row_ids=[14, 15, 16, 17, 18, 19],
    false_activation_rate=1.0 / 3.0,
)
derived = derive_calibrated_router_package(
    source,
    workspace / "code-routed",
    calibrated_router=calibrated,
    calibration_report=report,
    fitting_feature_bank=fitting_bank,
    fitting_utility_labels=fitting_labels,
    calibration_feature_bank=calibration_bank,
    calibration_utility_labels=calibration_labels,
    config=CalibratedRouterPackageConfig(
        specialization_version="1.0.0+router.1",
        calibration_dataset_version="synthetic-router-v1",
        ridge_regularization=1e-6,
        runtime_versions=runtime_versions,
    ),
)

Install, synchronize, save, and reload

Run the deterministic runtime stage:

python examples/verify_package_runtime.py

The fixture installs a safely bound package, explicitly materializes anchored top-1, verifies removal to the exact dense anchor, reinserts the package, and reloads an immutable composition.

model.install(package)
installed = model.synchronize_registry()
with torch.no_grad():
    routed_logits = model(input_ids=input_ids).logits

model.save_pretrained(composition_checkpoint)
reloaded = AnchoredMoEForCausalLM.from_pretrained(
    composition_checkpoint,
    local_files_only=True,
)
with torch.no_grad():
    reloaded_logits = reloaded(input_ids=input_ids).logits

assert_tensors_equivalent(
    reloaded_logits,
    routed_logits,
    name="reloaded composition logits",
    atol=0.0,
    rtol=0.0,
)
assert reloaded.synchronized_registry_sha256 == installed.registry_sha256

Current limitations

• The examples use tiny deterministic Qwen3 fixtures and synthetic data. They validate framework behavior, not code-specialization quality.
• Qwen3 is the only validated model-family backend.
• Full-MLP experts duplicate each converted MLP. Package-driven LoRA materialization remains separate future work.
• The composition references a compatible external anchor checkpoint; it does not bundle the anchor weights.
• The exact reload claim applies to the validated CPU path. Reduced precision, optimized kernels, multi-device execution, and distributed execution need separate validation.
• Installing or removing a package does not change live execution until synchronization is requested. Router recalibration is never automatic.
• The offline scripts validate consecutive lifecycle stages independently rather than acting as a persistent multi-process pipeline.

References and related pages

Anchored top-1 and explicit registry synchronization are project-specific framework mechanisms. Ridge fitting follows the classical Hoerl–Kennard method; temperature scaling is motivated by the following calibration work, then adapted to independent binary utility heads and explicit activation thresholds.

• Training explains forced-path isolation and anchor preservation.
• Calibration defines feature banks, ridge fitting, temperature scaling, thresholds, and immutable routed-package derivation.
• Registry covers package integrity, explicit lifecycle operations, and composition checkpoints.
• Inference describes sparse anchored top-1 dispatch and deterministic output reconstruction.
• Policies explains fixed-zero anchor semantics and stable tie-breaking.

Weighted top-2 specialists

The problem it solves

Anchored top-1 cannot combine two specialists whose transformations may be useful together. This workflow ranks the fixed-zero anchor beside two affine specialist scores, approves a pair only when both specialist scores are strictly positive, executes the selected experts sparsely, and mixes their activations with score-softmax weights.

The selection, combination, capacity, and dispatch parts are validated through public APIs, but they require explicit assembly. Package synchronization does not attach this strategy to RoutedMLP, and AnchoredMoEConfig does not accept top-2 as a model-level policy.

Workflow contract

• Expert representation: two compatible complete experts. The verified dispatch example uses frozen full-MLP experts derived from the same anchor.
• Router: append-only affine heads score identity features in stable expert order.
• Selection: anchor-veto top-2 ranks the permanent zero-score anchor with every specialist. The specialist pair runs only when both winning specialist scores are strictly positive.
• Combination: score-softmax converts the ordered specialist scores into non-negative weights that sum to one.
• Capacity: explicit per-token availability is applied only after a pair is approved. Both available experts run as top-2; one survivor runs alone with weight [1, 0]; no survivors return to the anchor.
• Training and calibration: this example consumes already constructed experts and affine scores. It does not train experts, fit heads, or establish that the selected mixture improves quality.
• Active work: an anchor token executes one MLP, a capacity-degraded token executes one specialist MLP, and an effective top-2 token executes two specialist MLPs.
• Packaging and serialization: standalone objects are assembled directly. Package-driven top-2 attachment and composition reload are not implemented.
• Expected result: no token is dropped, inactive experts do not execute, original token order is restored, and every effective pair has finite sum-to-one weights.

How it runs

Hidden states
  → append-only affine scores
  → rank [anchor = 0, specialist scores]
  → anchor veto unless two specialists are positive
  → apply explicit expert availability
  → sparsely execute zero, one, or two specialists
  → score-softmax effective pairs
  → reconstruct the original token order

Selection and capacity answer different questions. Anchor veto decides whether the evidence supports a specialist pair. Capacity handling then decides how much of that already-approved pair is currently available; it never searches for a replacement expert outside the selected pair.

Selection and mixture rule

Minimal explicit composition

The standalone dispatcher makes every strategy boundary visible:

result = AnchoredTop2Dispatcher(
    combination_policy=ScoreSoftmaxCombination(mix_temperature=1.0),
).dispatch(
    hidden_states,
    expert_scores=scores,
    expert_ids=("code:0", "math:0"),
    anchor_mlp=anchor,
    experts=(code_expert, math_expert),
    expert_availability=availability,
)

result.selected_routes preserves the original anchor-veto decision. result.capacity records the effective anchor, top-1, or top-2 execution after availability is applied, and result.mixture_weights keeps pair-shaped weights aligned with every input token.

Verify selection and score-softmax combination

Run the deterministic policy-level example:

python examples/verify_anchor_veto_top2.py

Two of its four tokens have two positive specialist scores. The other tokens retain anchor-containing ranked metadata and do not enter combination.

decision = AnchorVetoTop2Selection()(
    router(features, layer_index=1),
    expert_ids=router.expert_ids_for_layer(1),
)
approved_outputs = expert_outputs[decision.expert_pair_mask]
approved_indices = decision.selected_indices[decision.expert_pair_mask] - 1
ranked_outputs = torch.gather(
    approved_outputs,
    dim=1,
    index=approved_indices.unsqueeze(-1).expand_as(approved_outputs),
)
mixed = ScoreSoftmaxCombination()(
    ranked_outputs,
    selected_scores=decision.selected_scores[decision.expert_pair_mask],
)
assert torch.allclose(mixed.weights.sum(dim=-1), torch.ones(2))

Verify sparse dispatch and capacity degradation

Run the complete standalone execution example:

python examples/verify_top2_dispatch.py

Its three tokens begin with approved, approved, and vetoed selection metadata. Availability retains the first pair, degrades the second pair to its single available specialist, and leaves the third token on the anchor.

result = AnchoredTop2Dispatcher(
    combination_policy=ScoreSoftmaxCombination(mix_temperature=1.0)
).dispatch(
    hidden_states,
    expert_scores=scores,
    expert_ids=router.expert_ids_for_layer(1),
    anchor_mlp=anchor,
    experts=(code, math),
    expert_availability=torch.tensor(
        [[[True, True], [True, False], [False, False]]]
    ),
)
assert result.capacity.route_arity.tolist() == [[2, 1, 0]]
assert result.mixture_weights[0, 1].tolist() == [1.0, 0.0]

Current limitations

• This is a standalone composition, not a package-to-model lifecycle. install(), synchronize_registry(), configuration serialization, and composition reload currently materialize anchored top-1 only.
• The deterministic fixtures validate routing and tensor behavior, not whether a weighted pair improves language-model quality.
• Two positive scores do not prove expert compatibility. Use an explicit compatibility-constrained policy when unrelated experts must never mix.
• Availability is caller-supplied. The dispatcher does not estimate serving capacity or search beyond the already-selected pair.
• The current dispatcher is not attached to RoutedMLP; optimized grouping, top-2 telemetry, token-aligned auxiliary arguments, and distributed execution remain future work.
• Score-softmax weights are a tested baseline, not learned evidence that the mixture is optimal.

References and related pages

Anchor-veto selection, fixed-zero abstention, capacity degradation, and the standalone dispatcher are project-specific framework mechanisms rather than paper-derived algorithms. Score softmax uses the standard PyTorch operation; the repository architecture defines how it is applied to ranked expert activations.

• Policies defines stable anchor-veto ranking, score-softmax combination, and top-2-to-top-1 degradation.
• Inference explains sparse pair dispatch, differentiable reconstruction, and active-work accounting.
• Routers describes append-only affine heads and their stable expert order.
• Experts distinguishes complete replacement experts from always-on-anchor deltas.
• Evaluation covers counterfactual quality comparison and compute measurement without claiming that every mixture is beneficial.

Adaptive anchor/top-1/top-2

The problem it solves

Fixed top-1 always spends one MLP execution, while anchor-veto top-2 requires two positive specialists before either one can run. Adaptive selection keeps the anchor when specialist evidence is absent, runs one specialist when only the strongest score is convincing, and approves a pair when the second score clears a separate mixture threshold.

This makes the intended computation visible per token without forcing every specialist-routed token to pay for a second expert. The selection policy and its affine-router integration are validated through public APIs. Replacement expert execution still requires an explicit dispatcher that does not yet accept adaptive decisions.

Workflow contract

• Expert representation: zero, one, or two compatible complete experts may be selected. The verified fixture exercises route metadata rather than expert activations.
• Router: append-only affine heads score frozen identity features in stable expert order.
• Selection: the best specialist must be strictly positive to beat the fixed-zero anchor. A second specialist joins only when its score is strictly greater than mix_threshold; an exact threshold tie remains top-1.
• Combination and capacity: the decision records ranked candidates and route arity, but the fixture does not execute or combine replacement experts. No serving-capacity degradation is applied.
• Training and calibration: the example consumes deterministic affine heads. It does not train experts, fit router weights, or calibrate the mixture threshold.
• Active work: route arity zero means one anchor MLP execution; arity one means one specialist MLP execution; arity two means two specialist MLP executions. The evaluation utilities validate this accounting separately.
• Packaging and serialization: standalone objects are assembled directly. Adaptive package attachment, model-level configuration, and composition reload are not implemented.
• Expected result: the four tokens deterministically choose top-2, top-1, anchor, and top-1, including the exact-threshold boundary case.

How it decides

Calibrated specialist scores
  → no positive winner: anchor
  → one positive winner, second score at or below threshold: top-1
  → second score strictly above threshold: top-2
  → retain ranked candidates, masks, and route arity for auditing

The activation threshold and mixture threshold answer different questions. Zero asks whether any specialist should replace the anchor. mix_threshold asks whether the runner-up provides enough evidence to justify a second specialist execution.

Variable-compute rule

Four-token example

With mix_threshold = 0.5, the deterministic affine heads produce:

“Active paths” counts executed MLPs, while route arity counts selected specialists. The anchor row therefore has arity zero but still performs one anchor MLP execution.

Minimal explicit selection

features = IdentityRouterFeatureExtractor(2)(hidden_states)
scores = router(features, layer_index=1)
decision = AdaptiveTop1Top2Selection(mix_threshold=0.5)(
    scores,
    expert_ids=router.expert_ids_for_layer(1),
)
assert decision.route_arity.tolist() == [[2, 1, 0, 1]]

Run the verified example

From the repository root, run:

python examples/verify_adaptive_selection.py

The example is offline and uses deterministic tensors. It reports the stable ranked candidates and route arity [[2, 1, 0, 1]].

decision = AdaptiveTop1Top2Selection(mix_threshold=0.5)(
    router(features, layer_index=1),
    expert_ids=router.expert_ids_for_layer(1),
)

assert torch.equal(
    decision.route_arity,
    torch.tensor([[2, 1, 0, 1]]),
)
assert int(decision.anchor_mask.sum()) == 1
assert int(decision.top1_mask.sum()) == 2
assert int(decision.top2_mask.sum()) == 1

Current limitations

• The validated workflow stops at route selection. It does not execute adaptive replacement experts or produce combined activations.
• AnchoredTop2Dispatcher consumes top-2 selection policies, not AdaptiveRouteDecision. Adaptive sparse execution is currently available only for always-on-anchor delta experts, which belongs to the next guide.
• There is no package-driven adaptive attachment, RoutedMLP replacement mode, model-level configuration, composition checkpoint, or route-decision serialization.
• The fixture validates routing semantics and active-work accounting, not language quality, calibration quality, latency, or beneficial expert mixtures.
• The mixture threshold is supplied directly. Production threshold fitting and compatibility constraints remain separate concerns.

References and related pages

Adaptive anchor/top-1/top-2 selection is a project-specific framework mechanism derived from the architecture blueprint rather than a paper implementation. It uses stable PyTorch sorting, while its fixed-zero abstention and strict mixture threshold are Mutations design choices.

• Policies defines adaptive decision shapes, masks, threshold semantics, and safe padding for small registries.
• Routers explains the frozen feature and append-only affine score path used by the fixture.
• Inference distinguishes validated adaptive delta execution from the missing replacement-expert adaptive dispatcher.
• Evaluation defines active-MLP accounting for adaptive decisions.
• Weighted top-2 shows the existing standalone replacement-expert dispatcher and score-softmax combination boundary.

Always-on anchor with routed LoRA deltas

The problem it solves

Replacement routing chooses the anchor or a specialist. This workflow instead keeps the immutable anchor active for every token and adds zero, one, or two selected anchor-relative LoRA updates. Tokens with no useful specialist evidence therefore preserve the dense output exactly, while positive routes can adapt it without replacing the anchor path.

The delta expert, adaptive selection, availability fallback, sparse dispatch, combination, and explicit RoutedMLP mode are validated through public APIs. They still require direct assembly: installed packages do not automatically materialize LoRA experts or activate this execution mode.

Workflow contract

• Expert representation: each PEFTLoRAExpert owns adapter tensors over a frozen private copy of the compatible anchor MLP. Its forward_delta() path returns the complete private-expert output minus the already-computed live anchor activation.
• Router: the verified fixture supplies deterministic scores for code and math in stable expert order. A separate tiny-Qwen3 integration test binds the same delta runtime to an identity feature extractor and frozen affine head.
• Selection: adaptive selection chooses zero, one, or two specialists. The fixed-zero anchor wins when the best score is non-positive; the runner-up joins only when it is strictly above mix_threshold.
• Combination and capacity: unavailable selected experts are removed without searching outside the selected set. Zero survivors return the exact anchor, one survivor receives unit weight, and two survivors receive score-softmax weights.
• Training and calibration: the guide starts from deterministic standalone adapters and scores. It does not train LoRA factors, fit router heads, calibrate thresholds, or prove that a selected mixture improves quality.
• Active work: the anchor executes once for every token, followed by zero, one, or two sparse delta-expert calls. The current LoRA delta path computes a complete private expert before subtraction, so this is a correctness reference rather than a native delta-only performance kernel.
• Packaging and serialization: explicit RoutedMLP configuration is available, and standalone LoRA adapter payloads are validated. Package-driven LoRA materialization and composition-checkpoint reload are not implemented.
• Expected result: three tokens take exact anchor, top-1 delta, and top-2 delta routes; the anchor row is bitwise equal, active weights follow route arity, and the live anchor remains frozen.

How it runs

Live immutable anchor → execute every token once
Deterministic specialist scores → adaptive zero/top-1/top-2 selection
Explicit availability → keep selected survivors or return to anchor
Sparse LoRA delta calls → zero, unit, or score-softmax weights
Anchor activation + weighted deltas → final output and auditable route data

The anchor and delta branches have different responsibilities. The anchor is the stable base computation, while each selected specialist contributes only its change relative to that base.

Anchor-plus-delta rule

Three-token example

With mix_threshold = 0.25, the deterministic scores produce:

The pair weights are approximately (0.7311, 0.2689) and sum to one. These weights mix delta activations after selection; they do not interpolate LoRA factor matrices.

Minimal explicit composition

result = AlwaysOnAnchorDeltaDispatcher(
    selection_policy=AdaptiveTop1Top2Selection(mix_threshold=0.25),
).dispatch(
    hidden_states,
    expert_scores=scores,
    expert_ids=("code", "math"),
    anchor_mlp=anchor,
    experts=(code, math),
)

Run the verified example

From the repository root, run:

python examples/verify_always_on_anchor_deltas.py

The example is offline and deterministic. It reports route arity [0, 1, 2], one exact anchor row, the peft_lora parameterization, and the delta weights used for every token.

anchor_output = anchor(hidden_states)
assert torch.equal(result.output[0], anchor_output[0])
assert torch.equal(result.capacity.route_arity, torch.tensor([0, 1, 2]))
assert torch.equal(result.delta_weights[0], torch.tensor([0.0, 0.0]))
assert torch.equal(result.delta_weights[1], torch.tensor([1.0, 0.0]))
assert torch.allclose(result.delta_weights[2].sum(), torch.tensor(1.0))
assert all(not parameter.requires_grad for parameter in anchor.parameters())

wrapper.configure_always_on_anchor_deltas(
    layer_index=2,
    router_feature_extractor=IdentityRouterFeatureExtractor(hidden_size),
    router_heads=heads,
    experts={"code": expert},
)
actual = wrapper(hidden_states)
assert wrapper.routing_mode == "always_on_anchor_deltas"
assert torch.equal(actual, expected)

Current limitations

• Package-driven LoRA materialization, delta-mode registry synchronization, and delta composition-checkpoint reload are not implemented.
• PEFTLoRAExpert.forward_delta() currently evaluates a frozen private-base expert and subtracts the live anchor output. It is storage-efficient adapter state, but not a native delta-only execution kernel or a latency claim.
• The dispatcher supports anchored top-1 or adaptive zero/top-1/top-2 selection. Arbitrary-(k) delta routing and compatibility-constrained delta mixtures remain separate work.
• Delta routes expose selection, capacity, and weights, but they do not feed the existing route-telemetry history.
• The examples use tiny deterministic CPU fixtures. Production Qwen3-4B, reduced precision, multi-device, distributed, and fused-kernel behavior require separate validation.
• Synthetic scores and manually perturbed adapter factors prove framework semantics, not calibration quality, language quality, specialist compatibility, or beneficial mixtures.

References and related pages

LoRA supplies the compact low-rank update, while shared-plus-routed MoE work motivates preserving a common computation beside specialized contributions. Mutations applies those ideas to independently trained plug-ins around an immutable pretrained anchor.

• Experts explains LoRA ownership, exact initialization, adapter-only payloads, and the current private-base execution tradeoff.
• Policies defines adaptive selection, availability degradation, score-softmax weights, and always-on-anchor combination.
• Inference documents sparse delta dispatch, output reconstruction, and explicit RoutedMLP configuration.
• Training covers LoRA forced-path training and immutable anchor guarantees.
• Evaluation defines active-MLP accounting without converting reference semantics into unsupported performance claims.

Two-expert local mini-MoE training

The problem it solves

Independently trained experts can each be useful while still combining poorly. This workflow trains exactly two package-local experts and fresh affine router heads together, allowing their transformations and mixture weights to co-adapt without changing the anchor or shared backbone.

There are two supported entry points. TRN-003 starts the local pair directly from compatible expert objects. TRN-004 first trains the same pair independently through TRN-002, then applies the identical joint local runtime as a mixture-aware refinement stage. Both paths are composable training workflows rather than package-driven deployment paths.

Workflow contract

• Expert representation: exactly two complete experts cover the same converted layers in the same stable order. The verified examples use full-MLP experts copied from the immutable anchor.
• Router: fresh zero-initialized affine heads are owned by the local pair. Small opposite biases break the symmetry between anchor-copied experts deterministically before optimization begins.
• Selection and combination: no sparse top-1 decision occurs during local training. Both experts execute every specialization token, and COM-001 score-softmax weights mix their activations.
• Training: ordinary Hugging Face Trainer optimization updates only the expert-declared parameter subsets and local affine heads. The anchor, backbone, feature extractor, and incumbent artifacts remain frozen.
• Optional balancing: BAL-003 adds a Switch-style auxiliary objective from local mixture probabilities. It changes training pressure but not execution; both experts still run for every eligible token.
• Active work: two specialist MLPs per specialization token at every converted layer. There is no anchor abstention, capacity overflow, or token dropping inside this dense local training path.
• Lifecycle cleanup: success and failure both clear gradients, freeze the local artifacts, release retained objective observations, and restore every converted wrapper to exact forced_anchor execution.
• Packaging and serialization: training reports record immutable topology, optimizer ownership, progress, and final objectives. Packaging the trained local router with its expert pair and attaching it for inference remain future work.
• Expected result: both experts and router heads train, the anchor remains byte-exact, and no local training route remains attached after the run.

How the two paths run

Direct local training (TRN-003)
  compatible two-expert rectangle + fresh heads
  → validate ownership and storage isolation
  → jointly train dense score-softmax mixtures
  → freeze artifacts and restore forced anchor

Mixture-aware refinement (TRN-004)
  train the same two experts independently (TRN-002)
  → verify report order and layer topology
  → run the unchanged TRN-003 joint stage
  → return one immutable report linking both stages

The refinement coordinator does not infer historical identity from mutable objects. It requires the caller-owned bindings to match the completed independent report’s stable expert order and complete layer rectangle before it attaches the local route.

Dense local mixture

Fresh heads begin with zero weights and biases, then receive opposite +ε and −ε biases in stable expert order. This deterministic perturbation prevents two identical anchor copies from beginning with perfectly symmetric router behavior without introducing calibrated routing evidence.

Optional balancing objective

Train the local pair directly

Run the deterministic two-step CPU example:

python examples/verify_local_mini_moe.py

It constructs one converted tiny-Qwen3 layer, two full-MLP experts, fresh heads, and a masked synthetic dataset. After training, it proves that both experts changed, the heads became non-zero, the optional balancing objective was composed correctly, and the anchor remained exact.

trainer = LocalMiniMoETrainer(
    model=model,
    backend=backend,
    expert_bindings=bindings,
    router_feature_extractor=IdentityRouterFeatureExtractor(16),
    router_heads=heads,
    config=LocalMiniMoETrainingConfig(load_balancing_weight=0.05),
    args=training_arguments,
    train_dataset=dataset,
)
trainer.train()
objectives = trainer.training_report.final_objectives
assert objectives.load_balancing_loss > 0

Refine an independently trained pair

Run the deterministic three-stage CPU example:

python examples/verify_mixture_aware_refinement.py

The first two steps train code:0 and code:1 independently on distinct synthetic partitions. The final step binds the same expert objects and stable IDs to fresh local heads, jointly refines them, and returns a report linking the independent and local stages.

report = refine_independent_expert_set(
    model,
    backend=backend,
    source_training=independent_report,
    expert_bindings=local_bindings,
    router_feature_extractor=IdentityRouterFeatureExtractor(16),
    router_heads=fresh_heads,
    config=LocalMiniMoETrainingConfig(load_balancing_weight=0.05),
    args=refinement_arguments,
    trainer_kwargs={"train_dataset": refinement_dataset},
)
assert report.expert_ids == independent_report.expert_ids
assert report.total_global_steps > independent_report.total_global_steps

Current limitations

• Local mini-MoE training supports exactly two experts with identical complete converted-layer coverage. Larger or ragged local expert sets are not implemented.
• Both experts execute for every specialization token. There is no discrete local top-k, anchor fallback, capacity handling, token dropping, or sparse training-compute claim.
• Router heads must be fresh, temperature-one, threshold-zero affine heads. Previously calibrated or trained heads are rejected rather than silently overwritten.
• TRN-004 validates report topology and stable IDs, not cryptographic identity of caller-owned expert objects. Artifact hashes belong to later export and package boundaries.
• Training reports do not serialize expert weights or local router heads. Package-local router bundling, registry synchronization, model-level attachment, and composition reload are not implemented for this workflow.
• The optional balancing term measures preference while execution stays dense. It does not demonstrate reduced serving load or balanced sparse capacity.
• The deterministic tiny-Qwen3 fixtures validate ownership, gradients, objectives, and cleanup—not language quality, beneficial specialization, production memory, latency, mixed precision, multi-device, or distributed training.

References and related pages

Joint local training and the independent-then-refine coordinator are project-specific framework workflows defined by the architecture blueprint. The balancing term adapts the Switch auxiliary objective to a dense two-expert local mixture.

• Training defines TRN-002, TRN-003, TRN-004, and BAL-003, including parameter ownership and failure-safe cleanup.
• Experts explains full-MLP expert copying, independent storage, and declared trainable subsets.
• Routers documents affine heads, stable expert order, and frozen feature extraction.
• Policies defines COM-001 score-softmax activation combination.
• Evaluation explains why these deterministic checks are framework evidence rather than specialization-quality measurements.

Hierarchical package routing

The problem it solves

A flat router must compare every specialist for every token, even when specialists already belong to distinct packages such as code and mathematics. This workflow first asks whether any package has positive evidence over the anchor. Only the winning package then scores its local experts and contributes a pair, keeping every mixture inside one package.

The hierarchy is useful for studying larger expert collections without pretending that package-level and local scores form one globally calibrated score bank.

Workflow contract

• Expert representation: two or more globally unique, package-local expert IDs per package. The verified sparse execution fixture uses four deterministic complete expert modules across two packages.
• Routing and selection: anchored package top-1 runs first. Selected non-anchor rows are grouped by package and scored only by that package’s frozen local logical router. Stable local top-2 then selects a pair without a second anchor veto.
• Combination and capacity: AnchoredTop2Dispatcher.dispatch_preselected() reuses the ordinary top-2 validation, sparse execution, score-softmax combination, and reconstruction path. Capacity policies remain independently composable.
• Training and calibration: this workflow assumes that package and local affine heads already exist. It performs no fitting, calibration, or hidden recalibration.
• Active work: every token receives package scores. An anchor token runs one anchor MLP and receives no local scores; an approved token receives only its winning package’s local scores and runs that package’s selected expert pair.
• Packaging and serialization: the hierarchy is assembled explicitly in memory from copied, frozen logical routers. There is no hierarchy manifest, package-driven model attachment, or hierarchy checkpoint format.
• Expected result: package selection is anchor-first, local scoring omits anchor rows, selected pairs never cross package boundaries, and sparse execution restores the original token order.

How the two stages run

Token features
  → score every package against the fixed-zero anchor
  → anchor rows stop at the dense MLP
  → group approved rows by their winning package
  → score only that package's local experts
  → select the package-local top two
  → reuse sparse top-2 dispatch and score-softmax combination

Package ties follow stable package order after the anchor-first comparison. Once a package wins, its local pair is unconditional: negative local scores remain eligible because the package gate already made the abstention decision.

Package gate and local-pair rule

Four-token walkthrough

The offline fixture uses code and mathematics package heads. Candidate index zero is the anchor; global expert indices 1–2 belong to code and 3–4 belong to mathematics.

The mathematics local heads deliberately return negative scores. Their pair is still valid because the positive mathematics package score has already passed the anchor gate. This separates package abstention from local expert ranking.

Minimal explicit composition

package_routes = policy.select_packages(package_scores, package_ids=package_ids)
local_scores = hierarchy.score_selected_locals(
    hidden_states, package_indices=package_routes.selected_indices,
    package_ids=package_routes.expert_ids, layer_index=0,
)
routes = policy.select_locals(
    package_routes, local_scores, expert_ids_by_package=expert_ids_by_package,
)
result = dispatcher.dispatch_preselected(
    hidden_states, selected_routes=routes.expert_routes,
    anchor_mlp=anchor, experts=experts,
)

The abbreviated listing makes every stage explicit. The verified sources use keyword arguments and validate the package-major expert ordering before any candidate executes.

Verify staged selection

From the repository root, run:

python examples/verify_hierarchical_routing.py

The example is deterministic and offline. It reports package candidates [1, 2, 0, 1], global expert pairs [[1, 2], [3, 4], [0, 0], [1, 2]], and three locally scored tokens.

package_routes = policy.select_packages(
    router.score_packages(features, layer_index=0),
    package_ids=router.package_ids_for_layer(0),
)
local_scores = router.score_selected_locals(
    features,
    package_indices=package_routes.selected_indices,
    package_ids=package_routes.expert_ids,
    layer_index=0,
)
decision = policy.select_locals(
    package_routes,
    local_scores,
    expert_ids_by_package={
        package_id: router.expert_ids_for_package(package_id, layer_index=0)
        for package_id in router.package_ids
    },
)
assert package_routes.selected_indices.tolist() == [1, 2, 0, 1]
assert decision.expert_routes.selected_indices.tolist() == [
    [1, 2],
    [3, 4],
    [0, 0],
    [1, 2],
]

Verify sparse pair execution

The integration test carries the preselected pairs into the existing top-2 dispatcher. It proves that the anchor sees one token, code experts each see two tokens, mathematics experts each see one token, and no output contains a cross-package mixture.

result = AnchoredTop2Dispatcher().dispatch_preselected(
    hidden_states,
    selected_routes=decision.expert_routes,
    anchor_mlp=anchor,
    experts=experts,
)

assert torch.equal(result.output[2], hidden_states[2])
assert anchor.token_count == 1
assert tuple(expert.token_count for expert in experts) == (2, 2, 1, 1)
for token_index, package_index in enumerate(
    package_routes.selected_indices.tolist()
):
    pair = result.selected_routes.selected_indices[token_index].tolist()
    if package_index == 0:
        assert pair == [0, 0]
        continue
    owners = {
        decision.expert_package_ids[candidate - 1]
        for candidate in pair
    }
    assert owners == {package_routes.expert_ids[package_index - 1]}

Current limitations

• RTR-005 and POL-004 are validated standalone composition primitives, not a configuration-driven RoutedMLP or package-registry lifecycle.
• The current topology is rectangular: every covered layer has the same stable package order, and every package exposes at least two local experts with globally unique IDs. Ragged layers and single-expert local packages are not supported.
• The second stage always selects two local experts. Local top-1, adaptive arity, and a second local anchor veto are separate future policies.
• The hierarchy is rebuilt explicitly from existing affine score banks. It is not serialized in an expert package or composition checkpoint.
• The deterministic fixtures validate routing, ownership, sparse dispatch, combination, and reconstruction—not routing quality, specialization gains, production latency, accelerator behavior, or distributed execution.

References and related pages

The two-stage hierarchy is a project-specific framework design from the architecture blueprint. It reuses existing affine routing, anchored top-1, local top-2, and sparse dispatch primitives rather than claiming a paper-derived routing algorithm.

• Routers defines RTR-003 logical routers and the RTR-005 package/local score bank.
• Policies specifies POL-004, stable tie-breaking, global expert-index translation, and the absence of a second local veto.
• Inference documents INF-002 preselected pair dispatch, score-softmax combination, and exact reconstruction.
• Registry explains the package lifecycle that hierarchical routing does not yet automatically join.
• Evaluation separates deterministic framework evidence from quality and production-efficiency claims.

Prompt-shortlisted token routing

The problem it solves

A flat logical router evaluates every installed expert column for every token. That remains reasonable for a small registry, but its scoring and expert-memory overhead grows with the complete candidate set. This workflow uses external prompt-level evidence to retain at most (M) candidates before ordinary token-level routing.

Shortlisting bounds which affine columns are evaluated without weakening the anchor. A shortlisted specialist still needs positive token-level evidence; otherwise anchored top-1 returns the token to the dense MLP.

Workflow contract

• Expert representation: any complete experts that already satisfy the selected token-routing and dispatch contracts. The deterministic fixture uses code, mathematics, and science stand-ins.
• Routing and selection: PromptTopMShortlister consumes one finite retrieval score per prompt and candidate. PromptShortlistedLogicalRouter evaluates only the selected logical-router columns, reconstructs the stable global candidate order, and passes the resulting scores to POL-001 anchored top-1.
• Combination and capacity: top-1 execution uses the selected complete expert directly. Anchor fallback and capacity remain the responsibility of the existing selection and dispatch path.
• Training and calibration: external retrieval scores and the frozen logical-router heads are inputs. This primitive does not train a prompt encoder, fit retrieval scores, recalibrate token heads, or mutate experts.
• Active work: prompt retrieval considers the caller-provided candidate table once. Token routing evaluates at most (M) affine expert columns per prompt row, then sparse dispatch runs either the anchor or one specialist per token.
• Packaging and serialization: the shortlist and copied logical-router wrapper are assembled explicitly in memory. They are not stored in an expert package, registry router version, or composition checkpoint.
• Expected result: prompt ties preserve candidate order, unselected columns are never evaluated, global expert identity remains stable, and shortlisted negative evidence still falls back exactly to the anchor.

How the two routing scales compose

External prompt retrieval scores
  → select at most M candidates for each prompt
  → evaluate only those token-router columns
  → restore the stable global expert score order
  → apply fixed-zero anchored top-1 per token
  → sparsely execute the winner or the anchor

The prompt stage limits eligibility; it does not choose the executing expert. The token stage can abstain, but it cannot recover an expert omitted by the prompt shortlist.

Candidate-shortlisting rule

Three-prompt walkthrough

The offline fixture keeps exactly one candidate per prompt. Stable global candidate indices are 1 for code, 2 for mathematics, and 3 for science; candidate index 0 remains the anchor.

The science prompt demonstrates the two independent gates. Retrieval makes the science expert eligible, but its token scores remain negative, so the fixed-zero anchor wins both decisions.

Minimal explicit composition

shortlist = PromptTopMShortlister(max_candidates=1)(
    retrieval_scores, candidate_ids=expert_ids,
)
scores = PromptShortlistedLogicalRouter(logical_router)(
    hidden_states, shortlist=shortlist, layer_index=0,
)
result = AnchoredTop1Dispatcher().dispatch(
    hidden_states, expert_scores=scores.scores,
    expert_ids=scores.expert_ids, anchor_mlp=anchor, experts=experts,
)

The caller owns retrieval-score construction and explicitly connects the shortlist, logical router, and existing sparse top-1 dispatcher.

Verify bounded logical scoring

From the repository root, run:

python examples/verify_prompt_shortlisting.py

The example is deterministic and offline. It reports shortlists [[0], [1], [2]], global token routes [[1, 1], [2, 2], [0, 0]], and a copied shortlist wrapper with no trainable parameters.

shortlist = PromptTopMShortlister(max_candidates=1)(
    retrieval_scores,
    candidate_ids=("code:0", "math:0", "science:0"),
)
scores = PromptShortlistedLogicalRouter(
    compile_logical_router(heads)
)(features, shortlist=shortlist, layer_index=0)
routes = AnchoredTop1Selection()(
    scores.scores,
    expert_ids=scores.expert_ids,
)
assert shortlist.selected_indices.tolist() == [[0], [1], [2]]
assert routes.selected_indices.tolist() == [[1, 1], [2, 2], [0, 0]]

Verify sparse expert execution

The focused integration test carries the reconstructed scores through the existing top-1 dispatcher. It proves that code and mathematics each execute once, science never executes, and the anchor receives exactly the rejected science prompt.

result = AnchoredTop1Dispatcher().dispatch(
    hidden_states,
    expert_scores=scores.scores,
    expert_ids=scores.expert_ids,
    anchor_mlp=anchor,
    experts=experts,
)

assert result.selected_routes.selected_indices.tolist() == [
    [1, 1],
    [2, 2],
    [0, 0],
]
assert anchor.calls == 1
assert tuple(expert.calls for expert in experts) == (1, 1, 0)

Current limitations

• RTR-006 requires caller-provided finite retrieval scores. It does not include a prompt encoder, retriever training objective, or calibration workflow.
• A retrieval miss removes a candidate before token routing. The token router cannot expand the shortlist or recover an omitted specialist.
• Version 1 applies one fixed-width shortlist per prompt row. It does not reuse shortlist state across autoregressive generation, page expert weights into memory, or support ragged candidate counts.
• The validated end-to-end composition uses anchored top-1 and complete expert modules. Top-2, adaptive, compatibility-constrained, and micro-expert retrieval combinations remain separate explicit experiments.
• Prompt shortlisting has no package manifest, registry binding, configuration-driven RoutedMLP attachment, or composition-checkpoint representation.
• The fixtures validate stable retrieval, bounded scoring, exact fallback, and sparse execution—not retrieval quality, specialization gains, production latency, accelerator memory savings, or distributed behavior.

References and related pages

Prompt-level shortlisting is project-specific framework infrastructure from the architecture blueprint. It reuses stable sorting, copied logical router columns, anchored top-1, and sparse dispatch rather than claiming a paper-derived retrieval algorithm.

• Routers defines RTR-003, RTR-006, compact candidate scoring, stable ties, finite masking, and the external-retrieval boundary.
• Policies specifies POL-001 anchor-first token selection and explains which advanced policies are not part of this workflow.
• Inference documents INF-001 sparse top-1 execution and exact anchor reconstruction.
• Registry describes the package and composition lifecycle that prompt shortlisting does not yet automatically join.
• Evaluation separates deterministic routing evidence from retrieval quality and production-efficiency claims.

Compatibility-constrained global top-2

The problem it solves

Ordinary global top-2 can combine the two highest-scoring specialists even when they were trained independently and should never run as one mixture. Compatibility-constrained top-2 keeps global candidate scoring, but permits a pair only when both experts declare the same non-empty compatibility group.

The rule is conservative in two ways. Two positive scores are required, and a missing group means “not mixable” rather than membership in an implicit default group. When no eligible pair exists, the fixed-zero anchor vetoes specialist execution.

Workflow contract

• Expert representation: complete experts with stable IDs and explicit package payload bindings. Every expert considered at a converted layer must appear exactly once in the compatibility index.
• Routing and selection: calibrated scores retain global expert order. CompatibilityConstrainedTop2Selection evaluates distinct, positive, same-group pairs and ranks them lexicographically by their stronger and weaker member scores. Stable append order resolves exact ties.
• Combination and capacity: an eligible pair uses the existing score-softmax combination and INF-002 sparse pair dispatcher. If no eligible pair exists, selection produces an anchor-containing decision and the dispatcher executes the dense anchor exactly.
• Training and calibration: the policy consumes frozen scores and declared metadata. It does not learn compatibility, recalibrate router heads, or change expert parameters.
• Active work: all (K) specialist scores remain inputs, compatibility filtering considers candidate pairs, and execution remains bounded to the anchor or two specialists per token.
• Packaging and serialization: ExpertRegistry.mixture_compatibility() derives an immutable layer-local index from active packages. Selection and dispatch are then assembled explicitly; they are not automatically attached to RoutedMLP or saved in a composition checkpoint.
• Expected result: higher-scoring incompatible or ungrouped specialists never execute, compatible selected pairs preserve score gradients and sparse calls, and registry changes produce a new exact index or anchor fallback.

How compatibility filters global routing

Active package manifests and payload bindings
  → derive a layer-local compatibility index
  → score specialists in stable global order
  → keep distinct positive pairs in one explicit group
  → choose the lexicographically strongest eligible pair
  → mix that pair, or execute the anchor exactly

Compatibility is an eligibility rule, not a second router. It prevents invalid mixtures but does not reduce the number of router columns scored.

Eligible-pair rule

Two-token walkthrough

The deterministic fixture has code, two mathematics experts, and one ungrouped general expert. Only the two mathematics experts share math-v1.

For the first token, the ungrouped general expert and incompatible code expert have higher individual scores than at least one mathematics expert, but neither can enter a valid pair. For the second token, only one mathematics score is positive, so the decision records the strongest specialist beside the anchor and the dispatcher executes only the anchor.

Minimal explicit composition

compatibility = registry.mixture_compatibility(layer_index=2)
dispatcher = AnchoredTop2Dispatcher(
    selection_policy=CompatibilityConstrainedTop2Selection(compatibility),
)
result = dispatcher.dispatch(
    hidden_states, expert_scores=scores,
    expert_ids=compatibility.expert_ids,
    anchor_mlp=anchor, experts=experts,
)

The registry supplies immutable compatibility metadata; the caller explicitly connects it to the validated pair selection, combination, and dispatch path.

Verify compatible selection and sparse execution

From the repository root, run:

python examples/verify_compatibility_top2.py

The deterministic offline example reports pair decisions [[2, 3], [4, 0]] and anchor fallback [false, true]. The incompatible code expert and ungrouped general expert are never executed.

compatibility = MixtureCompatibilityIndex(
    expert_ids=("code:0", "math:0", "math:1", "general:0"),
    group_ids=("code-v1", "math-v1", "math-v1", None),
)
result = AnchoredTop2Dispatcher(
    selection_policy=CompatibilityConstrainedTop2Selection(compatibility),
).dispatch(hidden, expert_scores=scores, expert_ids=compatibility.expert_ids,
           anchor_mlp=anchor, experts=experts)
assert result.selected_routes.selected_indices.tolist() == [[2, 3], [4, 0]]
assert code.calls == general.calls == 0

Verify registry lifecycle behavior

The focused integration test derives compatibility from three installed packages. It first selects a declared code-and-mathematics pair, then unloads one member and proves that the remaining incompatible candidates return to the anchor.

index = registry.mixture_compatibility(layer_index=2)
assert index.group_ids == ("reasoning-v1", "reasoning-v1", None)

registry.unload(math.identity)
reduced = registry.mixture_compatibility(layer_index=2)
fallback = AnchoredTop2Dispatcher(
    selection_policy=CompatibilityConstrainedTop2Selection(reduced),
).dispatch(hidden, expert_scores=torch.tensor([[0.9, 1.2]]),
           expert_ids=reduced.expert_ids, anchor_mlp=anchor, experts=experts)
assert fallback.selected_routes.anchor_mask.item()
assert torch.equal(fallback.output, hidden)

Current limitations

• Compatibility requires explicit package payload bindings and complete, unambiguous layer coverage. Legacy binding-free packages cannot supply an index.
• A package-level group is a coarse permission. Version 1 does not learn pairwise utility, express layer-specific group membership, or prove that an allowed mixture is beneficial.
• Pair filtering does not reduce flat-router score work and a straightforward global pair search scales quadratically with the candidate count.
• The validated path is explicit standalone composition. Package-driven top-2 RoutedMLP attachment, model-level strategy configuration, composition-checkpoint persistence, and route telemetry remain future work.
• Capacity degradation beyond the existing pair-shaped dispatcher, accelerator performance, distributed execution, and production-scale registries are not validated here.
• The fixtures establish deterministic eligibility, lifecycle updates, sparse calls, gradients, and exact fallback—not specialization quality or gains from expert synergy.

References and related pages

Compatibility-constrained top-2 is project-specific framework infrastructure from the architecture blueprint. It applies declared metadata to the existing anchored pair policy and does not claim a paper-derived compatibility-learning method.

• Registry defines PKG-003, immutable compatibility indices, stable expert-ID alignment, and install/unload behavior.
• Policies specifies POL-005 positive-pair eligibility, lexicographic ranking, deterministic ties, and anchor fallback.
• Inference documents INF-002 sparse pair execution, score-softmax combination, and exact token reconstruction.
• Calibration explains where the frozen specialist scores come from; compatibility filtering does not recalibrate them.
• Evaluation separates deterministic framework evidence from mixture-quality and production-scaling claims.

Confidence-gated dynamic-k selection

The problem it solves

Fixed top-1 and top-2 policies assign the same maximum specialist count to every token. Dynamic-(k) selection lets clear anchor cases use no specialist, lets simple positive cases retain one, and lets stronger multi-expert evidence retain more—without exceeding a configured per-token ceiling.

This guide stops at the selection decision. It verifies which expert IDs and scores survive, but it does not execute or combine an arbitrary number of experts.

Workflow contract

• Expert representation: selection is representation-agnostic and consumes only stable expert IDs plus a score column for each candidate. The selected experts still need mutually compatible execution and combination semantics.
• Routing and selection: append-only affine heads produce specialist scores in deterministic order. DynamicKSelection stable-ranks them, keeps only scores strictly above activation_threshold, and caps the active prefix at max_experts.
• Combination and capacity: POL-006 returns auditable route metadata only. It does not define arbitrary-(k) weights, sparse dispatch, overflow, or batch-wide capacity allocation.
• Training and calibration: the policy does not train a difficulty model or recalibrate scores. The threshold must be chosen externally from held-out data if it is meant to represent calibrated confidence.
• Active work: all (K) specialist scores are inputs to selection. The decision retains at most K_max ranked candidates per token, but this fixture makes no execution-cost or latency claim.
• Packaging and serialization: the policy is a standalone public primitive. It is not attached through package synchronization, model configuration, or composition checkpoints.
• Expected result: route arity is deterministic, bounded between zero and K_max, exact threshold ties remain inactive, and arity zero records an anchor decision.

How confidence becomes a bounded route

Calibrated specialist scores
  → stable descending specialist ranking
  → discard scores at or below the activation threshold
  → retain at most Kmax candidates
  → expose arity, ranked IDs, scores, and masks

The threshold decides whether evidence is strong enough to activate a specialist. The budget decides how many qualifying specialists can survive. Neither step supplies mixture weights or runs an expert module.

Bounded active-arity rule

Four-token walkthrough

The verified fixture scores code, mathematics, and science candidates with activation_threshold=0.5 and max_experts=2.

The final mathematics score equals the threshold and is therefore inactive. This strict boundary is deliberate: a threshold tie cannot silently spend additional specialist compute.

Minimal explicit selection

scores = router(features, layer_index=1)
decision = DynamicKSelection(
    max_experts=2,
    activation_threshold=0.5,
)(
    scores,
    expert_ids=router.expert_ids_for_layer(1),
)

The result provides route_arity, ranked_indices, ranked_scores, anchor_mask, and per-expert or per-arity masks for an explicitly composed downstream experiment.

Run the verified example

From the repository root, run:

python examples/verify_dynamic_k_selection.py

The deterministic offline example reports route arities [[2, 1, 0, 2]] and ranked candidate indices [[[2, 3], [1, 3], [1, 3], [1, 3]]].

decision = DynamicKSelection(
    max_experts=2,
    activation_threshold=0.5,
)(
    router(features, layer_index=1),
    expert_ids=router.expert_ids_for_layer(1),
)
assert decision.route_arity.tolist() == [[2, 1, 0, 2]]
assert decision.anchor_mask.sum().item() == 1
assert decision.max_experts == 2

Current limitations

• POL-006 counts calibrated scores above a fixed threshold; it does not learn token difficulty, cumulative probability, entropy, or a compute-allocation objective.
• The budget is per token. There is no batch-wide, layer-wide, or model-wide budget optimizer and no interaction with expert capacity.
• Arbitrary-(k) combination, sparse dispatch, overflow handling, route telemetry, package-driven attachment, model configuration, and composition checkpoint persistence remain future work.
• Selecting multiple experts does not establish that they are mutually compatible or that their mixture improves quality.
• The fixture validates deterministic scores, ordering, masks, gradients, threshold behavior, and compute bounds—not specialist quality, calibrated utility, latency, memory savings, or distributed execution.

References and related pages

• Policies defines POL-006, the stable ranking contract, strict threshold semantics, shapes, masks, and validation boundaries.
• Routers explains the append-only affine scores consumed by this fixture.
• Calibration covers held-out score calibration and threshold selection; POL-006 performs neither automatically.
• Inference documents the currently validated top-1 and pair-shaped top-2 dispatchers, not an arbitrary-(k) dispatcher.
• Evaluation separates route determinism and active-work accounting from quality or production-efficiency evidence.

Product-key retrieval over micro-experts

The problem it solves

Scoring one independent routing head for every member of a very large fine-grained expert set grows linearly with the complete set. Product-key retrieval represents a square set of (C^2) stable micro-expert identifiers with two smaller (C)-row key tables, then recovers the exact top-(k) identifiers from their pairwise scores.

This guide covers retrieval only. A returned identifier names one partial micro-expert contribution; it is not yet a complete replacement MLP and the current framework does not execute or aggregate the selected subset.

Workflow contract

• Expert representation: FineGrainedMicroExpertSet partitions one compatible bias-free gated MLP into an atomic set of narrow channel contributions. EXP-005 version one requires a perfect-square expert count so two equal subkey tables can bind one-to-one to its stable row-major IDs.
• Routing and selection: ProductKeyMicroExpertRetriever accepts an already projected query with two equal-width halves. It stable-ranks each subkey table, scores the Cartesian product of their retained rows, and returns the exact global top-(k) identifiers, key pairs, masks, and raw scores.
• Combination and capacity: none. The retriever does not execute a micro-expert, assign output weights, aggregate partial contributions, enforce capacity, or fall back to the anchor.
• Training and calibration: the two key tables may be trainable, and selected raw scores remain connected to query and key gradients. Query projection, normalization, retrieval objectives, and training orchestration are caller-owned and not implemented by this primitive.
• Active work: the retriever scores two (C)-row tables, then compares at most (\min(k,C)^2) candidate pairs instead of materializing all (C^2) pair scores. No expert module runs.
• Packaging and serialization: an outer PromptTopMShortlister can explicitly choose a package before its local product-key retriever runs. Keys and retrieval configuration are not yet package payloads, registry router versions, or composition-checkpoint state.
• Expected result: bounded search exactly matches exhaustive Cartesian ranking, equal scores prefer the lower row-major ID, gradients and ownership stay explicit, and the anchor plus bound micro-expert set remain unchanged.

How the split index works

Caller-projected query q = [q1; q2]
  → rank rows in the first key table with q1
  → rank rows in the second key table with q2
  → score pairs from the two retained row sets
  → stable-rank the pairs by their global row-major IDs
  → return top-k IDs, pairs, masks, and graph-connected scores

The first stage cannot exclude a globally top-(k) pair: any such pair must use a row that appears in the top (\min(k,C)) rows of each corresponding subkey search. The second stable ranking therefore produces the same ordered identifiers as exhaustive scoring.

Product-key score

Two-query walkthrough

For a readable fixture, let both two-row key tables be the (2\times2) identity matrix. The four row-major product keys are 0=(0,0), 1=(0,1), 2=(1,0), and 3=(1,1).

This (C=2) example is intentionally small enough to inspect by hand. The retriever’s value appears when (C) is large and (k) stays small: it avoids building the complete (C^2)-score tensor while preserving exact ranking.

Minimal explicit retrieval

retriever = ProductKeyMicroExpertRetriever.from_micro_expert_set(
    micro_experts,
    first_subkeys,
    second_subkeys,
    config=ProductKeyRetrievalConfig(top_k=3),
    expert_id_prefix="code-micro",
)
retrieval = retriever(projected_queries)

retrieval.selected_indices, selected_pairs, selected_scores, and candidate_mask describe the decision. A downstream experiment must still define how the chosen partial contributions execute and combine.

Run the verified example

From the repository root, run:

python examples/verify_product_key_retrieval.py

The deterministic offline example binds four micro-experts to two two-row key tables, requests three results per query, and proves exact exhaustive equivalence, stable ties [0, 1, 2], gradient ownership, expert immutability, and explicit package shortlisting.

result = retriever(queries)
exhaustive = (
    (queries[..., :3] @ first_subkeys.T).unsqueeze(-1)
    + (queries[..., 3:] @ second_subkeys.T).unsqueeze(-2)
).flatten(start_dim=-2)
expected = torch.argsort(
    exhaustive, dim=-1, descending=True, stable=True
)[..., :3]
assert torch.equal(result.selected_indices, expected)
assert tied.selected_indices.tolist() == [[0, 1, 2]]

Current limitations

• EXP-005 version one supports one retrieval head over a perfect-square micro-expert count and two equal subkey tables. Non-square, ragged, hashed, or multi-head indices are not implemented.
• Queries must already be projected to two equal halves. Learned query projection, normalization, retrieval loss, score activation, calibration, and trainer integration remain future work.
• Selected micro-expert dispatch, subset combination, output aggregation, capacity, overflow, shared experts, and exact anchor-preserving execution are not implemented.
• Product keys and configuration are not exported in expert packages, synchronized through the registry, attached by RoutedMLP, or saved in composition checkpoints.
• The exact search uses ordinary PyTorch operations. It makes no fused-kernel, memory-paging, accelerator-speed, million-expert throughput, or distributed scaling claim.
• The fixtures validate indexing, exact ranking, ties, gradients, and ownership—not retrieval quality, useful specialization, language-model quality, or production efficiency.

References and related pages

• Experts defines EXP-004 micro-expert segmentation, EXP-005 product-key retrieval, tensor shapes, ownership, and API details.
• Routers documents RTR-006 prompt-level shortlisting, which can explicitly choose a package before local product-key retrieval.
• Inference explains why existing top-1 and pair-shaped top-2 replacement dispatch cannot execute partial micro-expert subsets.
• Training describes current expert and router training workflows; none yet orchestrates product-key query or key learning.
• Evaluation separates exact retrieval correctness from quality, memory, latency, and production-scaling evidence.

Needle branching and browser deployment

These examples construct tiny random Needle models locally. Install the optional backend and run the checks from the repository root:

python -m pip install -e ".[needle]"
python examples/verify_needle_branches.py
python examples/verify_needle_deployment.py
python examples/verify_needle_browser.py --output .cache/browser-demo

• verify_needle_branches.py checks shared-prefix equivalence, isolated specialist/router fitting, held-out calibration, and exact package reload.
• verify_needle_deployment.py checks cached/reference generation parity and writes full native response models.
• verify_needle_browser.py creates a base-plus-patch bundle and an HTML smoke page. Choose a fresh output directory, supply the official engine files, and serve the directory using the browser walkthrough.

The browser page loads and decodes all reconstructed variants and exercises fallback with explicit recent context. These fixtures validate implementation and transport behavior; they do not establish learned route quality or production-model performance. See Needle branches for training with caller-labeled requests and a real local checkpoint.