Evaluate MoE Quality, Routing, and Inference Cost

What this part does

Evaluation turns the framework’s architectural promises into explicit, inspectable measurements. It checks exact dense fallback, route and package lifecycle behavior, top-1 versus top-2 quality, execution cost, registry composition, and router-data scaling.

These evaluators are validated public APIs, but callers deliberately assemble them around a model, registry, policy decision, or completed experiment. They do not run automatically during inference, install packages, choose benchmark data, or turn synthetic measurements into production claims.

Where it fits

Evaluation observes completed framework operations without owning them:

1. conversion produces dense and anchored outputs for equivalence checks;
2. routing produces decisions for agreement and active-work accounting;
3. registry operations produce immutable snapshots for lifecycle checks;
4. counterfactual evaluation produces aligned losses for quality comparison;
5. callers wrap a fixed operation for timing and storage measurement;
6. completed routing and training runs provide scaling measurements.

The package returns immutable, JSON-compatible reports where practical. Evaluation does not silently synchronize a model, recalibrate a router, retain prompt content, or mutate the supplied tensors, modules, packages, or registry.

Implemented approaches

Exact equivalence and route agreement

EVAL-001 compares tensors with declared absolute and relative tolerances and reports whether PyTorch’s exact elementwise equality also holds. The dense conversion test compares every hidden state and final logits, then fingerprints the selected anchor MLP state using sorted names, shapes, dtypes, and tensor bytes.

EVAL-002 compares the candidates that actually executed under one stable expert ordering. It refuses mismatched token shapes or expert IDs, reports the matching token count and agreement fraction, and records no token text, hidden states, logits, or activations.

Lifecycle and registry composition

EVAL-003 checks removal and reinsertion from immutable registry snapshots. Exact restoration requires the package count to fall on removal and the original active order, registry digest, and compatibility binding to return after reinsertion.

EVAL-006 extends that check to the empty registry and every installation order of every non-empty package subset. It calculates the complete trial count before observation and rejects an experiment that exceeds the caller’s bound. Each order starts from a fresh registry, while an optional caller-owned digest can test whether model behavior is order-invariant.

Top-1/top-2 quality

EVAL-004 aligns top-1 and anchor-veto top-2 decisions with an explicit counterfactual loss table. It compares each selected route with the best evaluated candidate, separates best-single from best-pair quality, and reports how often an executed pair is worse than the selected single specialist.

The evaluator never invents a missing pair loss. Candidate coverage, token shape, and expert ordering must match exactly before any aggregate is returned.

Efficiency accounting

EVAL-005 measures synchronized warmup-aware latency and throughput for any zero-argument operation. It also counts unique resident parameter and buffer storage and derives active MLP executions directly from an immutable route decision.

Work units are caller-defined, so throughput is comparable only when two strategies use the same definition. Timing tests validate contracts and positive durations rather than assuming one machine-specific performance threshold.

Router-data scaling

EVAL-007 compares fixed strict append-only measurements, router-only recalibration at increasing replay sizes, and one matched local mini-MoE baseline. Every report requires disjoint specialization-interior, boundary, and mixed-input evaluation rows plus strictly nested router-training subsets.

The result is a deterministic grid of observed performance and regret gaps. It preserves RTR-004 and TRN-003 provenance, but it does not collect a dataset, train a new conventional MoE, render a plot, or claim that more data must improve results monotonically.

How it works

The evaluation workflow keeps measurement separate from the operation being measured:

Caller executes a fixed model, policy, registry, or training workflow
  → capture aligned outputs, routes, snapshots, losses, or metric blocks
  → validate identity, shape, provenance, and immutability contracts
  → calculate one narrowly defined family of measurements
  → return an immutable or JSON-compatible report

This boundary makes synthetic fixtures useful for contract testing without confusing them with production benchmark evidence. Callers remain responsible for representative data, identical runtime settings, meaningful work units, and any statistical analysis beyond the provided deterministic aggregates.

Mathematical intuition

Minimal usage

Compare one converted output and then inspect the work implied by a route:

equivalence = assert_tensors_equivalent(
    converted.logits,
    dense.logits,
    name="logits",
    atol=0.0,
    rtol=0.0,
)
work = measure_active_mlp_metrics(route_decision)
assert equivalence.exact
print(work.mean_active_mlps_per_token)

The caller supplies the two aligned tensors and a completed route decision. Neither evaluator executes or changes the model.

Verified implementation details

with torch.no_grad():
    dense = model(input_ids=input_ids, output_hidden_states=True)

Qwen3Backend().convert(
    model,
    AnchoredMoEConfig(converted_layers="upper_half"),
)
with torch.no_grad():
    converted = model(input_ids=input_ids, output_hidden_states=True)

assert_tensors_equivalent(converted.logits, dense.logits, name="logits")
for index, (actual, expected) in enumerate(
    zip(converted.hidden_states, dense.hidden_states, strict=True)
):
    assert_tensors_equivalent(actual, expected, name=f"hidden_states[{index}]")

installed = registry.snapshot()
registry.unload(code.identity)
removed = registry.snapshot()
registry.install(ExpertPackage.from_directory(code.package_directory))
reinserted = registry.snapshot()

evaluation = evaluate_registry_lifecycle(
    installed,
    removed,
    reinserted,
)
assert evaluation.exact

compositions = evaluate_registry_compositions((math, code))
assert compositions.subset_count == 4
assert compositions.trial_count == 5

quality = compare_top1_top2_to_oracle(
    losses,
    top1_result.selected_routes,
    top2_result.selected_routes,
)
top1_work = measure_active_mlp_metrics(top1_result.selected_routes)
top2_work = measure_active_mlp_metrics(top2_result.selected_routes)

assert top1_work.mean_active_mlps_per_token == 1.0
assert top2_work.mean_active_mlps_per_token > 1.0
print(quality.mean_top1_regret, quality.mean_top2_regret)

Current limitations

• Equivalence tests currently establish exact CPU behavior for local tiny Qwen3 fixtures. Production checkpoints, optimized kernels, accelerators, and distributed runtimes need separately declared compatibility tolerances.
• Route comparison covers final candidate decisions, not cross-layer trajectories, first-divergence depth, serialized route histories, or distributed aggregation.
• Counterfactual quality evaluation consumes losses measured elsewhere. It does not execute every candidate through the model or estimate an unevaluated expert pair.
• Efficiency reports exclude optimizer state, activations, allocator cache, serialized files, and energy use. CPU and MPS runtime peak memory are reported as unavailable.
• Registry composition is exhaustive only within the caller-approved trial bound. It does not persist reports, parallelize trials, or define the behavior artifact a caller should hash.
• Router-data scaling aggregates completed measurements. The current synthetic acceptance fixture is not evidence that router-only recalibration matches a jointly trained production MoE.
• Confidence intervals, statistical benchmark comparisons, production hardware baselines, and representative public datasets remain future work.

References

The architecture paper and repository blueprint listed in the page sources define exact fallback, route stability, lifecycle invariants, counterfactual regret, top-1/top-2 comparison, and the router-subset hypothesis test.

PyTorch numerical accuracy motivates explicit tolerances for future optimized runtime variants. The timing, synchronization, tensor-storage, and allocator APIs used by EVAL-005 are standard PyTorch infrastructure rather than a paper-derived benchmark method.

No paper-derived evaluation algorithm is claimed. EVAL-001 through EVAL-007 operationalize repository invariants using exact comparisons, standard finite permutations, synchronized timing, SHA-256 digests, and explicitly supplied experimental measurements.

Related pages

• Configuration and modeling establishes the exact dense fallback that evaluation checks.
• Policies defines the top-1, top-2, and adaptive decisions used by quality and active-work reports.
• Registry defines the immutable snapshots, lifecycle, and package ordering used by composition evaluation.
• Inference explains the sparse execution and route telemetry that evaluation observes.