What this index contains
This is the canonical bibliography for research ideas and reproducibility standards discussed in the Mutations documentation. Each entry links directly to its primary source, explains what the source motivates, and states where the repository intentionally differs.
Routine framework API documentation remains on the page where that API is used. This keeps the index focused on conceptual and algorithmic provenance rather than turning it into a dependency directory.
Expert representations
These papers motivate compact adaptations, fine-grained specialist structure, and retrieval from large collections of small experts.
Edward J. Hu et al. · 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv:2106.09685- Why it matters here
- Defines the frozen-weight low-rank update used for compact anchor-relative expert state.
- How Mutations differs
- Mutations injects LoRA into a private MLP copy and exposes an explicit anchor-relative delta behind its own immutable-anchor lifecycle.
Damai Dai et al. · 2024
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
arXiv:2401.06066- Why it matters here
- Motivates fine-grained expert segmentation and the separation of shared from routed computation.
- How Mutations differs
- Mutations partitions a pinned dense anchor for exact initialization and independently composes plug-ins rather than jointly training the DeepSeekMoE architecture.
Xu Owen He · 2024
Mixture of A Million Experts
arXiv:2407.04153- Why it matters here
- Defines split-query product-key search over a large expert identifier space.
- How Mutations differs
- Mutations implements one exact retrieval head over caller-projected queries and stops before query projection, selected-expert execution, aggregation, or package persistence.
Routing and composition
These sources cover sparse routing, fixed-capacity expert choice, route stabilization, parameter-space composition, and variable expert counts.
William Fedus, Barret Zoph, and Noam Shazeer · 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
arXiv:2101.03961- Why it matters here
- Motivates sparse routing comparisons and the assignment-fraction times mean-probability balancing objective.
- How Mutations differs
- Mutations retains permanent dense fallback and applies its balancing variant to package-local training rather than a Switch Transformer runtime.
Yanqi Zhou et al. · 2022
Mixture-of-Experts with Expert Choice Routing
arXiv:2202.09368- Why it matters here
- Motivates fixed expert token buckets, token-axis ranking, and variable expert counts per token.
- How Mutations differs
- Mutations returns standalone assignment metadata with explicit integer capacity and anchor fallback, without expert execution or distributed training.
Damai Dai et al. · 2022
StableMoE: Stable Routing Strategy for Mixture of Experts
arXiv:2204.08396- Why it matters here
- Motivates separating route discovery from a lightweight frozen deployment router.
- How Mutations differs
- Mutations distills fixed-zero-anchor assignments from an immutable composed teacher and keeps cross-layer switching as a separate deterministic rule.
Mohammed Muqeeth, Haokun Liu, and Colin Raffel · 2023
Soft Merging of Experts with Adaptive Routing
arXiv:2306.03745- Why it matters here
- Motivates constructing an input-dependent expert by combining parameters before execution.
- How Mutations differs
- Mutations provides a two-LoRA correctness backend driven by a frozen pair-weight head and does not reproduce SMEAR training.
Eric L. Buehler and Markus J. Buehler · 2024
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
arXiv:2402.07148- Why it matters here
- Motivates layerwise, token-dependent composition of frozen LoRA experts.
- How Mutations differs
- Mutations merges exactly one selected pair's effective updates and does not implement X-LoRA's scaling network or training lifecycle.
Quzhe Huang et al. · 2024
Harder Tasks Need More Experts: Dynamic Routing in MoE Models
arXiv:2403.07652- Why it matters here
- Motivates selecting a variable number of experts from routing confidence.
- How Mutations differs
- Mutations uses caller-supplied calibrated thresholds, deterministic ties, fixed-zero anchor fallback, and a hard per-token cap rather than a learned dynamic router.
Maryam Akhavan Aghdam, Hongpeng Jin, and Yanzhao Wu · 2024
DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
arXiv:2409.06669- Why it matters here
- Motivates allocating different expert counts according to estimated token importance.
- How Mutations differs
- Mutations exposes deterministic selection metadata and does not learn token importance, optimize a compute-allocation objective, or execute the selected set.
Training and calibration
These sources motivate teacher preservation, start-point regularization, temperature scaling, router fitting, and load balancing.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Distilling the Knowledge in a Neural Network
arXiv:1503.02531- Why it matters here
- Motivates temperature-scaled preservation against a fixed teacher distribution.
- How Mutations differs
- Mutations consumes precomputed immutable-anchor logits, supports token-level replay masks, and retains a separate causal-language-model objective.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger · 2017
On Calibration of Modern Neural Networks
arXiv:1706.04599- Why it matters here
- Motivates fitting a post-hoc temperature on held-out predictions.
- How Mutations differs
- Mutations calibrates independent binary utility heads and applies separate false-activation-constrained thresholds rather than reproducing multiclass calibration.
Xuhong Li, Yves Grandvalet, and Franck Davoine · 2018
Explicit Inductive Bias for Transfer Learning with Convolutional Networks
arXiv:1802.01483- Why it matters here
- Motivates regularizing trainable parameters toward their optimization starting point.
- How Mutations differs
- Mutations applies drift only to expert-owned parameters and normalizes the squared distance by the expert scalar count.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai · 2024
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
arXiv:2408.15664- Why it matters here
- Motivates expert-wise selection biases updated from historical load without auxiliary-loss gradients.
- How Mutations differs
- Mutations implements a standalone anchored primitive with explicit completed-batch updates and no distributed Trainer integration.
DeepSeek-AI et al. · 2024
DeepSeek-V3 Technical Report
arXiv:2412.19437- Why it matters here
- Documents auxiliary-loss-free expert balancing with routing biases at model scale.
- How Mutations differs
- Mutations provides a composable local balancing primitive and does not claim the report's distributed training or throughput results.
Arthur E. Hoerl and Robert W. Kennard · 1970
Ridge Regression: Biased Estimation for Nonorthogonal Problems
DOI:10.1080/00401706.1970.10488634- Why it matters here
- Motivates the positive L2 penalty used for deterministic affine router initialization.
- How Mutations differs
- Mutations leaves the intercept unregularized, solves layer-local multi-output utility targets together, and emits frozen router heads.
Standards and reproducibility
These specifications support deterministic artifact identities and canonical metadata. They are infrastructure references, not sources for a routing or training algorithm.
National Institute of Standards and Technology · 2015
FIPS PUB 180-4: Secure Hash Standard
DOI:10.6028/NIST.FIPS.180-4- Why it matters here
- Defines SHA-256, used for deterministic artifact identities and integrity checks.
- How Mutations differs
- Mutations uses the digest for reproducibility and integrity, not as a source of secret randomness.
A. Rundgren, B. Jordan, and S. Erdtman · 2020
RFC 8785: JSON Canonicalization Scheme
RFC 8785- Why it matters here
- Motivates deterministic JSON serialization before hashing.
- How Mutations differs
- Mutations defines a smaller project-specific canonical JSON contract and does not claim full RFC 8785 conformance.
Current limitations
This is a curated implementation bibliography, not an exhaustive survey of Mixture-of-Experts research. A paper appears here only when the repository documentation identifies a concrete implementation influence. Inclusion does not mean Mutations reproduces the paper’s architecture, training procedure, or reported results; those boundaries are stated on every entry.
Related pages
Start with Experts for parameterizations, Routers and Policies for selection and composition, or Training and Calibration for fitting workflows. The end-to-end examples show how those pieces are assembled.