>
M
Model Details
Domain:
Task:
Model Access:
Open weights (unrestricted)
AI Tools for You
Introduction
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion parameters activated for each token. Each sparse MoE layer contains 384 routed experts, while only eight are selected for each token. This fine-grained sparsity provides substantial expert capacity while limiting the expert computation required for each token. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert-Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. A collection of expert-balancing and numerical-stabilization techniques supports stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open-weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.
Benchmarking
FLOPs1.0999999999999999e+24
Notes: 1.1e24 FLOP reported by Motif Technologies in a spreadsheet provided via email (August 2026). Note an internal inconsistency in the developer-reported hardware figures: 768 B200 GPUs for ~100 days (~2,400 hours) at the claimed 18.3% MFU would imply ~2.7e24 FLOP at BF16 peak throughput; 1.1e24 FLOP over that duration implies ~7% MFU. The token-based cross-check supports the 1.1e24 figure, so the MFU claim was not recorded.
Training
Training Code AccessibilityReleased under the MIT license on Hugging Face (the earlier Motif-3-Beta preview carried a non-commercial license; the full release is MIT). Training code: only reference implementations of the training-system optimizations are public (motif3-training-example repo); the full in-house TorchTitan-based training stack is not released. An NVFP4-quantized inference checkpoint and a vLLM fork/Docker image are also provided.
HardwareNVIDIA B200
Hardware Quantity768
Size Notes: ~12.5 trillion pretraining tokens, measured with the Motif tokenizer (technical report). Motif's SuperBPE tokenizer has higher compression than comparison tokenizers; the report notes the same corpus corresponds to >15T tokens in comparison tokenizers. The developer-provided spreadsheet states 13T tokens. Post-training (SFT + RL teachers + distillation) adds more data.
Parameters
Parameters314000000000
Notes: Fine-grained sparse MoE: approximately 314B total parameters, ~13.2B activated per token. 53 Transformer layers (2 dense + 51 MoE); each MoE layer has 384 routed experts + 1 shared expert, with top-8 routing. Grouped Differential Latent Attention (GDLA), modified manifold-constrained hyper-connections, Expert-Specific PolyNorm, and a 1-layer MTP head. Trained from scratch.
Authors
Joon Son Chung, Sungmin Lee, Junghwan Lim (technical and management leadership); Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh (core contributors); see the technical report Contributions section for the full list