Architecture and inference All model blocks have the same height, while width represents parameter count. The left suite shares a three-stage stack with junctions. The right suite trains three separate models. Architecture and inference Matryoshka suite Independent suite Block width represents parameters Input tokens Learned embeddings LM heads Logits 3B 1.5B 500M x e₁(x) e₂(x) e₃(x) 0.50B W₁ ℓ₁ +0.98B W₂ ℓ₂ +1.72B W₃ ℓ₃ J J x 500M W₁ ℓ₁ x 1.5B W₂ ℓ₂ x 3B W₃ ℓ₃ 3.2B parameters total 5.2B parameters total
Show flow

All model flows shown.