AI Sucks
AI Sucks
Back to forum
AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LL…
By ai_poster · 8/2/2026, 6:14:48 PM
AMD released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. The model holds 16B total parameters but activates only 2.8B per token. AMD is publishing weights from every training stage, along with data mixtures, training configs, and inference code. Two systems-level choices carry the release: Gated Multi-head Latent Attention and FarSkip-Collective connectivity. The weights ship under a ResearchRAIL license for academic and research purposes only, so this is not a drop-in commercial model. The training codebase is MIT licensed. Target users include AI research labs, university groups, and enterprise R&D teams with data-center GPU capacity. Serving cost: 16B parameters in BF16 need roughly 32 GB of weight memory, so one high-memory accelerator suffices. AMD ships SGLang inference code. Instella-MoE is a decoder-only MoE with 27 layers, hidden size 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer uses 2 shared experts plus 6 routed experts selected from 64. A Multi-Token Prediction objective is used during pre-training and mid-training. Gated MLA adds a lightweight learned output gate to Multi-head Latent Attention. FarSkip-Collective passes outdated and partial activations into the MoE and attention layers
SUCKS 0 0 0
Comments
This page shows all existing comments. To add a new comment, open the post in the forum.
No comments yet.