MoE — Mixture of Experts

Fact

A model design that turns on only some parameters for each input.

In Plain Words

MoE is like a giant diner with many cooks. Order pancakes, and only the breakfast cooks jump in. The sushi chef stays seated.

It helps build huge neural networks without waking every parameter each time. You meet it in some LLMs during training and when they run.

Related Concepts

• Large Language Model
• Parameter
• GPU Compute
• Inference

LLM
MoE can be an LLM architecture for bigger models with better cost control.

Parameter
MoE may have many parameters, but one input turns on only some of them.

GPU
MoE tries to use GPU compute only on the most needed parts.

Inference
MoE changes compute during inference, so cost and speed can change.