MoE vs dense — what's actually different?

TL;DR. A dense model roughly uses its whole parameter stack for every token. A Mixture-of-Experts (MoE) model can advertise a huge total "B" while only waking a slice of specialist networks per token. That is why two models with similar headline sizes can feel very different on cost, latency, and hardware. Ask for both total and active size — then test on your work.

Dense, in plain language

Think of a dense model as one big team that always shows up together.

When the model writes the next word, most of those billions of parameters take part. The "70B" on the slide is therefore a decent rough guide to how much work happens at answer time (and, usually, how expensive it is to serve).

Useful shorthand from our sibling post: the B is capacity, not IQ. Dense vs MoE is about how that capacity is wired, not who wins a beauty contest.

MoE, in plain language

Mixture-of-Experts is more like a hospital with many specialists and a triage desk.

• The building holds a large total staff (total parameters — the headline B).
• For each patient (each token), triage calls only a few specialists (active experts / active parameters).
• The rest of the experts stay on call but do not all scrub in for every case.

So a model can honestly say "trillions of parameters" while only running a much smaller active set per token. The headline B is still real — it is the library of experts you keep resident — but it is not the same as "everything fires every time."

You will sometimes see both numbers on a careful card: e.g. total size vs active size per token. Prefer that honesty over a single brag number.

Why buyers should care

Question	Dense (roughly)	MoE (roughly)
What does the big "B" mean?	Close to what runs each token	Often total experts — active work can be smaller
Cost to answer	Scales with that full stack	Can be lower active FLOPs — but weights may still need a lot of memory
Ops feel	Simpler mental model	Extra routing; serving large sparse models needs real engineering
Slide risk	Overselling intelligence via size	Overselling "huge B" without saying what activates

Practical buyer questions:

1. Total vs active — Do you publish both? What activates per token?
2. Memory vs compute — Even if few experts fire, do we still need enough GPU memory to hold the full weight set?
3. Latency — Does routing + expert hops help or hurt our response-time budget?
4. Our work — Run a short eval on our documents and workflows. Architecture labels do not replace that.

Sparse MoE designs have been loud in mid-2026 open-model headlines (very large total parameter counts with sparse expert activation). Treat the news as a prompt to ask better questions — not as a reason to skip evaluation.

A calm way to read the next MoE slide

Try this line in the room:

"Interesting total size. What's the active parameter count per token, what hardware does serving need, and how does it score on our tasks?"

If those answers are clear, MoE vs dense becomes useful engineering context. If they are fuzzy, the bigger total B is mostly marketing paint — the same trap as treating any "B" as an IQ badge.

How this connects to what we build

At AiHPC we care about fit-for-purpose models under governance more than winning an architecture fashion contest. OrchAI is built so organisations can run AI on their own documents and workflows — with citations, control, and evaluation — whether the underlying model is dense or MoE. The wiring is a dial. The trust layer is the product.

Curious how that looks in practice? Talk to us or try the demo.

Frequently asked questions

What is a dense model? A model where (to a first approximation) the full parameter stack participates when producing each next token. The headline B is a fair rough guide to runtime work.

What does MoE mean? Mixture-of-Experts: many specialist sub-networks, with a router that wakes only a few per token. Total "B" can dwarf active "B."

Is MoE always cheaper or better? No. It can reduce active compute for a given total size, but memory, routing, and ops still cost. Judge on your tasks, latency, and hosting constraints.