Why combining several models — from Random Forests to LLM 'councils' like mixture-of-agents — beats a single model, how it works, and when the extra cost isn't worth it.
A single model, however large, has a single point of view — one architecture, one training run, one set of blind spots. Ensembling is the practice of combining several models so their independent mistakes cancel out and their independent strengths add up. It is one of the oldest and most reliable ways to squeeze more accuracy out of machine learning, and it has re-emerged in the age of large language models under a new name: the AI council, where several models (or several personas of one model) deliberate, vote, or critique each other before returning a final answer.
This guide explains why ensembles work, the classic patterns that dominate tabular machine learning, how the same idea now applies to LLMs, and — just as important — when the extra cost is not worth it.
An ensemble is a group of models whose predictions are combined into one. The intuition is the wisdom of crowds: if you ask enough reasonably-informed, independent people to estimate something, the average of their guesses is usually closer than most individuals.
The key word is independent. Ten copies of the same model make the same mistakes, so averaging them changes nothing. Ensembles only help when the members' errors are decorrelated — when they are wrong in different ways. Formally, an ensemble reduces the variance component of error (and, for some methods, the bias component) without the members having to individually improve.
A rough but useful mental model is the Condorcet Jury Theorem: if each voter is better than a coin flip and votes independently, the probability that the majority is correct rises toward certainty as you add voters. The catch is the same one that haunts every ensemble in practice — the "independent" assumption is rarely fully true, and correlated voters cap the benefit.
Decades of tabular ML converged on three families, and they are still the first thing a practitioner reaches for.
Bagging — bootstrap aggregating — trains many copies of the same algorithm on different random samples (with replacement) of the training data, then averages their outputs. Each model sees a slightly different world, so their errors decorrelate, and averaging shrinks variance.
The Random Forest (Breiman, 2001) is bagging applied to decision trees with one extra twist: at each split, each tree may only consider a random subset of features. That forced disagreement makes the trees more independent and the forest more accurate. Random Forests remain a superb default: hard to overfit, minimal tuning, and they run in parallel.
Where bagging builds members in parallel, boosting builds them in sequence, each new model focusing on the examples the previous ones got wrong. AdaBoost (Freund & Schapire) introduced the idea; gradient boosting (Friedman) generalised it to fit each new model to the residual errors of the ensemble so far.
Modern gradient-boosted tree libraries — XGBoost, LightGBM, CatBoost — are the workhorses of tabular prediction and still win a large share of Kaggle competitions on structured data. Boosting attacks bias as well as variance, which is why it often edges out Random Forests on accuracy — at the cost of being easier to overfit and harder to tune.
Stacking (Wolpert, 1992) trains a second-level model — a meta-learner — to combine the predictions of several diverse base models. Instead of a fixed average or vote, the meta-learner learns how much to trust each base model, and in which situations. Done carefully (with out-of-fold predictions to avoid leakage), stacking usually beats any single member and often beats simple averaging. The winning entries of the famous Netflix Prize were enormous stacked ensembles.
The same principle now applies to large language models, and this is where the term AI council comes from: rather than trusting one model's single forward pass, you convene several and aggregate.
Why does convening models help? The same reason bagging helps: different architectures, training data, and prompts produce decorrelated errors. One model hallucinates a citation; another catches it. One is strong at code, another at prose. Aggregation keeps the agreements and filters the idiosyncratic mistakes.
It comes down to error decomposition. A model's expected error splits into bias (systematic wrongness), variance (sensitivity to the particular training sample or random seed), and irreducible noise. Averaging many decorrelated models leaves bias roughly unchanged but drives variance down — the more members and the less correlated they are, the larger the drop. Boosting goes further and reduces bias by explicitly fitting what the ensemble still gets wrong.
The single most important lever is diversity. An ensemble of near-identical members barely beats one member. An ensemble of genuinely different members — different algorithms, different data, different prompts or models — can beat every one of them. If you remember one thing, remember that accuracy comes from decorrelated errors, not from sheer numbers.
Ensembling is not free, and more members is not automatically better.
Reach for an ensemble when:
Think twice when:
If you decide a council is warranted, a few patterns keep it effective and affordable:
Ensembles and AI councils are among the most dependable accuracy upgrades in machine learning because they exploit a simple truth: independent mistakes cancel. On tabular data, that means Random Forests, gradient boosting, and stacking. With language models, it means self-consistency, mixture-of-agents, debate, and routing. In every case the gain comes from diversity of errors, not from the raw count of models — and it is always paid for in cost, latency, and complexity. Convene a council when the accuracy is worth that price, keep the members genuinely different, and always measure the whole against its parts.
Bagging trains many models in parallel on different random samples of the data and averages them, which mainly reduces variance (Random Forests are the classic example). Boosting trains models in sequence, each one focusing on the mistakes of the last, which reduces bias as well — gradient-boosted trees like XGBoost and LightGBM are the common form. Boosting often edges out bagging on accuracy but is easier to overfit and harder to tune.
No. An ensemble only helps when its members make different (decorrelated) mistakes. Several near-identical models add cost with almost no accuracy gain, and on questions every member fails, voting simply cements the wrong answer. Councils win when members are genuinely diverse and each is better than chance on the task.
It is an ensemble of large language models. Instead of trusting one model's single answer, several models (or several samples of one model) draft answers and are then combined — by majority vote, by a judge model, or by another layer of models that synthesises the drafts. Techniques include self-consistency, mixture-of-agents, and multi-agent debate.
Accuracy gains come from errors cancelling out. If every member is wrong in the same way, averaging or voting changes nothing. Diversity — different algorithms, training data, prompts, or model providers — is what makes members wrong in different ways, so the ensemble keeps the agreements and filters the idiosyncratic mistakes.
When latency or per-request cost is the binding constraint (a council of five LLMs can cost and take several times more than one call), when the single model is already near the accuracy ceiling, when members are highly correlated, or when interpretability and maintainability matter — a stacked ensemble is much harder to run and debug in production.
RTI templates, FIR scripts, real escalation ladders — the same kind of thing you just read. Sundays only. No spam.
We don't share your email. Unsubscribe any time.
See a kid working at a dhaba or factory? Here is how to use Childline 1098 and the PENCIL portal to report child labour and ensure they get back to school.
Tired of WhatsApp university? Learn how to use the IT Rules 2021 and the GAC portal to hold social media platforms accountable for misinformation and dangerous fake news.
Applying for college or a government job? Learn how to navigate domicile rules, gather the right documents, and get your SC, ST, or OBC certificate without the headache.
Delivering food or fixing pipes via apps? The Code on Social Security 2020 changes your legal status. Here is how to register for benefits and what you are owed.