Understanding LLM Leaderboards: metrics, benchmarks, and why they matter

Toloka Team

Nov 1, 2023

Essential ML Guide

Toloka's new LLM Leaderboard: Finding the best model for your business

As AI advances across industries, LLMs remain key catalysts of its progress. Practitioners are developing countless LLMs for various applications in various domains. But with so many models available, how do we choose the right one for our use case?

The answer is LLM leaderboards.

What is an LLM leaderboard?

An LLM leaderboard is a ranking platform that compares the performances of different large language models. It is a contest-like environment where models are ranked for tasks like code generation, question answering, summarization, and general understanding.

LLM leaderboards are essential because they aim to provide unbiased and uncontaminated results by comparing models side-by-side using predefined criteria. These leaderboards help users and developers learn which model is right for them through transparent and objective model evaluations.

There are different types of leaderboards. Some assess general LLM skills, such as how relevant and coherent their responses are and whether they follow instructions. Others focus on domain-specific LLMs and their performance in specialized tasks pertinent to the field. Several platforms prioritize the practical use of LLMs and rank them based on speed and cost.

Leaderboards also address issues like hallucinations, bias, and toxicity, much-needed factors for ensuring responsible use of LLMs.

This article explores how leaderboards test and rank models, along with examples of the most popular ones.

How to decide which model is the best?

Leaderboards use different methods to grade LLMs, which generally include:

Quantitative metrics, usually accompanied by a minimum threshold that defines if a model is "good" at the task in hand or,
A relative comparison against other models using human evaluation.

A leaderboard tests the models using standardized criteria. Comprehensive evaluations ensure a fair comparison and help users learn which models achieve higher accuracies, lower costs, and turbo speeds. The criteria can vary from task to task based on required capabilities, price point, scale, and field of use. Most leaderboards include metrics that analyze these aspects in some way. Let's discuss some of the most common ones.

Accuracy is a simple but key concept. In tasks like classification, we want to know if the model produces the correct outputs and use accuracy to measure its success. F1 score is often used as an alternative to accuracy. It tells us how well the model balances precision and recall, which is especially valuable in imbalanced classification problems.
Speed-related metrics are crucial for real-time applications like chat. Therefore, model ranking platforms usually include data on how quickly a model responds or how many tokens it generates per second.
Then there's perplexity, a score that shows a model's predictive capabilities. A lower perplexity score means the model is better at generating coherent and logically structured text, essential for tasks like text generation, machine translation, and summarization.
Other metrics measure contextual relevance and semantic understanding. For example, the BERT score indicates how accurately the model represents the semantics of a reference text. Such evaluations are commonly used in text summarization or translation tasks.
Ethical and responsible AI is getting more attention and requires metrics addressing this aspect of LLMs. These metrics are crucial for real-world applications to ensure the model complies with law regulations and won't provide unethical content.
Specific leaderboards also evaluate a model's likelihood of hallucinating and whether its output is correct and based on a verifiable source.

These are some frequently used metrics, often combined in an overall score of a model's abilities.

The choice of metrics depends on what we seek in a model. Still, regardless of the use case, metrics must accurately assess the required capabilities of the LLMs while still being:

Simple enough to clearly understand what they measure,
Reliable to ensure consistency and invariant outcomes and
Relevant to the model's purpose.

How do we make sure the results are objective?

As many LLMs differ in nuances, selecting the best model can be challenging.

To minimize bias and assure fairness, we evaluate the models against designated standards - benchmarks. These consist of problems, questions, and tasks with ground-truth answers. The LLMs' outputs are scored based on their similarity to the expected answer. Depending on the benchmark, we can assess the model's proficiency in code generation, problem-solving, and even emotional intelligence. Benchmarks are essential for fair comparison because they define clear and consistent rules for measuring performance, scale, and capabilities. Some benchmarks have established their place in the community for their holistic approach to measuring LLM performance. Let's look at some of the most popular benchmarks at the moment.

EQ-Bench tests the emotional intelligence of LLMs, which is still relatively overlooked on other platforms. This benchmark evaluates the models' ability to interpret social interactions by asking them how intense emotions are in different dialogues. The benchmark even strongly connects to other multi-domain datasets like MMLU, suggesting it evaluates similar aspects of general intelligence.
HumanEval is another well-known benchmark that assesses how accurately models generate code from docstrings. Practitioners utilize it to determine a model's programming knowledge and general approach to problem-solving.
Measuring Massive Multitask Language Understanding (MMLU) is one of the most comprehensive benchmarks for LLM evaluations. The HuggingFace Open LLM leaderboard uses MMLU to demonstrate how LLMs perform in different areas, including law and computer science. LLMs that score high on this benchmark are skilled at multitasking and reasoning across various domains.

Toloka is developing benchmarks for specialized domains like natural sciences and university-level math to reveal limitations and opportunities for LLMs.

Which are some top-caliber leaderboards?

The Open LLM Leaderboard, LMSYS Chatbot Arena, and Artificial Analysis LLM Leaderboard are among the leading LLM leaderboards due to their robust and innovative evaluation methods. These platforms use comprehensive datasets with real-world tasks and a mix of human and automated assessments to provide transparent and reliable rankings. Let's have a look at their evaluation target and methodology:

The Open LLM Leaderboard by Hugging Face is an established resource for insights into LLMs' problem-solving skills and commonsense reasoning. The leaderboard uses six Eleuther AI LM Evaluation Harness benchmarks to test LLMs on math, physics, and general knowledge. It even reveals the environmental impact by reporting data on the C2O emissions from the models' evaluations. This leaderboard is popular among developers as it allows anyone to evaluate their model and see how it competes with existing open-source models.
LMSYS Chatbot Arena is a model ranking platform with a more hands-on experience. It creates a contest-like environment where the models are ranked based on the popular Elo rating system used in chess. On this crowdsourced platform, users compare responses from two models to the same prompt and vote for the better one. They do this iteratively until a winner is decided. This leaderboard relies on human input to rank models in conversational tasks, reasoning, and context handling.
The Artificial Analysis LLM Leaderboard compares over 30 models, including GPT-4o, Gemini, Llama 3, and GPT-3.5 Turbo, on diverse metrics such as price, latency, and context window. The LLM API Providers Leaderboard compares over 100 LLM endpoints on the same metrics. These platforms help customers and users choose the right AI model and hosting provider for their use cases.

In addition to the above-listed leaderboards, there are several other honorable mentions. For example, the AlpacaEval Leaderboard assesses the LLMs' ability to follow user instructions. The MTEB leaderboard showcases the top text-embedding models based on the Massive Text Embedding Benchmark. As hallucinations remain a weak spot for LLMs, leaderboards like HHEM try to measure their resilience against generating false information. LLMs have become prevalent in programming and automation, and practitioners rely on platforms like CanAI Code, EvalPlus, and Berkley Function-Calling Leaderboard to find the top-tier programming models in the field.

Key takeaways

Leaderboards are the best way to stay informed about the rapidly evolving field of LLMs. They provide equal ground for evaluating models. A good leaderboard is fair, reliable, and practical. It utilizes standardized benchmarks and metrics that are easy to understand and focus on tasks applicable to real-world scenarios.

Another essential characteristic is transparency. These platforms clearly explain how models are evaluated and make it easy to reproduce results.

They also encourage innovation by presenting challenging problems for models to solve. A well-designed leaderboard is inclusive and accessible to participants from eclectic backgrounds. It remains relevant by consistently tracking model progress and updating evaluation methods.