The Fine-tuning Leaderboard
Detailed performance benchmarks for the most popular small language models fine-tuned across 30 tasks and compared to commercial models like GPT-4
About the Fine-Tuning Leaderboard
Why
Developers are no longer constrained to costly commercial models like GPT-4. High-performing SLMs offer an attractive alternative, enabling organizations to reduce costs and own model IP. Through fine-tuning, smaller open-source models can deliver GPT-like performance. This leaderboard helps enterprise AI teams select the best open-source model for their specific applications and needs.
What
The Fine-Tuning Leaderboard is a comprehensive resource that showcases the performance of the latest Small Language Models (SLMs) like Llama 3.1, Solar 10.7B, and Mistral. It compares these models against each other and OpenAI baselines across 30 diverse tasks.
How
We conducted over 700 LLM fine-tuning experiments in April 2024 and documented the results in our Arxiv research paper. We continually update the Fine-Tuning Leaderboard with new SLMs, evaluating them across 30 diverse tasks. The results are published in a comparative table, providing valuable insights into the evolving landscape of language models.
Model Name | Average Score (30 datasets) | Base Model | Number of Parameters | Family | Alignment | % Improvement over GPT4 |
|---|---|---|---|---|---|---|
| Solar Pro Preview | 0.798 | Fine-Tuned | 22.1B | Solar | Instruct | 21.83% |
| Meta-Llama-3.1-8B-Instruct | 0.782 | Fine-Tuned | 8.03B | Llama | Instruct | 19.39% |
| Solar Mini | 0.781 | Fine-Tuned | 10.8B | Solar | Instruct | 19.24% |
| Meta-Llama-3-8B | 0.771 | Fine-Tuned | 8B | Llama | Pretrained | 17.71% |
| Meta-Llama-3.1-8B | 0.769 | Fine-Tuned | 8.03B | Llama | Instruct | 17.40% |
| Phi-3-mini-4k-instruct | 0.768 | Fine-Tuned | 3.82B | Phi | Instruct | 17.25% |
| Meta-Llama-3-8B-Instruct | 0.761 | Fine-Tuned | 8B | Llama | Instruct | 16.18% |
| HuggingFaceH4/zephyr-7b-beta | 0.756 | Fine-Tuned | 7B | Zephyr | Instruct | 15.42% |
| Mistral-7B-v0.1 | 0.745 | Fine-Tuned | 7B | Mistral | Pretrained | 13.74% |
| Mistral-7B-Instruct-v0.1 | 0.737 | Fine-Tuned | 7B | Mistral | Instruct | 12.52% |
| Llama-2-7b-chat-hf | 0.726 | Fine-Tuned | 7B | Llama | Instruct | 10.84% |
| Mistral-7B-Instruct-v0.3 | 0.722 | Fine-Tuned | 7B | Mistral | Instruct | 10.23% |
| Mistral-7B-v0.3 | 0.720 | Fine-Tuned | 7B | Mistral | Pretrained | 9.92% |
| Llama-2-7b-hf | 0.715 | Fine-Tuned | 7B | Llama | Pretrained | 9.16% |
| phi-2 | 0.687 | Fine-Tuned | 2.7B | Phi | Pretrained | 4.89% |
| gemma-2b | 0.677 | Fine-Tuned | 2B | Gemma | Pretrained | 3.36% |
| gemma-7b-it | 0.667 | Fine-Tuned | 7B | Gemma | Instruct | 1.83% |
| gemma-2b-it | 0.661 | Fine-Tuned | 2B | Gemma | Instruct | 0.92% |
| GPT-4 | 0.655 | Base Model | 1760B | OpenAI | Pretrained | 0.00% |
| gemma-2-27b-it | 0.654 | Fine-Tuned | 27.2B | Gemma | Instruct | -0.15% |
| gemma-7b | 0.652 | Fine-Tuned | 7B | Gemma | Pretrained | -0.46% |
| GPT-4o-mini | 0.617 | Base Model | 8B | OpenAI | Pretrained | -5.80% |
| GPT-3.5-Turbo | 0.596 | Base Model | 175B | OpenAI | Pretrained | -9.01% |
| Phi-3.5-mini-instruct | 0.578 | Fine-Tuned | 3.82B | Phi | Pretrained | -11.76% |
| Solar Mini | 0.543 | Base Model | 10.8B | Solar | Instruct | -17.10% |
| gemma-2-27b | 0.534 | Fine-Tuned | 27.2B | Gemma | Pretrained | -18.47% |
| Phi-3-mini-4k-instruct | 0.523 | Base Model | 3.82B | Phi | Instruct | -20.15% |
| Mistral-7B-Instruct-v0.3 | 0.509 | Base Model | 7B | Mistral | Instruct | -22.29% |
| Mistral-7B-Instruct-v0.1 | 0.472 | Base Model | 7B | Mistral | Instruct | -27.94% |
| Meta-Llama-3.1-8B-Instruct | 0.461 | Base Model | 8.03B | Llama | Instruct | -29.62% |
| Meta-Llama-3-8B-Instruct | 0.441 | Base Model | 8B | Llama | Instruct | -32.67% |
| Phi-3.5-mini-instruct | 0.431 | Base Model | 3.82B | Phi | Pretrained | -34.20% |
| gemma-7b-it | 0.389 | Base Model | 7B | Gemma | Instruct | -40.61% |
| gemma-2-27b-it | 0.382 | Base Model | 27.2B | Gemma | Instruct | -41.68% |
| Solar Pro Preview | 0.379 | Base Model | 22.1B | Solar | Instruct | -42.14% |
| Llama-2-7b-chat-hf | 0.378 | Base Model | 7B | Llama | Instruct | -42.29% |
| HuggingFaceH4/zephyr-7b-beta | 0.360 | Base Model | 7B | Zephyr | Instruct | -45.04% |
| gemma-2b-it | 0.336 | Base Model | 2B | Gemma | Instruct | -48.70% |
| Meta-Llama-3.1-8B | 0.312 | Base Model | 8.03B | Llama | Instruct | -52.37% |
| Mistral-7B-v0.3 | 0.307 | Base Model | 7B | Mistral | Pretrained | -53.13% |
| phi-2 | 0.282 | Base Model | 2.7B | Phi | Pretrained | -56.95% |
| Meta-Llama-3-8B | 0.276 | Base Model | 8B | Llama | Pretrained | -57.86% |
| Mistral-7B-v0.1 | 0.273 | Base Model | 7B | Mistral | Pretrained | -58.32% |
| Llama-2-7b-hf | 0.260 | Base Model | 7B | Llama | Pretrained | -60.31% |
| gemma-2-27b | 0.236 | Base Model | 27.2B | Gemma | Pretrained | -63.97% |
| gemma-7b | 0.192 | Base Model | 7B | Gemma | Pretrained | -70.69% |
| gemma-2b | 0.149 | Base Model | 2B | Gemma | Pretrained | -77.25% |
Learn More
We hope you enjoyed the Fine-tuning Index and find it helpful as you train your own task-specific LLMs. We plan to update with new models so check back periodically.
All of the models were fine-tuned for less than $8 and served in production using Predibase and LoRAX. Below are some resources to help you fine-tune models that outperform GPT-4.
![Read the Arxiv Paper]()
Read the Arxiv Paper
Deep dive on all of the technical details and results from our 300+ fine-tuning experiments with our research paper.
![Watch the Webinar]()
Watch the Webinar
Watch this on-demand session for a technical session to learn how we fine-tuned LLMs to rival GPT-4 and lessons learned.
![Free Fine-tuning Credits]()
Free Fine-tuning Credits
Sign-up for $25 in free credits on Predibase to fine-tune and serve your own models in production.
