Key Takeaways
Just a day after Meta released LLaMA 3.1, on Thursday, July 24, Mistral unveiled a new language model it says competes with Meta’s largest AI.
Head to head, Mistral Large 2 outperforms LLaMA 3.1 205B on several key benchmarks, especially those that measure models’ ability to generate code.
At 123 billion parameters, Mistral Large 2 sits between the medium and large-sized LLaMA models, yet it was better at certain tasks than the 205 billion parameter version of LLaMA 3.1.
While LLaMA 3.1 205B retains the lead in the crucial Massive Multitask Language Understanding (MMLU) benchmark, Mistral Large 2 pulls ahead in areas such as coding, mathematics and instruction following.
Considering its smaller footprint, Mistral’s model also performs well in multi-lingual text generation and reasoning.
Comparing the length of AI responses to different prompts, Mistral Large 2 output shorter answers than LLaMA 3.1, GPT-4o and Claude 3.
Emphasizing that shorter generations are more cost-effective for inference, Mistral said: “we spent a lot of effort to ensure that generations remain succinct and to the point whenever possible.”
This focus on conciseness highlights the growing emphasis on AI efficiency that has inspired the recent generation of smaller, more compact models.
While longer answers tend to improve scores on many benchmarks, Mistral Large is designed to perform well without being unnecessarily verbose, a principle that could help it gain traction with business users.
Benchmark | Meta-Llama 3.1 405B | Mistral Large 2 |
MMLU
MATH MBPP Base HumanEval Code |
87.3%
73.8% 88.6% 89% |
84.0%
71.5% 80% 92% |
The rivalry between Mistral Large 2 and LLaMA 3.1 highlights the rapid advancements and competitive nature of open-source AI development. In terms of performance, for the first time, open-source AI poses a viable alternative to proprietary models offered by Google and OpenAI.
Both models have been made freely available for research purposes. However, Mistral and Meta diverge in their commitment to open-sourcing their AI.
While Meta has moved toward more permissive licenses, since partnering with Microsoft, Mistral has gone in the opposite direction, restricting commercial applications of its latest releases to paying customers.