Meta’s Llama 3.1: The Reality of Running a 405B Parameter Model at Home

Key Takeaways

• Meta released LLaMA 3.1 on Tuesday, July 23.
• The latest iteration of the LLaMA family comes in 3 sizes, including a 405 billion-parameter model.
• However, running the open-source LLaMA 3.1 405B at home requires the right tech setup.

Continuing a tradition established with LLaMA 2, on Tuesday, July 23, Meta released LLaMA 3.1  in three different sizes, with the largest version of the AI model coming in at a massive 405 billion parameters (the variables a model learns during training).

So far, reception has been generally positive, with Meta being praised for LLaMA 3.1’s performance and for updating its license to allow a greater range of uses than before. But just because anyone can download the new model for free under a permissive license doesn’t mean they can run it at home.

Mark Zuckerberg: 405B Parameter Model A ‘Big Moment for Open-Source AI’

In a letter announcing the new AI models, Meta CEO Mark Zuckerberg called LLaMA 3.1 405B “the first frontier-level open-source AI,” referring to its superior size compared to other open-weight models.

He also touted significant performance improvements with each iteration of the LLaMA family, which is fast closing the gap with leading closed models from OpenAI:

“Last year, Llama 2 was only comparable to an older generation of models behind the frontier. This year, Llama 3 is competitive with the most advanced models and leading in some areas. Starting next year, we expect future Llama models to become the most advanced in the industry.” 

In an interview with Rowan Cheung, the Facebook boss expanded on his view that open-source AI isn’t just good for innovation but has important security benefits too.

“My own fear is that if we lock down development you end up in a world where you have a small number of companies plus all the adversaries who can steal the model are the only ones who have access,” he observed. Meanwhile, “all the startups, all the universities, all the individual hackers are left out.”

Llama 3.1 Hardware Requirements

Although the 200GB Llama 3.1 405B file might fit on many personal disk drives, most home computers aren’t set up for intensive AI workloads.

Nevertheless, with a little modification, do-it-yourself AI enthusiasts can run even the largest LLaMA model at home. 

One of the key players helping bring open-source AI to the masses is Exo Labs, which has created a way for users to run their own AI cluster at home with consumer-grade hardware. 

Demonstrating the technology in a video, Exo Labs’ Alex Cheema implemented an instance LLaMA 3.1 405B using nothing more sophisticated than two 128GB Macbook Pros. 

Lowering Barriers to Entry

For students, bedroom hackers, independent researchers and small startups, projects like Exo Labs’ clustering technology are an important counterpart to open-source models. Together, they can help democratize access to AI, significantly lowering technical and financial barriers to entry.

This emphasis on accessibility and experimentation lies at the heart of the open-source movement and has fueled generations of software innovation.

Zuckerberg himself used open-source tools like Linux, Apache, MySQL and PHP to build Facebook. Of course, his attempts to reinvent himself as an open-source champion might not go down well with critics of Meta’s business model. But compared to OpenAI and Google, which jealously guard their model weights and only let developers access them via paid-for APIs, LLaMA is a tinkerer’s dream.

Responding to Criticism

While Meta brands the LLaMA range as open-source, it has released the AI under a custom license which is more restrictive than traditional open-source software licenses.

However, after being criticized for imposing restrictions on researchers, the company has now updated its terms of use to allow developers to use the outputs from LLaMA to improve other models for the first time.

As well as lifting some restrictions on usage, Meta has released an in-depth research paper  and an extensive library  of documentation to equip developers with the knowledge to program their own LLaMA-based AI.

After all, open weights alone aren’t enough to justify the open-source label. If Meta is serious about making LLaMA the Linux of AI, it will need to foster an equivalent grassroots community to sustain the project.