Llama 3 was quite a good publicity stunt for Meta

"Llama 3 was quite a good publicity stunt for Meta! The Llama experiment feels like a well-documented attempt to catch up to OpenAI and Google when it comes to the GenAI game. It is open, so Meta is back in the ""good guy"" category in the eye of the public!

Llama 1 was their pre-trained model (https://lnkd.in/g3nGaGms). It came out in February 2023, and it was the equivalent of the OpenAI's GPT-3 that came out in 2020. It felt like they were testing the water and establishing the foundation for future work. They used 1 trillion tokens, a memory-efficient attention mechanism and focused on smaller architectures than the competition.

Llama 2 was their LLM trained to follow instructions (https://lnkd.in/gPhhSYND). It came out in July 2023, and it was the equivalent of OpenAI's InstructGPT and the original model used by ChatGPT in 2022. It is the same model as Llama 1 fine-tuned with supervised learning and reinforcement learning. They focused on high-quality data for the fine-tuning steps and increased the pre-training step to 2 trillion tokens.

Llama 3 is basically the same model as the previous 2. They used more data for the pre-training step (15 trillion tokens) and again focused on the quality of the data for the different training steps. They used a grouped query attention mechanism and expanded their token vocabulary from 32K to 128K. I guess that makes sense, as with more data, there is less of a need to regularize. Now, the models are fine-tuned with LoRA, and they came up with their own fine-tuning library (torchtune https://lnkd.in/gDbxHkp3) as they developed Llama 3.

Meta caught up to OpenAI and Google by establishing their own foundational technology when it comes to training LLMs with a focus on smaller and open-sourced models. My guess is this was just a warm-up for the real work to come. "

Author

ABN ASIA was founded by people with deep roots in academia, with work experience in the US, Holland, Hungary, Japan, South Korea, Singapore, and Vietnam. ABN Asia is where academy and technology meet opportunity. With our cutting-edge solutions and competent software development services, we're helping businesses level up and take on the global scene. Our commitment: Faster. Better. More reliable. In most cases: Cheaper as well.

Feel free to reach out to us whenever you require IT services, digital consulting, off-the-shelf software solutions, or if you'd like to send us requests for proposals (RFPs). You can contact us at contact@abnasia.org. We're ready to assist you with all your technology needs.

© ABN ASIA