Today is LlamaCon, the first official virtual conference for Meta’s open-source Llama models.

I don’t have any inside information about what will be announced, so I decided to ask the model itself. Using the Together AI playground, I asked Llama 4 Maverick 17B to predict some possible announcements and format them as bingo card labels, based on a few recent news articles.

Since this is Spreadsheets Are All You Need, I set it up in a spreadsheet. You can make a copy, ask your favorite model what might happen, and update the bingo card yourself.

So grab the popcorn, play along, and enjoy LlamaCon.

Update: Llama-as-Judge evaluates LlamaCon 2025

As a follow-up, I fed Llama4 transcripts from the event and asked it to judge how well its Bingo Card did. Unfortunately, despite Llama accurately predicting more than half the talking points, it failed to get a Bingo by a single square in multiple cases.

You can watch the video and read the transcript of the model's judgements below.

Disclosure notice: The content below was generated with AI.

Llama 4 Variants

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the first Bingo Card label "Llama 4 Variants" qualifies for being marked off.

The transcripts mention several variants of Llama 4, including:

Llama 4 Scout: a smaller model that runs on a single H100 and is best-in-class on several benchmarks for its size.
Llama 4 Maverick: a 17 billion active parameter model used in production for several Meta products, including Meta AI.
Llama 4 Behemoth: a massive model, one of the largest out there, used as a teacher model to improve the performance of Maverick.
These announcements indicate that there are multiple variants of Llama 4, which directly corresponds to the Bingo Card label "Llama 4 Variants".

Therefore, the answer is: Yes, the Bingo Card label "Llama 4 Variants" can be marked off.

Multimodal Capabilities

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Multimodal Capabilities" qualifies for being marked off.

The transcripts mention several instances of multimodal capabilities:

Chris Cox mentions that Llama 4 is "massively multimodal" and supports multiple modalities, including text, images, and potentially other modalities.
Angela Fan and Manohar Paluri discuss how Llama 4 was designed to be natively multimodal, with a training recipe that interweaves different modalities, such as text and images.
The Llama 4 models are described as being able to take image input and send multiple images per prompt.
Chris Cox highlights the use of a perception encoder, a visual encoder that works well across many visual domains, which is used in tools like SAM 2 (object detection system for image and video).
These announcements and discussions indicate that Llama 4 has multimodal capabilities, which directly corresponds to the Bingo Card label "Multimodal Capabilities".

Therefore, the answer is: Yes, the Bingo Card label "Multimodal Capabilities" can be marked off.

Agentic AI

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Agentic AI" qualifies for being marked off.

The transcripts mention several instances related to Agentic AI:

Mark Zuckerberg and Ali Ghodsi discuss the concept of "agents" and how they are being used to automate tasks, such as customer support and coding.
Ali Ghodsi mentions that one of the big trends is the development of "agents" that can navigate the web and use computers.
Mark Zuckerberg talks about building "coding agents" and "AI research agents" that can advance Llama research.
Satya Nadella discusses the concept of "agents" and how they are being used in Microsoft's products, such as GitHub Copilot, to automate tasks and improve productivity.
These discussions and mentions indicate that Agentic AI is a topic of interest and discussion at the LlamaCon event, and that companies like Meta and Microsoft are working on developing and integrating agentic AI capabilities into their products.

Therefore, the answer is: Yes, the Bingo Card label "Agentic AI" can be marked off.

Fine-tuning APIs

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Fine-tuning APIs" qualifies for being marked off.

The transcripts mention several instances related to Fine-tuning APIs:

Manohar Paluri and Angela Fan discuss the Llama API, which allows developers to fine-tune Llama models for their specific use cases.
Angela Fan explains that the Llama API provides a fine-tuning tab where developers can upload data, configure hyperparameters, and fine-tune their models.
The Llama API is described as allowing developers to take their custom models with them wherever they want to host them, without being locked into Meta's servers.
The news article "Everything we announced at our first-ever LlamaCon" also mentions that the Llama API provides tools for fine-tuning and evaluation, allowing developers to customize and run their models more efficiently.

These announcements and discussions indicate that the Llama API provides fine-tuning capabilities, which directly corresponds to the Bingo Card label "Fine-tuning APIs".

Therefore, the answer is: Yes, the Bingo Card label "Fine-tuning APIs" can be marked off.

Nvidia Partnership

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Nvidia Partnership" qualifies for being marked off.

The news article "Everything we announced at our first-ever LlamaCon" mentions that Meta has collaborated with partners to build Llama Stack distributions, including a recently announced integration of Llama Stack with NVIDIA NeMo microservices.

This announcement indicates a partnership between Meta and Nvidia, which directly corresponds to the Bingo Card label "Nvidia Partnership".

Therefore, the answer is: Yes, the Bingo Card label "Nvidia Partnership" can be marked off.

GPU Roadmap

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "GPU Roadmap" qualifies for being marked off.

Upon reviewing the transcripts and news articles, I did not find any direct mention of a "GPU Roadmap". While there are discussions about hardware and infrastructure related to AI, such as the need for more compute power and the development of specialized hardware like the H100, there is no explicit mention of a GPU roadmap.

The closest related discussion is Mark Zuckerberg's mention of the need for physical infrastructure, including compute and energy, to support the growth of AI. He also mentions that NVIDIA needs time to stabilize their new generation of systems.

However, this is not a direct announcement or discussion of a GPU roadmap.

Therefore, the answer is: No, the Bingo Card label "GPU Roadmap" cannot be marked off based on the provided information.

WhatsApp Integration

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "WhatsApp Integration" qualifies for being marked off.

The transcripts mention that Meta AI is "most used in WhatsApp" (Mark Zuckerberg's interview). This indicates that there is an integration of Meta AI with WhatsApp.

Therefore, the answer is: Yes, the Bingo Card label "WhatsApp Integration" can be marked off.

Paid API Tiers

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Paid API Tiers" qualifies for being marked off.

The transcripts mention a discussion about monetizing AI, where Mark Zuckerberg talks about the possibility of having a premium service that supports people using arbitrary amounts of compute to do more amazing things. He states that "for that, I'm sure we'll end up having a premium service" (Mark Zuckerberg's interview).

This indicates that Meta is considering or planning to have paid tiers for their AI services, which could be related to the API.

Therefore, the answer is: Yes, the Bingo Card label "Paid API Tiers" can be marked off.

EU Privacy Concerns

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "EU Privacy Concerns" qualifies for being marked off.

Upon reviewing the transcripts and news articles, I did not find any direct mention of "EU Privacy Concerns". While there are discussions about data privacy and security, such as the Llama API being designed with a focus on data privacy and security, and not using user prompts or model responses to train AI models, there is no explicit mention of EU-specific privacy concerns.

Therefore, the answer is: No, the Bingo Card label "EU Privacy Concerns" cannot be marked off based on the provided information.

Llama 5 Teaser

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Llama 5 Teaser" qualifies for being marked off.

Upon reviewing the transcripts and news articles, I found a mention of "Llama 5 B model" by Angela Fan, where she talks about the synthetic data tool kits that help generate good post-training data for use cases, and mentions that there will be more information on this topic.

However, this is not a direct teaser for Llama 5, but rather a mention of a related topic.

Mark Zuckerberg also mentions "Little Llama" as an internal code name for a future Llama model, but it's not clear if this refers to Llama 5.

There is no explicit teaser or announcement for Llama 5.

Therefore, the answer is: No, the Bingo Card label "Llama 5 Teaser" cannot be marked off based on the provided information.

Open Source Development

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Open Source Development" qualifies for being marked off.

The transcripts and news articles are filled with discussions and announcements about open source development, including:

Chris Cox's opening remarks highlighting the importance of open source AI and the growth of the Llama ecosystem.
Mark Zuckerberg and Ali Ghodsi's discussion on the benefits of open source AI, including the ability to customize and fine-tune models.
Satya Nadella's discussion on Microsoft's commitment to supporting open source models and providing a platform for developers to build on top of them.
The announcement of the Llama API, which is designed to provide a flexible and open way for developers to build with Llama models.
The news article "Everything we announced at our first-ever LlamaCon" highlights Meta's commitment to open source AI and the Llama ecosystem.
These discussions and announcements demonstrate a strong focus on open source development and the Llama ecosystem.

Therefore, the answer is: Yes, the Bingo Card label "Open Source Development" can be marked off.

Meta AI

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Meta AI" qualifies for being marked off.

The transcripts and news articles mention Meta AI multiple times, including:

Chris Cox's discussion on the Meta AI app, which was launched during the event, and its features, such as full-duplex voice and personalization.
Mark Zuckerberg's discussion on Meta AI, including its usage and growth, with almost a billion monthly active users.
The Meta AI app is described as a standalone app that provides a new experience for users, with features like voice interface and personalization.
These discussions and announcements demonstrate a significant focus on Meta AI, its features, and its growth.

Therefore, the answer is: Yes, the Bingo Card label "Meta AI" can be marked off.

Developer Tools

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Developer Tools" qualifies for being marked off.

The transcripts and news articles mention several developer tools and features, including:

The Llama API, which provides a lightweight SDK in Python and Typescript, and is compatible with the OpenAI SDK.
Interactive playgrounds to explore different Llama models.
Fine-tuning and evaluation tools for customizing Llama models.
Synthetic data tool kits to help generate good post-training data for use cases.
Satya Nadella's discussion on Microsoft's developer tools, including GitHub Copilot, which is being used to improve developer productivity.
These discussions and announcements demonstrate a significant focus on providing developer tools and features to support the growth of the Llama ecosystem.

Therefore, the answer is: Yes, the Bingo Card label "Developer Tools" can be marked off.

Meta AI Web Interface

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Meta AI Web Interface" qualifies for being marked off.

The transcripts mention the launch of a new Meta AI app, which is a standalone app that provides a new experience for users. Chris Cox discusses the app's features, including a voice interface and personalization.

However, there is no explicit mention of a "Meta AI Web Interface". The discussion is focused on the Meta AI app, which is a mobile or standalone application, rather than a web interface.

Therefore, the answer is: No, the Bingo Card label "Meta AI Web Interface" cannot be marked off based on the provided information.

Enterprise Licensing

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Enterprise Licensing" qualifies for being marked off.

The transcripts mention a discussion between Mark Zuckerberg and the interviewer about the Llama license, including the requirement for companies to contact Meta if they want to use Llama models and reach 700 million users.

Mark Zuckerberg states that the goal of the license is to have a conversation with large cloud companies, like Microsoft and Amazon, before they use Llama models, and to find a productive way to work together.

This discussion implies that there are licensing terms and conditions for enterprise use of Llama models, which is related to the concept of "Enterprise Licensing".

Therefore, the answer is: Yes, the Bingo Card label "Enterprise Licensing" can be marked off.

Ray-Ban Smart Glasses

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Ray-Ban Smart Glasses" qualifies for being marked off.

The transcripts mention Ray-Ban Meta glasses, which are described as being "incredibly popular" and having AI capabilities. Chris Cox discusses how the Meta AI app focuses on taking the experience of using the glasses and making it coherent, whether you're using the glasses or your phone.

Mark Zuckerberg also mentions the Ray-Ban Meta product as an example of a successful augmented reality product that "gets out of the way" and is a good-looking pair of glasses.

These discussions indicate that Ray-Ban Smart Glasses (or Ray-Ban Meta glasses) are being referenced and discussed during the event.

Therefore, the answer is: Yes, the Bingo Card label "Ray-Ban Smart Glasses" can be marked off.

Horizon Avatars

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Horizon Avatars" qualifies for being marked off.

The transcripts mention "Codec Avatars" by Mark Zuckerberg, which is a technology related to creating realistic digital avatars. He discusses how this technology can be used to create a more immersive experience, such as having an "always-on video chat with the AI" with realistic gestures.

However, there is no explicit mention of "Horizon Avatars". While Codec Avatars is a related concept, it is not the same as Horizon Avatars.

Therefore, the answer is: No, the Bingo Card label "Horizon Avatars" cannot be marked off based on the provided information.

Cloud Integrations

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Cloud Integrations" qualifies for being marked off.

The transcripts and news articles mention several cloud integrations, including:

A discussion between Mark Zuckerberg and Satya Nadella about the importance of cloud infrastructure for AI development and deployment.
The news article "Everything we announced at our first-ever LlamaCon" mentions that Meta has collaborated with partners to build Llama Stack distributions, including integrations with NVIDIA NeMo microservices, IBM, Red Hat, Dell Technologies, and others.
These discussions and announcements indicate that there are cloud integrations being developed and announced, which directly corresponds to the Bingo Card label "Cloud Integrations".

Therefore, the answer is: Yes, the Bingo Card label "Cloud Integrations" can be marked off.

AI Infrastructure Investments

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "AI Infrastructure Investments" qualifies for being marked off.

The transcripts mention several discussions related to AI infrastructure investments, including:

Mark Zuckerberg's discussion on the need for significant investments in physical infrastructure, including compute and energy, to support the growth of AI.
Mark Zuckerberg mentions that building a gigawatt cluster of compute requires significant infrastructure investments, including building data centers and obtaining energy.
Satya Nadella discusses the importance of building out infrastructure, including compute, storage, and networking, to support AI workloads.
These discussions indicate that there are significant investments being made in AI infrastructure, which directly corresponds to the Bingo Card label "AI Infrastructure Investments".

Therefore, the answer is: Yes, the Bingo Card label "AI Infrastructure Investments" can be marked off.

Databricks Partnership

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Databricks Partnership" qualifies for being marked off.

The transcripts include a conversation between Mark Zuckerberg and Ali Ghodsi, the CEO of Databricks, where they discuss the use of Llama models on the Databricks platform and the benefits of open-source AI.

This conversation indicates a partnership or collaboration between Meta and Databricks, which directly corresponds to the Bingo Card label "Databricks Partnership".

Therefore, the answer is: Yes, the Bingo Card label "Databricks Partnership" can be marked off.

Together AI Integration

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Together AI Integration" qualifies for being marked off.

Upon reviewing the transcripts and news articles, I did not find any mention of "Together AI Integration". The discussions and announcements are focused on other topics, such as Llama models, Meta AI, and partnerships with companies like Nvidia and Databricks.

Therefore, the answer is: No, the Bingo Card label "Together AI Integration" cannot be marked off based on the provided information.

Groq Partnership

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Groq Partnership" qualifies for being marked off.

The news article "Everything we announced at our first-ever LlamaCon" mentions a collaboration between Meta and Cerebras and Groq to provide faster inference speeds using the Llama API. It states that early experimental access to Llama 4 models powered by Cerebras and Groq are available by request.

This announcement indicates a partnership between Meta and Groq, which directly corresponds to the Bingo Card label "Groq Partnership".

Therefore, the answer is: Yes, the Bingo Card label "Groq Partnership" can be marked off.

Reasoning Model

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Reasoning Model" qualifies for being marked off.

The transcripts mention several discussions related to reasoning models, including:

Mark Zuckerberg's discussion on the importance of reasoning models, which consume more test-time or inference-time compute to provide more intelligence.
Mark Zuckerberg mentions that Meta is building a Llama 4 reasoning model, which will be released at some point.
Ali Ghodsi discusses the trend of using reasoning models and the importance of distillation and fine-tuning for specific tasks.
These discussions indicate that reasoning models are a topic of interest and development, and that Meta is working on building a Llama 4 reasoning model.

Therefore, the answer is: Yes, the Bingo Card label "Reasoning Model" can be marked off.

Meta Connect

Based on the provided transcripts and news articles from the LlamaCon event, I can help you determine if the Bingo Card label "Meta Connect" qualifies for being marked off.

Upon reviewing the transcripts and news articles, I did not find any mention of "Meta Connect". The discussions and announcements are focused on LlamaCon, Llama models, and related topics.

Meta Connect is likely a different event or conference, and there is no indication that it was discussed or announced during LlamaCon.

Therefore, the answer is: No, the Bingo Card label "Meta Connect" cannot be marked off based on the provided information.
]]>



https://spreadsheets-are-all-you-need.ai/arrrzero-why-deepseek-r1-is-less-important-than-r1-zero/
		
		
Fri, 14 Mar 2025 21:57:23 +0000

https://spreadsheets-are-all-you-need.ai/?p=285

The headlines about DeepSeek R1's cost are missing an even bigger story, the R1-Zero model. This model, and research behind it, are reshaping how we thought AI assistants should be trained.












To understand why the R1-Zero model matters, we have to look at how AI chatbots like ChatGPT are typically created. When an AI model is trained on text from the internet, the resulting model is actually pretty dumb. Chatting with it is useless because it only knows how to literally imitate text on the internet. For example, in the video for this blog post, when the model (called a "base model") is given a prompt like "First Name:", it replies "Name:, Address:". Essentially, the model is trying to guess what kind of web page might contain the prompt and then replies with the imagined content from the rest of that hypothetical web page. In this case, it sees "First Name:" and guesses it's part of a form, so it replies with the rest of the form (i.e. Address). This is effective enough for crafting fake news stories about Unicorns but it can't carry a conversation nor answer user questions directly.







To turn this "base model" (also known as a "pretrained model") into a chatbot like ChatGPT has typically involved a complex and labor-intensive process. The process requires the three major steps shown in the diagram below (adapted from the State of GPT) that are labeled "Supervised Finetuning", "Reward Modeling", and "Reinforcement Learning".







The "Supervised Finetuning" step requires humans to write thousands of example conversations teaching the model how a helpful AI assistant should behave. Even once the model has learned from those examples, humans are needed to grade the quality of the responses the model produces which further fine tunes the quality of the model in a process called "Reinforcement Learning from Human Feedback (RLHF)".



DeepSeek R1-Zero changes the game by eliminating most of this human work. Instead of requiring armies of people to train and fine-tune the AI, it uses math and coding problems to teach the AI how to think and reason. Because these problems have clear right and wrong answers, computers can automatically check if the AI is correct without any human oversight needed.







"It's an aha moment, not just for the model, but for the researchers," notes the DeepSeek team, describing the moment their AI learned to catch its own mistakes and consider different approaches to problems.



This breakthrough could accelerate AI development by removing what experts call the "human bottleneck" (the need for extensive human input in training AI systems). It's already being recognized as a major advancement, with some experts considering it even more significant than other recent AI developments that have grabbed headlines.







What makes this particularly exciting is that several teams have already successfully reproduced similar results using smaller-scale versions of this approach, suggesting this could become a new standard way of creating AI assistants that can think and reason effectively.



As AI continues to evolve, DeepSeek R1-Zero's innovative training method might just be the key to creating smarter AI assistants more quickly and efficiently than ever before.
]]>



https://spreadsheets-are-all-you-need.ai/flops-demystified-ai-and-the-math-behind-deepseek-training-costs/
		
		
Sat, 22 Feb 2025 18:24:09 +0000

https://spreadsheets-are-all-you-need.ai/?p=1508


.entry-content {
font-size: 1.35em;
}




This might be my most important AI article yet but it's also my biggest FLOP.









FLOPs are one of the most fundamental metrics in AI. To understand how AI works and what it costs to train amazing models like ChatGPT or DeepSeek, you need to understand FLOPs.



In the accompanying video and this article we will:




Explore the difference between FLOP, FLOPS, & FLOPs



Reveal why AI is fundamentally different from traditional software 



Crack open GPT-2 (using spreadsheets-are-all-you-need) to count every FLOP



Apply these insights to analyze DeepSeek's viral "$5.3M training cost" claim and uncover what this number really means (and what it doesn't)




What Are FLOPs?







FLOP stands for FLoating point OPeration. Any basic mathematical operation (like addition or multiplication) performed on decimal numbers is considered a FLOP. For example, if you add 5.2 and 4.4 to get 9.6 you just did a single FLOP.



Confusingly, FLOPs (with a lowercase 's') and FLOPS (with a capital 'S') are different: 




FLOPs (with a lowercase 's'): how many floating point operations occur



FLOPS (with a capital 'S'): how many operations happen per second.




As an analogy you can think of FLOPs as “miles” (a quantity of something) and FLOPS as "miles per hour" (how fast that number changes over time).



Why FLOPs Matter in AI







Traditional software applications like Microsoft Word or PowerPoint primarily use logic and control flow statements - if/then conditions, loops, and function calls. AI models, particularly large language models, work fundamentally differently. They convert words into numbers, perform massive amounts of mathematical operations on these numbers, and then convert the resulting numbers back into words.



Let's look at a simplified example. You’ve probably heard that AI models are trained to fill in the blank and complete passages like this one: "Mike is quick. He moves ___". 







To solve this, the AI model will,




Convert each word into floating point numbers (aka decimal numbers)



Perform complex mathematical calculations on those numbers



Produce a final decimal number



Maps that final decimal number back to the known words in its vocabulary, with closer matches getting higher probabilities




For our example passage "Mike is quick. He moves ___", words like "quickly" and "fast" would be chosen in step 4 because they map to numbers that are close to the model's calculated result in step 3.



Measuring FLOPs in Practice







Using the web version of Spreadsheets Are All You Need, we can actually count these operations in a real language model without leaving our browser. Watch the accompanying video and follow along the steps yourself to see how easy it is to actually count FLOPs!



When processing just six tokens (words or parts of words), GPT-2 Small performs approximately one billion floating point operations. 







This matches closely with theoretical estimates derived from the model's architecture. A common useful estimate is that the number of FLOPs needed to process one token is approximately two times the number of parameters in the model. This makes sense when you consider that each neuron in a neural network primarily performs two operations (multiplication and addition) for each parameter.



Understanding AI Training Costs



This brings us to the recent discussion around DeepSeek's training costs. DeepSeek reported spending $5.3 million to train their latest model. As we show in the video, this estimate does line up with theoretical estimates and the reported data from their technical report:







Unfortunately, this led to articles like the one below, comparing this figure to OpenAI's reported $5 billion in development costs. 







This comparison is deeply misleading.



The $5.3 million represents just the GPU compute costs for the final training run. It doesn't include:




Costs of research and experimentation leading up to the final architecture



Failed training attempts



Data collection and processing



Personnel costs



Infrastructure beyond raw compute




Yes DeepSeek probably spent $5.4 million on their final training run but that was not the total amount they spent to build the model.







Think of it like measuring the cost of building a house by only counting the lumber used in the final construction. Yes, lumber is a significant expense, but it's far from the total cost of creating a house.



The Reality of Model Development



As I argue in the video, the development of frontier AI models is more akin to Thomas Edison's journey to create the light bulb. Edison didn't just build one light bulb. He made nearly 3,000 attempts before finding a design that worked. Each attempt represented real costs in materials, time, and labor.



Similarly, creating a successful AI model requires numerous experiments, failed attempts, and iterations. The final training run is just the tip of a very expensive iceberg.



The next time you see headlines about FLOPs or AI training costs, hopefully you’ll now be better prepared to not only understand them but also put them in the proper context. 



Best of luck on your AI journey.
]]>