Nvidia has dominated the AI chip market by repurposing semiconductors originally designed for video games. Now, a small startup named Groq, founded eight years ago, aims to challenge one of the world's most valuable companies by launching a new chip developed from the ground up specifically for neural networks.
Speaking in February 2024 to an audience in Oslo, Jonathan Ross, the 42-year-old CEO of Groq, noticed something unusual happening. Ross was demonstrating his company's chatbot, which he hoped would help his chip-making business for neural networks emerge from a prolonged slump, to members of the Norwegian parliament and tech executives. The chatbot was designed to respond to inquiries almost instantly, even faster than a human can read. However, during the demonstration, the chatbot experienced slight delays. This greatly frustrated Ross, who was hoping to gain support for his project to establish a Groq-based data center in Europe to showcase the advantages of specialized chips that enable neural networks to operate at such high speeds. "I kept checking the metrics," he recalls. "And the people at the presentation couldn't understand why I was so distracted."
It turned out that the reason for the chatbot's delayed response during the demo was a surge of new users. The day before Ross's presentation in Oslo, one of the founders of a successful startup tweeted enthusiastically about a "super-fast AI assistant." As a result, the demo page received so much new traffic that the company's servers couldn't handle the load. It was a problem, but one that was pleasant to have.
Ross founded Groq eight years ago with the goal of creating chips specifically designed for what the industry calls "inference" – the ability of artificial intelligence to mimic human thinking by learning and applying acquired knowledge to new situations. It is inference that allows your smartphone to recognize your corgi in a newly uploaded photo and enables an image generator to create a realistic picture of the Pope in a Balenciaga cloak. This is significantly different from another computationally intensive AI task – training models on large datasets.
However, before OpenAI released ChatGPT in late 2022, sparking a global surge in interest in neural networks, the demand for ultra-fast inference was limited, and Ross's startup was barely staying afloat. "Groq has almost sunk multiple times," Ross says during an interview in his semiconductor lab in San Jose, California. He recalls a moment in 2019 when the company had less than a month’s worth of funds left: "Perhaps we started Groq too early."
But now, with the demand for computing power to create and maintain neural networks so high that it contributes to a global energy crisis, it seems that Groq's time has come – either as a potential new market leader or as a target for acquisition by larger players.
The need is so great that Nvidia's market capitalization has soared to $3 trillion compared to $60.9 billion in 2023. In contrast, Groq is still a small player with an estimated revenue of $2 million, according to sources familiar with the company's financial situation. These same sources report that, given the sharp rise in interest in chips, the company forecasts optimistic sales of $100 million this year. "Computing power is the new oil," Ross states.
AI product developers occupy 16 spots in our ninth annual Cloud 100 ranking, which highlights the best private companies in the cloud technology sector worldwide. Last year, there were only eight, and five years ago, none at all. With the AI chip market expected to grow to $1.1 trillion by 2027, Ross sees an opportunity to capture some of Nvidia's market share, which currently holds an astonishing 80% of the market. Groq focuses exclusively on developments for inference. According to research firm IDC, this market segment will grow to $39 billion this year and reach $60.7 billion over the next four years.
Startups like Groq are optimistic because Nvidia's chips were not originally designed for AI. When CEO Jensen Huang first presented graphics processing units (GPUs) in 1999, they were products developed for graphically intensive video games. The fact that they became the most suitable processors for training AI was merely a happy coincidence. However, Groq, along with other new next-generation chip startups like Cerebras (valued at $4 billion) and SambaNova (valued at $5.1 billion), sees this as a fertile ground for new opportunities. "If we were to start working with AI now, no one would think of using GPUs for neural networks," says Andrew Feldman, CEO of Cerebras.
It's not just startups that dream of displacing Nvidia from its leading position. Amazon and Microsoft are developing their own processors for neural networks. However, Groq's chips – known as Language Processing Units (LPUs) – are so impressively fast that the company has every chance of success. In a presentation released this year for investors, Groq claims that its chips for inference are four times faster, five times cheaper, and three times more energy-efficient than Nvidia's GPUs. The startup has recently announced new investments in a Series D funding round, with BlackRock as the main investor. Groq aims to raise at least $350 million at a company valuation of at least $2 billion, according to sources familiar with the process. "The speed of their inference chips clearly surpasses all other products on the market," says Emish Shah, co-founder of General Global Capital, which has participated in several funding rounds for Groq.
Groq launched its chips two years ago, and now among its clients is Argonne National Laboratory. This government research center, founded in the wake of the "Manhattan Project," uses Groq chips to study nuclear fusion – a type of energy produced by the sun. A deal to use Groq processors has also been made with Aramco Digital – the technology division of the Saudi oil company.
In March, Groq launched GroqCloud, a service allowing users to rent access to its processors via an online interface. To demonstrate the product's advantages to developers, Groq made access to the cloud product free. In the first month, 70,000 users registered on the site. Now there are already 280,000, and growth continues. On June 30, the company launched paid subscriptions and recently invited former Intel employee Stuart Pann to serve as Chief Operating Officer to boost revenues and scale operations. Pann has every reason to be optimistic about the future: 40 percent of requests from GroqCloud users are for additional capacity payments.
"The Groq chip really hit the mark," says Yann LeCun, Chief AI Scientist at Meta, who once taught Ross computer science at New York University. He recently joined Groq as a technical advisor. Ross began his career at Google, where he worked on the team that created tensor processing units optimized for machine learning. He left in 2016 to found Groq with his former Google colleague Doug Whitman, who became the startup's first CEO. In its first year, the startup raised $10 million in a round led by venture fund Social Capital. However, finding new investors afterward proved challenging. Whitman left the startup a few years later and did not respond to requests from Forbes for an interview for this article.
There are still those who remain skeptical. One venture capitalist who declined to participate in the upcoming funding round for the company describes Groq's approach as "innovative" but doubts that the company's intellectual property will be protected in the long term. Mitesh Agarwal, head of cloud solutions at the $1.5 billion company Lambda Labs, states that his startup does not plan to offer Groq chips or other specialized chips in its cloud. "It's still too complicated to go beyond what Nvidia offers," he says.
Ross knows that the company faces a tough road ahead. "It's like winning the 'Rookie of the Year' award," he says. "We're still a long way from Nvidia. So everyone will be watching us closely now, as if constantly asking, 'So what will you do next?'"
***
Déjà vu: Pseudobots
Artificial intelligence can perform certain tasks better than most humans. For example, drawing cats in space suits or passing standardized tests on school curricula. However, heated discussions about whether these language models are truly intelligent or merely reproduce previously learned information continue. Here’s a brief overview of how seemingly intelligent machines have emerged throughout human history.
Around 75 AD: Greek mathematician Hero of Alexandria creates statues that automatically pour wine in temples, presented to worshippers as divine acts.
1769: In Austria, Wolfgang von Kempelen introduces a chess automaton called "The Turk" – a box-shaped machine that actually housed a short (as believed) chess player. It is said that "The Turk" won against both Benjamin Franklin and Napoleon.
1939: Electro, a talking and smoking robot, appears at the World’s Fair in New York. However, the words it speaks are pre-recorded, and it requires a human to clean its "lungs" of tar.
1965: The EL