Startups Seek Alternatives to Transformers in Large Language Models
Photo: MIT Technology Review
Quick answer
Startups are pioneering transformer alternatives for LLMs to address high energy costs and contextual processing limitations, aiming to deliver more efficient and scalable AI solutions.
Transformers, the foundation of modern large language models (LLMs), are struggling to meet the industry's growing demands. Their core mechanism—dense attention—delivers high accuracy but requires massive computational resources. For instance, processing a 10,000-word document can demand up to 50 million multiplication operations, making these models energy-intensive and costly.
According to OpenAI President Greg Brockman, the company plans to spend $50 billion on computing this year. The International Energy Agency projects that data center electricity consumption will double by 2030. These figures highlight the urgent need for alternative solutions.
The problem is exacerbated by transformers' limited context window. Models struggle to handle large-scale data simultaneously, a critical limitation for tasks requiring analysis of entire document libraries or codebases. Startups like Subquadratic see this as an opportunity for innovation and are developing new architectures to overcome these constraints.
The LLM market is open to change, and companies unburdened by legacy technologies could lead the next generation of models. Their success hinges on delivering solutions that are not only more efficient but also more cost-effective than today's transformers.
* Facebook, Instagram, WhatsApp, and other Meta services are owned by Meta Platforms Inc., an organization recognized as extremist and banned in the Russian Federation.
Common questions
- Why are transformers becoming outdated?
- Transformers demand massive computational power for processing long texts, leading to high energy consumption and limited context window sizes. This restricts their ability to handle complex tasks requiring large-scale data analysis.
- Which startups are developing transformer alternatives?
- Companies like Subquadratic and others are exploring novel model architectures to overcome transformer limitations. Their goal is to create more efficient and scalable AI solutions.
- What benefits will new technologies bring to LLMs?
- Alternative approaches could reduce energy consumption, increase data processing speed, and expand context windows. This would enable AI to tackle more complex tasks, such as analyzing entire libraries of documents or codebases.
Dzen feed: /feed/dzen.xml · RSS: /feed.xml