OpenAI’s Jalapeño Chip Breakthrough: Up to 1.9× More AI Work Per Watt Revealed

Chun Zhang
2 days ago
9 min read

The economics of artificial intelligence are increasingly being shaped not only by how capable models become, but by how efficiently those models can be served to millions of users. Training remains enormously compute-intensive, but inference, the process of generating responses after a model has been trained, is becoming an equally important infrastructure challenge as AI assistants, autonomous agents, coding systems, and multimodal applications move into everyday use.

OpenAI’s Jalapeño, its first custom inference accelerator, represents a significant step toward addressing that challenge through specialized silicon and full-stack system design. Initial benchmark results indicate that the chip can simultaneously improve throughput, power efficiency, and response latency across several demanding model families. More importantly, Jalapeño illustrates a broader strategic shift among leading AI companies, from relying almost entirely on general-purpose accelerators toward designing hardware specifically around the characteristics of modern AI workloads.

Why AI Inference Has Become an Infrastructure Battleground

AI inference is fundamentally different from conventional computing. A language model does not simply execute one fixed calculation. It repeatedly processes information, retrieves and updates model state, generates tokens, and coordinates data across memory and compute resources.

Two phases are particularly important. During prefill, the system processes the user's input context, a stage that is generally more compute-intensive. During decode, the model generates output sequentially, token by token. Decode workloads place much greater pressure on memory bandwidth and data movement.

This distinction creates a difficult optimization problem. A processor optimized for maximum computational throughput may not necessarily provide the lowest response time for an interactive user. Conversely, a system designed for extremely low latency can sacrifice utilization and efficiency when serving large numbers of simultaneous requests.

That tradeoff becomes particularly consequential for AI agents. An agent may need to perform numerous reasoning, retrieval, tool-use, and execution steps before completing a task. Even relatively small delays can accumulate across those sequential operations.

Jalapeño is designed around this reality rather than treating inference as a generic acceleration problem.

Jalapeño’s Benchmark Advantage

OpenAI evaluated Jalapeño using InferenceX, a public benchmark from SemiAnalysis designed to measure AI serving performance across different operating conditions. The comparison included commercially available accelerator systems and examined both highly interactive workloads and higher-throughput serving.

Across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T, the reported results showed a consistent advantage in performance efficiency and latency.

Model	Peak throughput per watt advantage	Lower latency	Peak per-user decoding advantage
GPT-OSS 120B	~1.9×	~1.7×	~2.7×
DeepSeek R1 670B	~1.7×	~3.6×	~4.1×
Kimi K2.5 1T	~1.5×	~3.4×	~3.8×

The significance of these numbers is not simply that Jalapeño produces more tokens. Its stronger proposition is the combination of speed and efficiency.

OpenAI reported that Jalapeño achieved between 1.5 and 1.9 times more AI work per watt at peak throughput across the three models, while end-to-end latency was between 1.7 and 3.6 times lower. For highly interactive workloads, the company reported performance improvements ranging from 2.1 to 4.1 times.

The results are particularly notable because the benchmark was conducted across models originating outside OpenAI as well as its own open-weight model. That suggests the accelerator is not narrowly tuned to a single proprietary workload.

The Technical Principle Behind the Performance

The central idea behind Jalapeño is co-design.

Instead of developing a chip independently and adapting software afterward, OpenAI is designing models, serving software, silicon, memory, networking, and system architecture as interconnected components.

This approach addresses one of the fundamental inefficiencies in large-scale AI inference: moving data.

Modern language models require enormous amounts of model state. During generation, the key-value cache, commonly called the KV cache, stores information needed to maintain context efficiently across successive tokens. If that state has to move repeatedly between processors, memory resources, or network domains, communication overhead can undermine the theoretical performance of the underlying compute hardware.

Jalapeño attempts to reduce that penalty by explicitly controlling where model state resides and keeping important data local where possible.

The network is also treated as part of the computational architecture rather than simply as an external connectivity layer. A large connected system can allow workloads to remain within a coordinated domain, reducing unnecessary movement and helping different components cooperate more efficiently.

The result is intended to be a more balanced accelerator, one capable of handling both compute-heavy prefill and memory-intensive decode without forcing the entire infrastructure to optimize around only one phase.

Performance Per Watt Matters More Than Raw Chip Speed

One of the most important aspects of the Jalapeño results is the emphasis on performance per unit of power.

A faster accelerator is not necessarily a more economically attractive accelerator. Data centers must account for electricity, cooling, power delivery, physical infrastructure, and the cost of operating thousands or potentially millions of accelerator devices.

Jalapeño has a published chip power rating of 700 watts, while OpenAI reported that measured sustained power remained at or below 550 watts during the tested workloads. The benchmark comparisons normalized systems using their published chip power ratings.

This distinction matters because AI infrastructure is increasingly constrained by energy availability as well as semiconductor supply.

Improving useful inference generated per kilowatt can therefore have several consequences at once:

More AI requests can be served using the same power budget.
Data-center operating costs can potentially fall.
Existing infrastructure can support greater demand.
Interactive applications can become more responsive.
Higher-capability models can become economically practical at larger scale.

For an AI provider facing rapidly increasing demand, efficiency is not merely an engineering metric. It becomes a strategic business variable.

AI Helped Build Jalapeño, and AI Can Help Program It

Another notable aspect of Jalapeño is that artificial intelligence was used during the chip's development.

OpenAI says AI-assisted engineering helped the team move from initial design toward tapeout in nine months. AI was used to explore implementations, accelerate design and verification cycles, and optimize arithmetic circuits.

This points toward a potentially important feedback loop in semiconductor development. AI can increasingly assist in designing the infrastructure required to run AI itself.

The relationship does not end with chip design. Jalapeño was also constructed around a relatively predictable programming model involving local tensors, explicit communication, and synchronization.

That structure can make parallel programming more tractable for AI systems. Rather than requiring an AI coding system to reason about an opaque hardware environment, the architecture exposes clearer computational and communication relationships that can be optimized.

OpenAI reported that, using Codex with GPT-Astra, its team brought three open-weight models that were not originally planned for Jalapeño production to high performance within two months. For selected GPT-OSS attention and mixture-of-experts components, AI-generated implementations reportedly ran 1.5 to 1.8 times faster than previous implementations written by human experts.

These results apply to selected components rather than complete models, but they demonstrate an emerging development model in which AI helps optimize both the hardware and the software running on it.

The Strategic Importance of Custom AI Silicon

Jalapeño also fits into a much larger industry movement.

The dominant AI infrastructure model has historically depended heavily on commercial accelerators, particularly those supplied by NVIDIA. But major AI companies increasingly have incentives to develop specialized processors because the economics of inference differ from those of general-purpose GPU workloads.

Google has developed its TPU family, while Amazon and Microsoft have pursued their own AI silicon strategies. Anthropic has also signaled interest in developing an internal chip-design capability.

For OpenAI, Jalapeño provides another layer of infrastructure control.

The company can study its real workloads, identify bottlenecks, modify the architecture, optimize the software stack, and feed those lessons into subsequent generations. That creates a development cycle that can be difficult to replicate when hardware and software are supplied by separate organizations.

Importantly, Jalapeño is not positioned as a replacement for commercial accelerators across the entire AI infrastructure stack. OpenAI has indicated that NVIDIA and other accelerator providers will continue to play a major role, particularly for training and broader compute requirements.

The company's hardware strategy is therefore better understood as diversification and specialization rather than an immediate abandonment of external silicon.

Jalapeño Is an Inference Platform, Not a Product for Sale

Despite demonstrating Jalapeño on public models, OpenAI does not currently plan to sell the accelerator to external customers.

That decision reflects the immediate strategic value of the hardware internally. If inference capacity is one of the company's most important constraints, keeping specialized silicon for internal infrastructure can provide greater value than turning it into a standalone semiconductor business.

This also distinguishes Jalapeño from traditional chip companies. Its purpose is fundamentally tied to OpenAI's broader AI platform.

The chip can be optimized around the company's serving workloads, software infrastructure, models, and future products without needing to satisfy the requirements of a broad external customer base.

A Multigenerational Roadmap for AI Infrastructure

Jalapeño is described as the first generation of a longer hardware program. OpenAI has said that a second-generation design is already deep in development and that work on a third generation is beginning to take shape.

That roadmap is important because the AI workload itself is changing rapidly.

Models are becoming larger, reasoning workloads are becoming more complex, agentic systems are performing longer sequences of operations, and inference increasingly involves heterogeneous combinations of computation, memory access, networking, and orchestration.

A fixed accelerator architecture can therefore become obsolete if it is optimized around yesterday's workload.

A multigenerational strategy allows OpenAI to continuously incorporate lessons from deployed systems into subsequent hardware. The potential advantage is not simply faster chips, but an increasingly coordinated infrastructure stack in which each generation responds to observed bottlenecks.

What Jalapeño Could Mean for the AI Industry

The broader significance of Jalapeño extends beyond benchmark rankings.

AI infrastructure is entering an era where efficiency may become as important as raw model capability. As AI adoption expands, the cost of generating every response becomes a central economic consideration.

More efficient inference could make sophisticated models available at lower cost, allow providers to serve greater demand, and enable applications that would be impractical under higher latency or power requirements.

For agentic AI in particular, low latency can influence the user experience directly. An agent that completes multiple operations rapidly feels fundamentally different from one that pauses between every step. Faster infrastructure can therefore translate into new product capabilities rather than simply shorter loading times.

The development also demonstrates why full-stack optimization is becoming increasingly important. Future AI performance may depend less on isolated improvements in a processor and more on how effectively models, compilers, memory, networking, silicon, and software operate together.

The Road Ahead

OpenAI plans to begin deploying Jalapeño into its compute infrastructure by the end of 2026, initially in small volumes, with broader deployment expected to increase during 2027.

The immediate challenge will be translating benchmark performance into reliable operation at production scale. Software maturity, model coverage, manufacturing, system reliability, workload diversity, and infrastructure integration will all influence the real-world impact.

There is also a moving competitive landscape. By the time Jalapeño reaches substantial deployment, commercial accelerator generations will have advanced as well. Its long-term value will therefore depend not on maintaining a single benchmark lead, but on whether OpenAI can sustain an architectural advantage across successive generations.

That is ultimately what makes Jalapeño strategically significant. It represents an attempt to turn AI infrastructure into a continuous feedback system, where models generate workloads, workloads reveal bottlenecks, bottlenecks inform silicon design, and new silicon enables more capable AI.

For the AI industry, this signals a broader transition. The race is no longer limited to building the most capable models. It increasingly includes building the most efficient machinery for delivering those models to the world.

The work being pursued by teams such as Dr. Shahid Masood and 1950.ai highlights why developments in AI infrastructure deserve attention beyond semiconductor engineering. The next phase of artificial intelligence will be shaped not only by algorithms and model intelligence, but by the computing systems capable of making advanced intelligence fast, affordable, scalable, and widely accessible.

Jalapeño is an early example of that transition, and its most important contribution may ultimately be less about one benchmark result than about demonstrating what becomes possible when AI models and the infrastructure that serves them are designed as one system.

Key Takeaways

Jalapeño is OpenAI's first custom inference chip, designed specifically around modern and future AI serving workloads.
Initial benchmark results show simultaneous gains in throughput, efficiency, and latency across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T.
The architecture emphasizes minimizing data movement, particularly for model state and KV-cache operations.
Performance per watt is central to the design, reflecting the growing importance of energy and operating costs in AI infrastructure.
AI assisted both chip development and software optimization, creating a feedback loop between AI and the infrastructure used to run it.
Jalapeño is part of a broader custom-silicon trend, as AI companies seek greater control over inference economics and infrastructure capacity.
OpenAI intends to retain the chip for internal use, rather than commercializing it as a standalone accelerator.
Second- and third-generation systems are already part of the roadmap, indicating that OpenAI views custom inference silicon as a long-term infrastructure strategy.
The ultimate test will be production-scale deployment, where software maturity, reliability, workload diversity, supply, and competing accelerator generations will determine its lasting impact.