Microsoft unveils Maia 300 inference chip featuring 2nm chiplets and 288GB HBM4 memory

Microsoft has officially announced the Azure Maia 300, its latest custom silicon designed for high-scale AI inference. The company states this accelerator will become the primary source of inference capacity for its cloud services over time.

What happened

Microsoft unveiled the Maia 300 accelerator, which is built specifically for artificial intelligence inference tasks rather than model training. According to SiliconANGLE, the company estimates the chip can process 40 percent more tokens per megawatt than leading GPU-based systems. This efficiency gain likely targets competitors such as Nvidia, though Microsoft did not name them explicitly in the announcement.

The hardware features a multichiplet architecture consisting of four computing chiplets. These components are manufactured using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometer process node. This advanced fabrication technology allows for higher density and improved performance characteristics within the same physical footprint.

Memory configuration plays a central role in the Maia 300 design. The accelerator includes eight HBM4 modules that provide a combined 288 gigabytes of capacity. This total memory amount matches the flagship Rubin accelerator from Nvidia, according to the reporting outlet. However, Microsoft acknowledges that Maia 300 offers approximately 20 percent less memory bandwidth than the Rubin chip.

To mitigate the lower bandwidth, Microsoft implemented a dataflow architecture that minimizes data movement. The computing chiplets retain information in an internal SRAM cache whenever possible, avoiding the slower transfer to HBM stacks. When HBM access is necessary, the chip directs data to physically nearby memory modules to reduce latency.

The system also employs data compression techniques to improve throughput. Maia 300 compresses data into lightweight formats such as MXFP4, which reduces the size of the information being processed. Microsoft executives Rani Borkar and Saurabh Dighe noted that these narrower representations allow for larger models and longer contexts within the same power envelope.

Supporting components include a device control chiplet that manages power flow and collects telemetry data. Two additional auxiliary chiplets handle the movement of data in and out of the accelerator. These elements ensure the main computing units can focus on processing tasks without managing peripheral operations.

Deployment will occur in specialized hardware configurations. Microsoft plans to install Maia 300 accelerators in trays containing four units each. These trays will then be placed into liquid-cooled racks that house a total of 96 accelerators, enabling multi-gigawatt scale operations.

Why it matters

The shift toward custom silicon reflects Microsoft’s strategy to reduce reliance on third-party hardware for core cloud services. By developing its own inference chips, the company aims to optimize cost and efficiency for AI workloads. This move could influence the broader market dynamics between cloud providers and hardware vendors like Nvidia.

The emphasis on energy efficiency is critical for large-scale data center operations. Processing 40 percent more tokens per megawatt directly impacts the operational costs and carbon footprint of AI services. As demand for AI inference grows, such efficiency gains become a significant competitive advantage for cloud infrastructure providers.

What to watch

• Monitor Microsoft’s integration of Maia 300 into Azure services and any performance benchmarks released.
• Observe how competitors like Nvidia respond to the increased efficiency of custom silicon accelerators.
• Track the adoption of liquid-cooled rack designs as a standard for high-density AI deployments.
• Watch for updates on the availability of HBM4 memory modules and their impact on chip pricing.

What to do and how to stay safe: Microsoft

• Review your organization’s AI inference workload requirements to determine if custom silicon solutions offer better cost efficiency.
• Evaluate the power and cooling infrastructure of your data centers to ensure they can support high-density, liquid-cooled accelerator racks.
• Assess the compatibility of your current AI models with new hardware architectures and memory configurations.
• Stay informed about vendor updates on chip availability and performance metrics to plan future procurement strategies.

Step-by-step guide: Software Updates Best Practices: Secure Patching Without Downtime

General security guidance from the Patch Gazette newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is the Azure Maia 300 chip?

It is a custom silicon accelerator developed by Microsoft, optimized for artificial intelligence inference workloads at multi-gigawatt scale.

How does Maia 300 compare to Nvidia’s Rubin accelerator?

Maia 300 has the same 288 gigabytes of HBM4 memory capacity but offers approximately 20 percent less memory bandwidth than Rubin.

What manufacturing process is used for Maia 300?

The computing chiplets are made using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometer node technology.