NVIDIA's 25-year roadmap has been leaked! Old Huang is betting big on B100 to beat AMD, and the secret weapon X100 has been revealed.

Recently, foreign media exposed a new GPU roadmap from NVIDIA, revealing the full details of the most powerful B100 technology ever, with the most mysterious X100 set to be released in 2025.

NVIDIA, the AI hardware leader, has held its position for too long!

Now, major technology companies are eagerly waiting to overthrow its dominance.

Of course, NVIDIA will not sit idly by.

Recently, foreign media SemiAnalysis revealed a hardware roadmap for NVIDIA in the coming years, including the highly anticipated H200, B100, and "X100" GPUs.

Along with this, some hardcore information has also been leaked, including NVIDIA's process technology plans, the speed/capacity of HBM3E, PCIe 6.0, PCIe 7.0, NVLink, and the 1.6T 224G SerDes plan.

If these plans are successfully implemented, NVIDIA will continue to crush its competitors.

Of course, being the dominant player is not easy—AMD's MI300, MI400, Amazon's Trainium2, Microsoft's Athena, and Intel's Gaudi 3 will not make it easy for NVIDIA.

Get ready, high energy is coming!

NVIDIA, not just aiming to be the hardware leader

Google has long been laying the groundwork for its AI infrastructure. Their TPUv5 and TPUv5e can be used for internal training and inference, as well as for external customers such as Apple, Anthropic, CharacterAI, and MidJourney.

Google is not the only threat to NVIDIA.

In terms of software, Meta's PyTorch2.0 and OpenAI's Triton are also rapidly developing, enabling other hardware vendors to achieve compatibility.

Currently, the gap in software still exists, but it is not as significant as before.

In the software stack, AMD's GPU, Intel's Gaudi, Meta's MTIA, and Microsoft's Athena have all made some progress.

Although NVIDIA still maintains its hardware leadership position, the gap is narrowing and will continue to do so at an increasing pace.

NVIDIA's H100 will not hold the spotlight for too long.In the coming months, both AMD's MI300 and Intel's Gaudi 3 will launch hardware products that surpass the H100 in terms of technology.

In addition to formidable opponents like Google, AMD, and Intel, there are also some companies that have put significant pressure on NVIDIA.

Although these companies may temporarily lag behind in hardware design, they receive subsidies from major players behind the scenes. NVIDIA has been suffering for a long time, and these companies hope to break NVIDIA's monopoly on huge profits in HBM.

Amazon's upcoming Trainium2 and Inferentia3, as well as Microsoft's Athena, are all investments that have been planned for years.

With fierce competition from rivals, NVIDIA certainly won't sit idly by.

In the eyes of foreign media SemiAnalysis, NVIDIA is one of the "most skeptical companies in the industry" in terms of management style and strategic decision-making.

And in Huang Renxun, there is a spirit reminiscent of Andy Grove.

Success leads to complacency. Complacency leads to failure. Only the paranoid survive.

In order to firmly hold the top spot, NVIDIA is ambitious and has adopted a multi-pronged adventurous strategy.

They are no longer interested in competing with Intel and AMD in traditional markets, but instead want to become technology giants like Google, Microsoft, Amazon, Meta, and Apple.

Behind NVIDIA's DGX Cloud, software, and acquisition strategies in non-semiconductor fields is a grand plan.

Latest Details of the Roadmap Revealed!

The important details of NVIDIA's latest roadmap have been revealed.

This includes the networks, memory, packaging, and process nodes used, various GPU and SerDes options, PCIe6.0, collaborative packaging optical devices, and optical switch details.

Clearly, under the pressure of competition from Google, Amazon, Microsoft, AMD, and Intel, NVIDIA has accelerated the development of B100 and "X100" overnight.

B100: Launch Time Above All

According to internal sources, NVIDIA's B100 will enter mass production in the third quarter of 2024, with some early samples shipping in the second quarter of 2024.From the perspective of performance and TCO, whether it's Amazon's Trainium2, Google's TPUv5, AMD's MI300X, Intel's Gaudi 3, or Microsoft's Athena, they are all no match for it.

Even considering the subsidies obtained from design partners, AMD, or TSMC, they are still no match.

In order to quickly bring B100 to the market, NVIDIA has made a lot of compromises.

For example, NVIDIA originally wanted to set the power consumption at a higher level (1000W), but in the end, they chose to continue using the 700W of H100.

This way, when B100 is launched, it can continue to use air cooling technology.

In addition, in the early series of B100, NVIDIA will also insist on using PCIe 5.0.

The combination of 5.0 and 700W means that it can be directly plugged into existing H100 HGX servers, greatly improving the supply chain capability and enabling earlier mass production and shipment.

The reason for deciding to stick with 5.0 is also partly because AMD and Intel are still far behind in PCIe 6.0 integration. And even NVIDIA's own internal team is not ready to use PCIe 6.0 CPUs.

In addition, they will use faster C2C-style links.

In the future, ConnectX-8 will be equipped with an integrated PCIe 6.0 switch, but no one is ready for it yet.

It is reported that Broadcom and AsteraLabs will not be ready for mass production of PCIe 6.0 retimers until the end of the year, and considering the size of these boards, more retimers will be needed.

This also means that the initial B100 will be limited to 3.2T, and the speed when using ConnectX-7 will only be 400G, not the 800G per GPU claimed by NVIDIA in the PPT.

If air cooling, power, PCIe, and network speed remain unchanged, both manufacturing and deployment will be easy.

Later, NVIDIA will release a water-cooled version of B100 with a power consumption of 1,000W+.This version of B100 will provide a complete 800G network connection for each GPU through ConnectX-8.

For Ethernet/InfiniBand, these SerDes are still 8x100G.

Although the network speed of each GPU has doubled, the radix has been halved because they still need to pass through the same 51.2T switch. The 102.4T switch will no longer be used in the B100 generation.

Interestingly, there are rumors that the NVLink component on B100 will use 224G SerDes, which would undoubtedly be a huge advancement if NVIDIA can achieve this.

Most industry insiders believe that 224G is not reliable and cannot be achieved by 2024, except for NVIDIA.

It should be noted that whether it is Google, Meta, or Amazon, their goal of mass production for 224G AI accelerators is set for 2026/2027.

If NVIDIA can achieve this by 2024/2025, it will undoubtedly outperform its competitors.

It is reported that B100 is still based on TSMC's N4P technology, not the 3nm process.

Obviously, TSMC's 3nm process is not mature enough for such a large chip size.

According to the substrate size revealed by NVIDIA's substrate supplier Ibiden, NVIDIA seems to have switched to a design consisting of 2 single-chip MCMs, including 8 or 12 HBM stacks.

SambaNova and Intel's chips next year have adopted similar macro designs.

The reason why NVIDIA did not use hybrid bonding technology like AMD is because they need mass production, and cost is a major concern for them.

According to SemiAnalysis estimates, the memory capacity of these two B100 chips will be similar to or higher than AMD's MI300X, reaching 24GB stacking.

The speed of the air-cooled version of B100 can reach 6.4Gbps, while the liquid-cooled version may reach up to 9.2Gbps.

In addition, NVIDIA also showcased GB200 and B40 in its roadmap.

Both GB200 and GX200 use G, which is obviously a placeholder because NVIDIA will launch a new CPU based on the Arm architecture. Grace will not be used for a long time.B40 is likely to be only half of B100, with only one N4P chip and a maximum of 4 or 6 layers of HBM. Unlike L40S, this makes sense for small model inference.

"X100": A Fatal Blow

The most eye-catching part of the leaked roadmap is Nvidia's "X100" schedule.

Interestingly, it aligns perfectly with AMD's current MI400 schedule. Just one year after the launch of H100, AMD released the MI300X strategy.

AMD's packaging for MI300X is impressive, as they have packed in more computing power and memory, hoping to surpass the H100 from a year ago and outperform Nvidia in pure hardware.

Nvidia has also realized that their biennial release of new GPUs gives their competitors a great opportunity to seize the market.

Feeling the pressure, Nvidia is now accelerating its product cycle to once a year, leaving no chance for its competitors. For example, they plan to launch the "X100" in 2025, just one year later than B100.

Of course, the "X100" is not yet in mass production (unlike B100), so everything is still up in the air.

It's worth noting that in the past, Nvidia never discussed products beyond the next generation. This is unprecedented.

Moreover, the name "X100" is highly likely to change.

Nvidia has always followed the tradition of naming their GPUs after outstanding female scientists such as Ada Lovelace, Grace Hopper, and Elizabeth Blackwell.

As for the "X," the only logical choice would be Xie Xide, who researches semiconductors and metal band structures. However, considering her status, the probability is probably low.

Supply Chain Master: Huang's Bold Gamble

Since its inception, Nvidia has been actively pushing for control over the supply chain to support its ambitious growth targets.

They are not only willing to take on non-cancelable orders - up to $11.15 billion in procurement, capacity, and inventory commitments, but also have $3.81 billion in prepayment agreements.

It can be said that no other supplier can compare.

And Nvidia's track record has shown more than once that they can creatively increase supply in times of shortage.

2007 Dialogue between Huang Renxun and Zhang Zhongmou

In 1997, when Zhang Zhongmou and I met, NVIDIA, with only 100 employees, achieved a revenue of 27 million US dollars that year. You may not believe it, but Zhang Zhongmou used to make phone calls and even visit in person to promote our products. And I would explain to Zhang Zhongmou what NVIDIA does and how big our chip size needs to be, and it will continue to grow every year. Later, NVIDIA produced a total of 127 million wafers. Since then, NVIDIA has grown by nearly 100% every year, until now. That is, in the past 10 years, the compound annual growth rate has reached about 70%.

At that time, Zhang Zhongmou couldn't believe that NVIDIA needed so many wafers, but Huang Renxun persisted.

NVIDIA achieved great success through bold attempts in the supply chain. Although they occasionally had to write off billions of dollars worth of inventory, they still obtained positive returns from excessive orders.

This time, NVIDIA directly seized most of the upstream components of the GPU-

They placed very large orders with the three HBM suppliers, SK Hynix, Samsung, and Micron, squeezing out the supply of everyone except Broadcom and Google. At the same time, they also acquired most of TSMC's CoWoS supply and Amkor's production capacity.

In addition, NVIDIA also fully utilized downstream components required for HGX boards and servers, such as retimers, DSPs, and optical devices.

If suppliers ignore NVIDIA's requirements, they will face Huang's "carrot and stick" approach-

On the one hand, they will receive unimaginable orders from NVIDIA; on the other hand, they may be excluded from the existing supply chain by NVIDIA.

Of course, NVIDIA will only use commitments and non-cancelable orders when suppliers are crucial and cannot be eliminated or diversified.

Each supplier seems to think that they are the winner in the AI field, partly because NVIDIA has placed a large number of orders with all suppliers, and they all believe that they have won the majority of the business. But in fact, it is only because NVIDIA's growth rate is too fast.

Returning to the market dynamics, although NVIDIA's goal is to achieve over 70 billion US dollars in data center sales next year, only Google has sufficient capacity upstream- with more than one million devices. AMD's total capacity in the AI field is still very limited, at most a few hundred thousand units.

Business Strategy: Potential Anti-competition

It is well known that NVIDIA is leveraging the huge demand for GPUs to promote and cross-sell products to customers.

There is a lot of information in the supply chain that reveals that NVIDIA will provide preferential allocation to certain companies based on a series of factors. These factors include but are not limited to: diversified procurement plans, independent development of AI chip plans, and purchases of NVIDIA's DGX, NIC, switches, and/or optical equipment.In fact, Nvidia's bundling sales strategy has been very successful. Despite being a small-scale fiber optic transceiver supplier before, their business volume doubled in one quarter, and it is expected that next year's shipments will exceed $1 billion, far exceeding the growth rate of their own GPU or network chip business.

These strategies can be said to be quite meticulous.

For example, the only way to achieve 3.2T network and reliable RDMA/RoCE on Nvidia's system is to use Nvidia's NIC. Of course, on the one hand, it is also because Intel, AMD, and Broadcom's products are not competitive enough and still remain at the level of 200G.

And through the management of the supply chain, Nvidia also shortens the delivery cycle of 400G InfiniBand NIC compared to 400G Ethernet NIC. And these two NICs (ConnectX-7) are actually identical in chip and circuit board design.

The reason for this is Nvidia's SKU configuration, not the actual supply chain bottleneck, which forces companies to purchase more expensive InfiniBand switches instead of standard Ethernet switches.

This is not all, just look at how fascinated the supply chain is with L40 and L40S GPUs, and you will know that Nvidia has also manipulated the allocation. In order to win more allocations of H100, OEM manufacturers need to purchase more L40S.

This is also consistent with Nvidia's operations in the PC field. Notebook manufacturers and AIB partners must purchase a larger quantity of G106/G107 (mid/low-end GPUs) in order to obtain more scarce and higher-profit G102/G104 (high-end and flagship GPUs).

As a complement, people in the supply chain have also been instilled with the idea that L40S is better than A100 because it has higher FLOPS.

But in reality, these GPUs are not suitable for LLM inference because their memory bandwidth is less than half of A100's, and they do not have NVLink either.

This means that running LLM on L40S and achieving good TCO is almost impossible, unless it is a very small model. And large-scale processing will also result in token/s allocated to each user being almost unusable, rendering the theoretically high FLOPS useless in practical applications.

In addition, Nvidia's MGX modular platform, while saving the arduous work of server design, also reduces the profit margin for OEMs.Dell, HP, Lenovo, and other companies are clearly resistant to MGX, but companies such as AMD, Quanta, Asus, and Gigabyte are rushing to fill this gap and commercialize low-cost "enterprise artificial intelligence" products.

And these OEM/ODM companies participating in the hype of L40S and MGX can also obtain better distribution of mainstream GPU products from NVIDIA.

Co-Packaged Optics (CPO)

NVIDIA also attaches great importance to CPO.

They have been researching various solutions, including solutions from Ayar Labs, as well as solutions they have obtained from Global Foundries and TSMC.

Currently, NVIDIA has inspected several start-ups' CPO solutions, but has not made a final decision yet.

Analysis believes that NVIDIA is likely to integrate CPO into the NVSwitch of "X100".

Because direct integration into the GPU itself may be too expensive and difficult in terms of reliability.

Optical Circuit Switch (OCS)

One of Google's biggest advantages in artificial intelligence infrastructure is its optical circuit switch.

Clearly, NVIDIA is also pursuing something similar. Currently, they have contacted multiple companies in the hope of collaborating on development.

NVIDIA realizes that Fat Tree has reached its limit in terms of further expansion, so it needs another topology.

Unlike Google's choice of 6D Torus, NVIDIA is more inclined to adopt the Dragonfly structure.

It is understood that NVIDIA is still far from shipping OCS, but they hope to be closer to this goal by 2025, but it is highly unlikely to achieve it.OCS + CPO is the holy grail, especially when OCS can achieve packet switching, it will directly change the rules of the game.

However, no one has demonstrated this ability yet, not even Google.

Although Nvidia's OCS and CPO are still just two sets of PPTs in the research department, analysts believe that CPO will further advance towards productization between 2025 and 2026.

Source: New Intelligence, original title: "Nvidia's 25-year roadmap leaked! Huang Renxun bets big with B100, secret weapon X100 exposed"