Qualcomm is pushing hard on NPU, focusing on low power consumption and high bandwidth, aiming to become the "NVIDIA of the edge"!

Title: The Commencement of the 2024 Intelligent Terminal Battle: Qualcomm's AI Revolution

Author: Ge Jiaming

Source: Hard AI

In 2024, the battle of intelligent terminals kicks off, with Qualcomm, the "mobile chip overlord," making its move.

On March 4th, at Qualcomm's annual shareholders meeting, Qualcomm CEO Cristiano Amon boldly stated that Qualcomm is currently leading a revolution in AI smartphones and actively expanding into the AI PC field, competing with rivals such as Intel, AMD, and NVIDIA. In the coming months, Qualcomm will launch the Snapdragon X Elite chip, surpassing competitors like Intel in AI workloads. Amon expressed:

"The Snapdragon X Elite is our first chip featuring custom Qualcomm technology, setting a new industry benchmark in AI performance and battery life. We look forward to bringing revolutionary changes to devices with its generative AI and Copilot experiences."

Qualcomm believes that although the current focus of AI processing is mainly on the cloud, including ChatGPT, Microsoft Copilots, and Google Gemini, deploying AI on the edge has become crucial for the scalable expansion and practical application of AI due to issues such as high cloud inference costs, energy consumption, reliability, latency, user privacy, and data security.

Simultaneously, deploying AI on the edge can create personalized local knowledge bases for users by running "personal large models" locally, achieving a "one thousand faces, one thousand experiences" personalized experience. In the future, edge AI will become the preferred way of human-machine interaction.

Media analysis points out that the global consumer electronics market can also emerge from its slump with the implementation of edge AI. IDC predicts that 2024 will be the year of AI PCs, with an expected penetration rate of 85% by 2027. Boston Consulting Group forecasts a global AIPC penetration rate of 80% by 2028.

According to Qualcomm, as edge AI rapidly advances, the industry needs to balance performance, power consumption, efficiency, programmability, and area, with NPU being a processing unit specially designed for low-power accelerated AI inference and one of Qualcomm's strengths in chip design:

"By adopting a new power supply system, the NPU can adapt power according to workloads, balancing high performance and low power consumption. This technology upgrade also introduces micro-slice inference, further enhancing acceleration effects, providing users with superior performance and longer battery life."

Amon believes that Qualcomm's leading position in wireless connectivity and communication technology is often overlooked by the market, possessing the ability to provide high bandwidth, low latency, high energy efficiency, and high precision connectivity. The Snapdragon X80 can improve data transfer speed, coverage, and efficiency while reducing latency. When connecting to millimeter-wave networks, power consumption can be reduced by 10%, and positioning accuracy can be increased by 30%.

Why is NPU the key to AI implementation on the edge?

Recently, Qualcomm officially released the Chinese version of the technical white paper "Enabling Terminal-side Generative AI through NPU and Heterogeneous Computing," revealing the three major advantages of NPU.

The report points out that NPU (Neural Processing Unit) is a processor based on domain-specific architecture (DSA) technology, which is more suitable for neural network operations from a hardware architecture perspective compared to general processors like CPU and GPU, and can be specifically used to accelerate AI with hardware.

AI workloads mainly consist of neural network layer calculations composed of scalar, vector, and tensor mathematics, as well as nonlinear activation functions. An excellent NPU design can make the correct design choices for processing these AI workloads and remain highly consistent with the direction of the AI industry.

Qualcomm emphasizes that their NPU's differentiation advantages lie in system-level solutions, custom design, and rapid innovation. By custom designing the NPU and controlling the Instruction Set Architecture (ISA), Qualcomm can quickly evolve and expand the design to address bottlenecks and optimize performance.

Specifically, Qualcomm's system-level solutions consider the architecture of each processor, SoC system architecture, and software infrastructure to optimize the overall AI solution. Furthermore, by custom designing the NPU and controlling the Instruction Set Architecture, Qualcomm can quickly evolve and expand the design to address bottlenecks and optimize performance.

Having an NPU is not enough; heterogeneous computing is equally important. Heterogeneous computing can leverage the strengths of each processor, such as NPU's proficiency in scalar, vector, and tensor mathematical operations, making it more suitable for core AI workloads.

Qualcomm's AI Engine is Qualcomm's heterogeneous computing architecture, and currently, almost all Snapdragon mobile chips are equipped with Qualcomm's AI Engine.

Is Qualcomm aiming to become the "NVIDIA of edge AI"?

Some media analysis points out that NVIDIA's dominant position in the chip industry has not yet extended to the edge AI market, where competition is intensifying.

Qualcomm CEO Cristiano Amon stated that Qualcomm's processors are widely used in global Android smartphones:

"We are leading a revolution to empower global smartphone users with generative AI. With our latest Snapdragon mobile platform, we will continue to strengthen our leadership position in the high-end Android device market and provide significantly enhanced artificial intelligence processing performance."

It is reported that the Snapdragon 8 Gen 3, as the company's latest flagship product, has been adopted by leading smartphone manufacturers such as Samsung and plays a key role in their latest Galaxy S24 series smartphones. The processor not only enhances the overall performance of smartphones but also optimizes generative artificial intelligence functions, such as Samsung's Generative Edit tool, allowing users to edit photos in unprecedented ways, easily erasing or moving objects in photos.

Amon believes that the market often overlooks Qualcomm's profound heritage and leadership in the field of communication. Edge AI will demand higher data and bandwidth, and Qualcomm's leading advantages in modems and wireless technologies are often overlooked.

For Edge AI, high-bandwidth connections mean lower latency, greater availability, and better coverage. High-bandwidth connections can support edge devices in executing machine learning tasks with faster response times, improving data transfer efficiency, and ensuring connection stability and reliability.

Qualcomm recently released its latest flagship 5G modem - Snapdragon X80, equipped with a tensor accelerator to enhance data transfer speed, coverage, and efficiency while reducing latency. When connected to mmWave networks, power consumption can be reduced by 10%, and positioning accuracy can be improved by 30%.

In addition, at MWC2024, Qualcomm announced the launch of the new Qualcomm AI Hub, a central resource hub for developers to build AI applications based on Snapdragon or Qualcomm platforms.

Qualcomm's Senior Vice President of Technology and General Manager of Technology Planning and Edge Solutions, Madhav Chakravarthy, said: "With the launch of the third-generation Snapdragon 8 for smartphones and Snapdragon X Elite for PCs, Qualcomm has initiated the commercialization of terminal-side AI. Now, with Qualcomm AI Hub, Qualcomm will empower developers to fully unleash the potential of these cutting-edge technologies and create attractive AI-enabled applications."

Ryan Shrout, an analyst at Shrout Research, stated that the AI PC trend will arrive in 2025, and Qualcomm's early layout gives it a significant advantage in this competition. Although Qualcomm currently has a minimal share in the AI PC market, its Snapdragon X Elite platform provides over 4 times the AI performance of Intel or AMD chips. The first laptops equipped with X Elite chips will be launched in June 2024.

By 2024, the market share battle for Edge AI among Intel, NVIDIA, AMD, ARM, and Qualcomm is unstoppable.