Data Center Directions: Servers and Infrastructure for Generative AI Fuel Future Growth

The allure and potential of AI to enterprises and hyperscalers has captured the imagination of businesses worldwide, but what must they consider to implement it in their data centers? Specifically, they need an infrastructure that meets the compute, networking, and storage requirements of their AI implementations and workloads.

To that end, a look at infrastructure spending shows the impact of AI on enterprises today. Driven largely by sales of AI-powered accelerated servers, global data center purchases surged 38% year over year in the first half of 2024, according to a report covering all aspects of data center equipment spending from the Dell’Oro Group, a market research and analysis firm.

AI Servers and Infrastructure Spending Fuel Future Growth

Looking ahead to the full year 2024, data center CapEx is projected to increase by 35 percent to over $400 billion, with spending on AI servers and infrastructure leading the way, according to the report. Hyperscale cloud service providers are racing to expand their AI offerings, creating a strong demand for these specialized systems.

“AI infrastructure adoption is still in its initial stages for enterprises, according to Baron Fung, Senior Research Director at Dell’Oro. “Organizations are figuring out how much to invest, as the return on investment (ROI) depends on the scale of AI models they plan to support on-premises and how well they can utilize those models. “

In the short term, he continued, using public cloud services may make more sense while enterprises refine their usage models. “One of the most common questions we encounter is how companies can monetize AI investments and achieve their desired returns.”

Infrastructure Progress Even with The Challenges

Despite these challenges, AI infrastructure deployments are gaining traction, particularly outside of hyperscalers. Server OEMs such as Dell, Supermicro, and HPE, which cater to non-hyperscaler markets, have significantly increased their AI system sales.

Beyond high costs, enterprises must address limitations in power and data center capacity, as AI infrastructure often has different power and cooling needs compared to traditional IT systems, explained Fung.

Organizations investing in AI for private data centers include Tier 2 cloud providers and enterprises in industries such as finance and high-tech manufacturing.

"While some may deploy large AI clusters, most are implementing smaller-scale systems (e.g., a few racks) for tasks like fine-tuning models or inferencing; others are building private AI clouds to ensure sensitive data remains isolated from the broader internet," explained Fung.

Data Center AI Infrastructure Spending Trends

While the report covered advancements in many types of data center equipment spending, AI/accelerated service CapEx is more important than the rest as it is growing faster, followed by physical infrastructure. Potential customers should realize that all the pieces of the data center need to be deployed in synchronization.

This rapid increase was primarily driven by the rise of accelerated servers, which are critical for generative AI applications. This marks the fourth consecutive quarter of triple-digit year-over-year revenue growth in accelerated server shipments.

Driving Accelerated Computing

AI has performance requirements that are different from those of traditional HPC. As such, hardware accelerators and co-processors for AI are designed to complement a high-performance CPU or other processor for the purpose of accelerating a specific function or workload.

To that point, accelerated computing utilizes specialized hardware like GPUs, ASICs, DPUs, TPUs, and FPGAs to execute computations more efficiently than CPUs, enhancing speed and performance. It is especially beneficial for tasks that can be parallelized, such as high-performance computing, deep learning, ML, and AI.

Server upgrades, particularly to 4th and 5th generation CPU platforms, have been long overdue, and despite ongoing global economic uncertainties, demand for these systems is expected to rise, says the report.

New Data Center Physical Infrastructure (DCPI)

The data center physical infrastructure (DCPI) market outperformed expectations in the first half of this year. Growth was attributed to new data center construction with AI-related design modifications to support increasing rack power densities, according to the report. North America led the way with the fastest growth rate, while revenues in the Asia-Pacific region, excluding China, also saw double-digit growth.

Server and storage system component revenues reached record highs in the first two quarters of the year. The rapid growth of accelerators, which include GPUs and custom accelerators, as well as memory and storage drives, was a key factor behind this revenue increase. “Generative AI applications were the primary drivers of accelerated server demand, but higher commodity prices, particularly for memory and storage drives, also contributed to the revenue surge, explained Fung.

With respect to component revenues, the report forecasted them to double in 2024, fueled by the increased deployment of specialized processors such as accelerators and SmartNICs. Commodity component prices, such as memory and storage drives, are expected to rise throughout the year.

Another element of data center infrastructure that needs to be updated for AI is Ethernet switches. It might be several more quarters for this segment to grow due to inventory issues. "We project the recovery in the Ethernet switch market to be led by the hyperscale cloud SPs on both networks for general-purpose computing and AI clusters," said Fung.

The Road Ahead on Infrastructure for AI Data Center Directions

AI impacts far more than data center operations of users and service providers operations, especially in the area of network management. And since it’s a fast-evolving area, learning more about AI is solid advice. Is your infrastructure ready?