Inside an AI cluster: scale-up, scale-out and why the network is the product

Discussion of AI hardware tends to stop at the GPU. But a large model is split across many GPUs, and they spend part of every step exchanging data. How fast that exchange happens is set by networking, and it is why chip vendors now describe racks and clusters, not single chips, as the product.

Two networks, two jobs

Scale-up links the GPUs inside one rack so they behave like a single, very large accelerator. It needs the highest bandwidth and lowest latency, so it uses a dedicated interconnect. In Nvidia's design this is NVLink. Scale-out links racks to each other across the data center. It uses standard-style networking, Ethernet or InfiniBand, through a network card for each GPU.

The distinction matters because different parts of a workload use them differently. Splitting one model across GPUs makes frequent, large exchanges, which belong on scale-up. Spreading independent copies or batches across many racks tolerates the slower scale-out fabric.

The specifications

Nvidia's technical description of its Rubin platform, which Nvidia says it unveiled during its fiscal fourth quarter ^[3], lists six chips, of which four are networking components. The figures below come from that document ^[1].

Two notes on reading it. Scale-up is quoted in terabytes per second and scale-out in terabits per second (one byte is eight bits), so the table's columns are not directly comparable by eye; 1.6 Tb/s is 0.2 TB/s. All figures are Nvidia's own and describe the platform specification, not measured performance in a customer's data center ^[1].

Even allowing for the units (and for the fact that one figure is bidirectional), scale-up bandwidth per GPU is far above scale-out per GPU. That gap is the reason model-parallel work is kept inside the rack where possible.

Why optics and Ethernet are in the story

Nvidia's approach for scale-out is co-packaged optics, in which optical engines sit beside the switch silicon instead of in pluggable modules; the Spectrum-X Photonics switch uses 32 silicon-photonics engines of 3.2 Tb/s each, and Nvidia claims about 5x better network power efficiency and optical loss reduced from about 22 dB to about 4 dB ^[1]. Those are vendor claims for a new product.

On the open-standards side, the Ultra Ethernet Consortium released Specification 1.0 on June 11, 2025. It defines an Ethernet-based communication stack for AI and HPC covering NICs, switches, optics and cables, including RDMA over Ethernet, and the consortium says it scales to millions of endpoints. The consortium's stated aim is open, interoperable standards that avoid vendor lock-in ^[2]. How quickly the market adopts it remains to be seen.

The market signal

Nvidia's results on February 25 put numbers to the demand. Data Center revenue was $62.3 billion for the quarter ended January 25, up 75% year on year, and the company guided to about $78.0 billion in total revenue for the next quarter ^[3]. The release also cites a Meta partnership and says AWS, Google Cloud, Microsoft Azure and Oracle Cloud Infrastructure will be among the first to deploy Vera Rubin-based instances ^[3]. Networking is bought as part of the cluster, not as an accessory to it.

What it means for buyers

Most businesses will not build a cluster. They will rent capacity, and the network shows up as a limit on how well a workload scales across GPUs, and in prices for larger instances. Ask a cloud or GPU provider what interconnect sits between GPUs within a node and between nodes before you size a multi-GPU training or serving job.

For businesses: if a workload fits within one rack-scale or single-node system, it avoids the slower scale-out fabric entirely, so confirm interconnect specs before you commit to multi-node capacity.