Abstract

We study the network locality improvements that can be achieved by using co-packaged optics in data center and high-performance computing (HPC) networks. The increased escape bandwidth offered by co-packaged optics can enable switches with speeds of 51.2 Tb/s and beyond. From a network architecture perspective, the key advantages of introducing co-packaged optics at the switch points include the implementation of large-scale topologies of ${\gt}12{,}000$ end points with ${{4\times}}$ higher bisection bandwidth and the reduction of the required number of switches by ${\gt}{{40}}\%$ compared with state-of-the-art approaches. From a network operation perspective, improved network locality and faster operation can be achieved since the higher-radix switches can mitigate the impact of network contention. Placing applications under fewer leaf switches reduces the number of packets that cross the spine switches in a leaf-spine topology. The proposed scheme is evaluated via discrete-event simulations: we initially evaluate the network locality properties of the system by using virtual-machine traces from a production data center, and we subsequently quantify the performance improvements by simulating an all-to-all pattern for a variety of message sizes over a number of nodes. The results suggest that co-packaged optics form a promising solution for keeping up with bandwidth scaling in future networks. The virtual-machine analysis shows that large-scale applications can be placed under up to 50% fewer first-level switches, while the network analysis shows speedups of up to 7.1, which translates to execution time reductions of up to 26% and 42.7% for applications with communication ratios of 0.3 and 0.5, respectively.

© 2022 Optica Publishing Group

Toward lower-diameter large-scale HPC and data center networks with co-packaged optics

Pavlos Maniotis, Laurent Schares, Benjamin G. Lee, Marc A. Taubenblatt, and Daniel M. Kuchta
J. Opt. Commun. Netw. 13(1) A67-A77 (2021)

Exploring the benefits of using co-packaged optics in data center and AI supercomputer networks: a simulation-based analysis [Invited]

Pavlos Maniotis and Daniel M. Kuchta
J. Opt. Commun. Netw. 16(2) A143-A156 (2024)

Optics enabled networks and architectures for data center cost and power efficiency [Invited]

Marc Taubenblatt, Pavlos Maniotis, and Asser Tantawi
J. Opt. Commun. Netw. 14(1) A41-A49 (2022)

Intra-node high-performance computing network architecture with nanosecond-scale photonic switches [Invited]

Pavlos Maniotis, Nicolas Dupuis, Laurent Schares, Daniel M. Kuchta, Marc A. Taubenblatt, and Benjamin G. Lee
J. Opt. Commun. Netw. 12(12) 367-377 (2020)

Optical switching will innovate intra data center networks [Invited Tutorial]

Ken-ichi Sato
J. Opt. Commun. Netw. 16(1) A1-A23 (2024)