OmNICCL: Zero-cost Sparse AllReduce with Direct Cache Access and SmartNICs

Abstract

AllReduce is a collective communication pattern commonly used in Distributed Deep Learning (DDL) and High Performance Comput- ing (HPC). Sparse AllReduce, which compresses the data transmit- ted, achieves significant acceleration on specific workloads. How- ever, compression introduces a non-negligible performance over- head. Therefore, we propose the OmNICreduce algorithm, an effi- cient inter-node sparse AllReduce method, as well as its implemen- tation, OmNICCL. It utilizes Direct Cache Access (DCA) to achieve zero-overhead lossless compression and employs SmartNICs for aggregation on the data plane. We demonstrate that our method can provide up to a 7.24× speedup over conventional dense AllRe- duce methods under a 100Gbps RoCEv2 network and 1.76-17.37× performance improvement over state-of-the-art implementations when performing sparse AllReduce.