Abstract
Federated learning (FL) algorithms are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of devices with different platforms, using different datasets and data distribution, while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence. In this talk, we describe CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. Porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs. CoLExT is available to interested researchers.
Bio
Amandio is a Research Software Engineer at the SANDS Lab, KAUST, working under Prof. Marco Canini. He focuses on developing support systems for Machine Learning and is the system administrator of the group’s 40-node research testbed. He is devoted to improving workflows and reducing friction in order to spend more time on the fun parts.
