Schedule and Reading List

Date	Topic
Aug 26	Introduction and Course Overview [pptx]
Aug 29	ML Systems Research SysML: The New Frontier of Machine Learning Systems Read Chapter 1 of Principles of Computer System Design. You will need to be on campus or use the KAUST VPN to obtain a free PDF. A Few Useful Things to Know About Machine Learning A Berkeley View of Systems Challenges for AI Optional How to Read a Paper
Sep 2	Background Review The second week will be dedicated to review the developments occurred in the most recent wave of AI successes. We will overview the field of deep learning, and touch on convolutional neural networks, recurrent neural networks, generative adversarial networks. This is not a introductory class to ML. Students taking the class should already be familiar with most of the concepts reviewed. Below is a list of additional resources that would be helpful to brush up some concepts. Kevin Murphy’s Textbook Introduction to Machine Learning. This provides a very high-level overview of machine learning. You should probably know all of this. Stanford CS231n Tutorial on Neural Networks. Module 1 is especially recommended for a quick crash course in machine learning and some of the techniques encountered in this class. Google’s Machine Learning Crash Course. This includes several short video lectures giving a practical introduction to machine learning with TensorFlow API. 3Blue1Brown’s Neural Network Basics For a more complete array of resources, see The ML Hub @ KAUST’s page on the topic.
Sep 5	Background Review We continue with a background review and initiate the discussion of ML Systems as Software 2.0. Software 2.0 Blog Post Software 2.0 and the Paradigm Shift in Programming ML Systems
Sep 9	Machine Learning Platforms [pptx] Hidden Technical Debt in Machine Learning Systems TFX: A TensorFlow-Based Production-Scale Machine Learning Platform Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective Optional On Challenges in Machine Learning Model Management Democratizing Production-Scale Distributed Deep Learning Michelangelo FBLearner Flow Towards Unified Data and Lifecycle Management for Deep Learning Practical MLflow Weights & Biases
Sep 12	Discussion of Machine Learning Platforms
Sep 16	Model Development and Training Frameworks MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems TensorFlow: A System for Large-Scale Machine Learning
Sep 19	Discussion of Model Development and Training Frameworks
Sep 23	No class (Holiday)
Sep 26	Prediction Serving Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications TensorFlow-Serving: Flexible, High-Performance ML Serving
Sep 30	Hyperparameter Tuning and AutoML Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Efficient Neural Architecture Search via Parameter Sharing DARTS: Differentiable Architecture Search Optional Neural Architecture Search: A Survey Designing Neural Network Architectures using Reinforcement Learning Efficient and Robust Automated Machine Learning Large-Scale Evolution of Image Classifiers A Generalized Framework for Population Based Training Cyclical Learning Rates for Training Neural Networks SGDR: Stochastic Gradient Descent with Warm Restarts MSR Advanced Machine Learning: Debadeepta Dey, Neural Architecture Search
Oct 3	Discussion of Hyperparameter Tuning and AutoML
Oct 7	Distributed Model Training PipeDream: Generalized Pipeline Parallelism for DNN Training Scaling Distributed Machine Learning with In-Network Aggregation Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD Optional A Generic Communication Scheduler for Distributed DNN Training Acceleration Scaling Distributed Machine Learning with the Parameter Server Natural Compression for Distributed Deep Learning Beyond Data and Model Parallelism for Deep Neural Networks Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Oct 10	Discussion of Distributed Model Training
Oct 14	No class
Oct 17	No class
Oct 21	Hardware Acceleration for Machine Learning A Configurable Cloud-Scale DNN Processor for Real-Time AI Eyeriss: A Spatial Architecture for Energy-Efficient Dataflowfor Convolutional Neural Networks Plasticine: A Reconfigurable Architecture For Parallel Patterns Optional Efficient Processing of Deep Neural Networks: A Tutorial and Survey In-Datacenter Performance Analysis of a Tensor Processing Unit
Oct 24	Mid-semester Project Presentations
Oct 28	No class (Mid-semester break)
Oct 31	Debugging and Interpretability “Why Should I Trust You?”: Explaining the Predictions of Any Classifier The Mythos of Model Interpretability MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis Optional TBA Practical InterpretML
Nov 4	Testing and Verification DeepXplore: Automated Whitebox Testing of Deep Learning Systems AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation Reluplex: An Efficient SMT Solver for VerifyingDeep Neural Networks Optional Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatment
Nov 7	Discussion of Testing and Verification
Nov 11	No class
Nov 14	No class
Nov 18	Model Compilation TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Supporting Very Large Models using Automatic Dataflow Graph Partitioning TASO: Optimizing Deep Learning Computation with Automated Generation of Graph Substitutions Optional Astra: Exploiting Predictability to Optimize Deep Learning Learning to Optimize Tensor Programs SysML 19 Keynote: Ofer Dekel, Compiling AI for the Edge Practical Embedded Learning Library (ELL)
Nov 21	Discussion of Model Compilation
Nov 25	Machine Learning Applied to Systems The Case for Learned Index Structures Learning Scheduling Algorithms for Data Processing Clusters
Nov 28	Discussion of Machine Learning Applied to Systems
Dec 2	Final Project Presentations
Dec 5	Final Project Presentations