Project

The project of the class requires original work and is intended to give an introduction to the research process. We would like you to start early, send updates often to the instructor, and manage the project to finish by the end of the term. You may choose to do either:

  • an original research project in the area of computer systems or

  • reproduce a result from a systems paper that hasn’t been reproduced in previous iterations of CS345.

The idea of reproduction project is inspired by the gradute course in computer networks at Stanford from which we borrow structure and ideas. Examples of reproduction projects from that course are on the Reproducing Network Research site.

Reproduction project

For a reproduction project, here are some sample questions your project might try to answer:

  • Does the primary result in the paper hold up? Can you generate a plot or graph that looks the same as one of the main figures in the original paper, and explain where it came from?

  • What kind of “reproduction” is appropriate? Most plots can be reproduced if you simply re-plot the authors’ original data on the same axes! Can the data be generated afresh? Are the assumptions behind the data-generating process, and the way the system was compared with prior work, realistic?

  • What was difficult to reproduce? Were you able to package the contribution of the paper in a way that makes it easier for future “reproducteurs” to reproduce the major results?

  • What happens if you vary a parameter the original experimenter didn’t consider?

  • Having reproduced the primary result, can you now extend or improve the work?

  • Can you find a simplification to the system that still preserves most of the claimed improvements? If so, this can indicate that the original authors are mistaken about why their system outperforms prior work.

See the “Searching for Interesting Papers” section below for ideas on where to look for recent or interesting papers.

Original project

An original project can be a proof of concept of a novel system, an extension of existing work, or any research project that you could realistically extend to a conference-quality paper. You will have to complete substantive work on an instructor-approved problem and have original contribution. Surveys are not permitted as projects; instead, each project must contain a survey of background and related work.

Much of this project is intentionally underspecified, and you should pick a project that you have sufficient time and experience to accomplish. Depending on what you propose, your project could require using a profiler, modifying the Linux kernel, implementing a custom simulator, learning a new scripting language, proving an interesting new result, or coordinating a software development effort across multiple group members. You will likely need to read research papers outside of the class reading list or the particular paper you are trying to reproduce. (Check the references of any paper for the most closely related prior work, and read or skim the prior work!)

For a new sytem, here are questions you would want to answer:

  • How and why did you design the system the way you did?

  • How does the system perform in comparison with prior work in similar areas?

  • Why does it perform as it does (i.e., which design decisions account for what benefits)?

  • What are the system’s weaknesses and limitations?

Final report

Your final deliverable will be a 8 page report about your project. You will be expected to:

  • Introduce the project, placing it in context, giving an overview, and summarizing your major contributions or findings.

  • Discuss prior work and the landscape of the area.

  • Explain your methods and/or the design of your system.

  • Present the experimental setup in detail (assuming there are experiments, rather than proofs).

  • Describe the results in figures and text.

  • Discuss the meaning of the results and what the reader should take away.

  • Describe the limitations and weaknesses of your system, measurement, or effort at replication.

  • Propose a plan for future work in the area. What would you do next if you had more time?

For reproduction projects, you will post your final writeup on the class site.

Be wise and mindful of your (other) deadlines and other commitments before you decide your deliverable.

Milestones and deliverables

You must meet the following milestones (unless otherwise specified in future announcements) to ensure a high-quality project at the end of the semester:

Turn in a 2-page draft proposal (including references) by February 15.

  • Before you submit your proposal, we encourage you to talk to us (e.g. in office hours, or in lecture).

  • Pick either:

    • a paper to reproduce OR

    • an original project idea

  • Write a proposal describing:

    • Who is in your group. Remember to include the names and KAUST email addresses of the group members.

    • The particular results you would like to replicate, or the overall goal of your original project.

    • Details about what you will do.

    • A brief outline of incremental steps you will need to do to finish the project as well as a timeline. The goal here is to convince both us and yourself that your project is neither too small nor too big.

Keep revising your initial idea and incorporate instructor feedback. However, your team and project proposal must be finalized and approved on or before February 22.

Each group must submit a 4-page mid-semester progress report by March 28 and present mid-semester progress during class hours on March 29.

  • Should have the structure of the final report.

  • Complete introduction written.

  • Status of the project.

  • Plan for the remaining time.

Each group must present their final results during a presentation or poster session on May 6.

Each group must turn in an 8-page final report and your code via email on or before 11:59PM on May 8.

  • The report must be submitted as a PDF file, with formatting similar to that of the papers you’ve read in the class.

  • The self-contained (i.e., include ALL dependencies) code must be submitted as a zip file. Each zip file containing the code must include a README file with a step-by-step guide on how to compile and run the provided code.

Regular check-ins and meetings

During the course of the project, you’ll have one mandatory meeting with the us around the time the milestone is due. The goal is to make sure you’re on track to complete the project. In addition, we will expect you to send an update (via Piazza) each week to the intructor describing:

  1. what you did this week,

  2. which papers you read this week, and

  3. what you need to do next week to stay on track.

These updates will not be graded; the idea is to make sure you are making incremental progress. You may also visit the instructor by appointment.

Grading

You’ll be graded out of 50 points total. The grades will be based on the following:

  • 10% Research proposal

  • 15% Mid-semester checkpoint

  • 25% Final report

More than the success of the final outcome, we look for how thorough your efforts have been.

Searching for interesting papers

We recommend searching for interesting papers published in recent years in top systems and networking conferences, or by searching through the references section of another paper in the field. Relevant venues include:

  • OSDI: major conference in computer systems

  • SOSP: alternates years with OSDI

  • NSDI: networking and systems (overlap with OSDI)

  • MLSys: work at the intersection of systems and AI/ML

  • SIGCOMM: major conference in computer networking

  • CoNEXT: conference in computer networking

  • USENIX ATC: conference in computer systems

  • EuroSys: conference in computer systems

  • Systems for ML and Open Source Software: workshop at NeurIPS 2018 with exciting but half-baked ideas

  • Marco’s SysML reading list

Other places would be the website of authors whose papers you’ve read in class and liked.