Project

The project of the class requires to reproduce a result from a systems security paper that hasn’t been reproduced in previous iterations of CS230. This project will give you a chance to learn something new about an exciting piece of work, and will give you an introduction to the research process. We would like you to start early, send updates often to the instructor, and manage the project to finish by the end of the term.

Much of this project is intentionally underspecified, and you should pick a project that you have sufficient time and experience to accomplish.

The idea of reproduction project is inspired by the gradute course in computer networks at Stanford from which we borrow structure and ideas. Examples of reproduction projects from that course are on the Reproducing Network Research site.

Here are some sample questions your project might try to answer:

  • Does the primary result in the paper hold up? Can you generate a plot or graph that looks the same as one of the main figures in the original paper, and explain where it came from?

  • What kind of “reproduction” is appropriate? Most plots can be reproduced if you simply re-plot the authors’ original data on the same axes! Can the data be generated afresh? Are the assumptions behind the data-generating process, and the way the system was compared with prior work, realistic?

  • What was difficult to reproduce? Were you able to package the contribution of the paper in a way that makes it easier for future “reproducteurs” to reproduce the major results?

  • What happens if you vary a parameter the original experimenter didn’t consider?

  • Having reproduced the primary result, can you now extend or improve the work?

  • Can you find a simplification to the system that still preserves most of the claimed improvements? If so, this can indicate that the original authors are mistaken about why their system outperforms prior work.

See the “Searching for Interesting Papers” section below for ideas on where to look for recent or interesting papers.

Milestones and deliverables

You must meet the following milestones (unless otherwise specified in future announcements) to ensure a high-quality project at the end of the semester:

Turn in a 1-page draft proposal by June 11, 11:59PM.

Before you submit your proposal, we encourage you to talk to us (e.g. in office hours, or in lecture).

Please propose the topic for your project. If you’re not sure of what you want to do by the proposal deadline, you can submit a few possible options and we can help you decide.

Please be wise and mindful of your (other) deadlines and other commitments when picking the project.

  • Write a 1-page proposal describing:

    • The particular results you would like to replicate.

    • A brief outline of incremental steps you will need to do to finish the project as well as a timeline. The goal here is to convince both us and yourself that your project is neither too small nor too big.

Submit an intermediate project report by July 2, 11:59PM.

Please include the following in the report:

  • A draft introduction for the final report.

  • A baseline reproduction result. This doesn’t need to be correct, but we would like to see progress

  • The status of the project. What’s working, what are you having trouble with?

  • If you’ve reached out to the original paper authors, any responses you’ve gotten.

  • Plan for the remaining time

Present the final results during a presentation on July 22.

Turn in the final report and your code via email on or before 11:59PM on July 23.

  • The report must be submitted as a PDF file, with formatting similar to that of the papers you’ve read in the class.

  • The self-contained (i.e., include ALL dependencies) code must be submitted as a zip file. Each zip file containing the code must include a README file with a step-by-step guide on how to compile and run the provided code.

Final report

Your final deliverable will be a 4-6 page report about your project. You will be expected to:

  • Introduce the project, placing it in context, giving an overview, and summarizing your major contributions or findings.

  • Discuss prior work and the landscape of the area.

  • Explain your methods and/or the design of your system.

  • Present the experimental setup in detail.

  • Describe the results in figures and text.

  • Discuss the meaning of the results and what the reader should take away.

  • Describe the limitations and weaknesses of your system, measurement, or effort at replication.

  • Propose a plan for future work in the area. What would you do next if you had more time?

For reproduction projects, you will post your final writeup on the class site.

Be wise and mindful of your (other) deadlines and other commitments before you decide your deliverable.

Report guidelines

Formatting:

Please use the USENIX LaTeX template. If you would like to use something else besides LaTeX, please make sure your paper adheres to the USENIX style guidelines

If there is anything you need from the course staff, please let us know.

Writing an Introduction:

The introduction is the first, and unfortunately sometimes only, part of your research that someone will read. It’s important for it to get the main points of your result across in a clear and precise way.

Please make sure your introduction does the following:

  • States what problem the original paper was solving, and why the original paper was important. This should be self-contained: a reader shouldn’t have to read the original paper to understand your introduction.

  • Includes the result of the original paper

  • Briefly describe your reproduction methodology (in 1-3 sentences)

  • States the high-level results of your reproduction (when you have them)

Weekly check-ins and meetings

During the course of the project, you’ll have one mandatory meeting with the us around the time the milestone is due. The goal is to make sure you’re on track to complete the project.

In addition, we will expect you to send an update (via Piazza) each week to the intructor describing:

  1. what you did this week,

  2. what you need to do next week to stay on track, and

  3. if there’s anything the course staff can help with.

These updates will not be graded; the idea is to make sure you are making incremental progress. You may also visit the instructor by appointment.

Grading

You’ll be graded out of 20 points total.

The grades will be based on the following:

  • Intermediate progress report

  • Final report

  • Quality of experimental setup and results

  • Presentation

More than the success of the final outcome, we look for how thorough your efforts have been and we are more interested in the subtle things that you learn while reproducing a paper.

Searching for interesting papers

We recommend searching for interesting papers published in recent years in top systems and networking conferences, or by searching through the references section of another paper in the field. Relevant venues include:

  • ACM CCS, USENIX Security, IEEE S&P, NDSS: major conferences in security

  • OSDI, SOSP, NSDI, SIGCOMM: major conferences in computer systems and networking

  • EuroSys, CoNEXT, USENIX ATC, MLSys: other prestigious systems and networking venues

  • NIPS, ICML: major ML conferences

Other places would be the website of authors whose papers you’ve read in class and liked.