CSE 509 Computer System Security

Spring 2018

Course Description/Topics Lecture Notes Grading Class Hours
Important Dates Instructor and TA Texts Special needs

Course Description

In the class, we will discuss the principles and practice of computer system security. We will cover software vulnerabilities and defenses, with a focus on

One of the main objectives of this course is adversarial thinking: students should be able to quickly zoom in on the weakest link in any security technology, or system design. Students should be able to imagine how an attacker might break their system, and build in protection and mitigation measures to thwart such attacks.

This is a hands-on course, where you will learn by carrying out programming assignments and a final course project. Through the programming assignments, you will acquire an in-depth understanding of software vulnerabilities by developing exploits and mitigation techniques. The project will give you a taste of security research. Some of the programming assignments will be done in groups, others will be done individually. The final project will be a group project. We will try to schedule programming assignments within the first two months of the semester, so that you will have nearly half the semester left for working on your final project.

Course Topics

The topics covered in the course can be divided into two parts: foundations, and applications. Although the two parts are presented in sequence below, in reality, the lectures will alternate between the two.


Contemporary Threats, Vulnerabilities and Defenses

Lecture Notes

A single PDF file containing all the slides covered in class is available here

Text notes (last column in the table) included were put together by students in a past offering of the course. They are being provided for your benefit, but please keep in mind that they may not be complete, and in some cases, may not have been updated to match the slides.

Description/Reading Slides Notes
1 Introduction PDF  
2a Background: Runtime memory organization
Layout of code, static area, stack and heap
Struct/Object layout in C/C++ (Review only the last part of notes.)
2b Stack-smashing, Heap overflows and Format string attacks
Reading: Smashing the stack for fun and profit
2c Systematic defenses for Memory corruption exploits
Optional Reading: Memory exploitation defenses in Windows
Optional Reading: (Not so) Recent advances in exploiting buffer overruns
Optional Reading: Basic Integer Overflows
See Prev. Topic
2d Memory-error detection: Bounds-checking, etc. See Prev. Topic
3a Web security PDF  
3b Injection Attacks, Taint-tracking
Race conditions
Principles of Secure System Design
4a Malware
Evasion, obfuscation, Software tamper-resistance
A very short article from 2011 on specific malware trends.
4b Securing Untrusted Code: System-call interception,
Inline-reference monitoring
4c Securing Untrusted Code: Inline-reference monitoring,
Software-based fault isolation, Control-flow integrity
5 Reverse engineering: Disassembly, Binary analysis,
Binary rewriting, Dynamic binary translation
6 Vulnerability analysis: Fuzzing and Symbolic Execution PDF
7 Virtual Machines PDF
8 Cryptography Basics
Reading: Who is guarding the guardians: The Comodohacker's postings
9 Identification and Authentication
Reading: Password Security: A Case History
Reading: Lamport's One-Time Password Scheme
Optional Reading: How anonymous hacked into a security firm
10 OS Security and Security policies
Reading: Revisiting "Setuid Demystified"
Reading: Confining Root Programs with Domain and Type Enforcement
11 Side-channel attacks
Reading: Metdown and Spectre attacks

Class Place and Time:

Lectures: Tue, Thu 10:00am to 11:20am     Room Old CS Rm 2114
Final Exam: Fri, May 11 11:15am to 1:45pm     Room Old CS Rm 2114


R . Sekar
Office: Rm 364 New Computer Science
Office Hours: Wed 11:00am to noon, Tue/Thu 11:20am to 11:50am


There is no official textbook for this course. We will rely on class notes and some papers. Some of the lectures will draw on material from the following books.


You will be handed homework problems sets in order to help you prepare for the exams. You will not have to submit solutions to these problem sets, but in order to encourage you to actually work out the problems, we will hold short quizzes in the class that test you on problems very similar to those in the problem sets. In order to reduce the stress involved in these quizzes, we automatically scale up quiz scores by a factor of 4/3, up to a maximum of 100%.

Your final grades will be computed as follows. You should expect some (small) changes to the weightings over the semester.

We will likely have a single midterm exam and then the final. Some programming assignments will be done by groups of two.

You will get full credit for quizzes as long as you score 75% or more. More generally, if you score m_1 through m_k in homeworks/quizzes, your score for these assignments will be given by (4/3)*average(min(75, m_1),...,min(75, m_k)). This has been done so as to reduce the amount of preparation time needed for homework quizzes. My intent is that quizzes wil require no preparation beyod solving problems in the associated homework problem set. (This policy of scaling up scores does not apply to programming assignments.)

Copying homework solutions or programming assignments from a fellow student or from the Internet, and all other forms of academic dishonesty, are considered serious offenses. They will be prosecuted to the maximum extent permitted by university policies.

Special Needs

If you have a physical, psychological, medical or learning disability that may impact on your ability to carry out assigned course work, I would urge that you contact the staff in the Disabled Student Services office (DSS), in the ECC building, 632-6748v/TDD. DSS will review your concerns and determine, with you, what accommodations are necessary and appropriate. All information and documentation of disability is confidential.