CS480/680: INTRODUCTION TO MACHINE LEARNING, Spring 2025, University of Waterloo

Overview

This course focuses on the introduction of machine leanring. However, we will cover what you are particularly interested in, e.g., technical details of how to train your own ChatGPT.

Pre-requisite:

Format:

Graded Student Work for CS480/680: Homeworks (We do not accept hand-written submission. Typeset using LaTeX is recommended. HW solutions will not be released. Please attend TA's office hour for a solution.): Homework Policy: Completed assignments will be submitted through LEARN. Submit early and often! You must write your solutions independently and individually, and you should always acknowledge any help you get (book, friend, internet, etc.). Using AI to write homeworks is prohibited. We may use tools to detect your submission. Mark appeals should be requested within two weeks of receiving the mark. The appeal could go either ways, so request only if you truly believe something is wrong.

Late Policy: We do NOT accept any late submissions, unless you have a legitimate reason with a formal proof (e.g., hospitalization, family urgency, etc.). The proof date should be within 7 days before your homework deadline. Traveling, being busy with other stuff, internet disconnection, or simply forgetting to submit, are not considered legitimate. Without a proof, your score will be 0 as long as you are late, even for 1min (LEARN submission portal will be closed on time. We DO NOT accept homework submission by email.). With a proof and instructor's approval, you can get a 7-day homework extension. According to the school policy, undergraduate students are allowed to use short-term absence once per term. Please inform the TA head Songcheng Cai (s56cai@uwaterloo.ca) and provide a screenshot if you have submitted an application to Quest for a 2-day extension. Failing to do so (e.g., only informing instructors or other TAs) will make your application invalid, and your delayed homework will still be marked as late.

Textbook:

There is no required textbook, but the following fine texts are recommended.

Schedule (tentative)

Date Category Topic Slides Suggested Readings
Lecture 1 May 5 Introduction
Link Deep Learning, Section 1
Lecture 2 May 7 Classic ML Perceptron
Link Patterns, Predictions, and Actions, Page 37
May 12 Classic ML Perceptron - Cont'
Link Patterns, Predictions, and Actions, Page 37
Lecture 3 May 14 Classic ML Linear Regression
Link Probabilistic Machine Learning: An Introduction, Page 363
Lecture 4 May 21 Classic ML
  • Linear Regression - Cont'
  • Logistic Regression
  • Link
  • Link
    		• Probabilistic Machine Learning: An Introduction, Page 333
    Lecture 5 May 26 Classic ML Hard-Margin SVM
    		Link The Elements of Statistical Learning, Section 12.3
    Lecture 6 May 28 Classic ML Soft-Margin SVM
    		Link The Elements of Statistical Learning, Section 12.3
    Lecture 7 June 2 Classic ML
  • Soft-Margin SVM - Cont'
  • Reproducing Kernels
  • Link
  • Link
    		• The Elements of Statistical Learning, Section 12.3
    Lecture 8 June 4 Classic ML Gradient Descent
    		Link Convex Optimization, Section 9.3
    Lecture 9 June 9 Neural Nets
  • Gradient Descent - Cont'
  • Fully Connected NNs
  • Link
  • Link
    		• Deep Learning, Section 6
    June 11 Neural Nets Fully Connected NNs - Cont'
    		Link Deep Learning, Section 6
    Lecture 10 June 16 Neural Nets Convolutional NNs
    		Link Deep Learning, Section 9
    June 18 Neural Nets Convolutional NNs - Cont'
    		Link Deep Learning, Section 9
    No class June 23 - Mid-term Exam
    		- Time: 11:30AM - 12:50PM, Location: HLTH Expansion Building (EXP), Room 1689
    Lecture 11 June 25 Neural Nets Transformer
    		Link
  • “Attention Is All You Need”. Vaswani et al. 2017 link
    		•
    Lecture 12 July 2 Modern ML Paradigms Large Language Models
    		Link
  • “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. Devlin et al. 2018 link
  • (GPT-1) “Improving Language Understanding by Generative Pre-training”. Radford et al. 2018 link
  • (talk by Andrej Karpathy) State of GPT link
    • July 7 Modern ML Paradigms Large Language Models - Cont'
    		Link
  • (GPT-2) “Language Models are Unsupervised Multitask Learners”. Radford et al. 2019 link
  • (GPT-3) “Language Models are Few-Shot Learners”. Brown et al. 2020 link
  • (GPT-3.5) “Training Language Models to follow Instructions with Human Feedbacks”. Ouyang et al. 2022 link
  • (GPT-4) “GPT-4 Technical Report”. OpenAI 2023 link
    • Lecture 13 July 9 Modern ML Paradigms Speculative Decoding
    		Link
  • Fast Inference from Transformers via Speculative Decoding
  • Accelerating Large Language Model Decoding with Speculative Sampling
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
    • July 14 Modern ML Paradigms Speculative Decoding - Cont'
    		Link
  • EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
  • EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
  • EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
    • Lecture 14 July 16 Modern ML Paradigms GANs Link
  • Generative Adversarial Networks
    • Lecture 15 July 21 Modern ML Paradigms Self-Supervised Learning
    		Link
  • A Simple Framework for Contrastive Learning of Visual Representations
  • Learning Transferable Visual Models From Natural Language Supervision
    • Lecture 16 July 23 Trustworthy ML Robustness
    		Link
  • (White-box) DeepFool: a simple and accurate method to fool deep neural networks
  • Towards Deep Learning Models Resistant to Adversarial Attacks
  • Theoretically Principled Trade-off between Robustness and Accuracy
    • Lecture 17 July 28 Trustworthy ML Certified Defenses
    		Link
  • Exploring the limits of model-targeted indiscriminate data poisoning attacks
    • Lecture 18 July 30 Trustworthy ML
  • Other Threats to ML
  • Mid-term Review
    		• Link
  • (Physical) “Robust Physical-World Attacks on Deep Learning Models”. Eykholt et al. CVPR 2018 link
  • (Physical) “Adversarial examples in the physical world”. Kurakin et al. ICLR 2017 link
  • (Physical) “Synthesizing Robust Adversarial Examples”. Athalye et al. ICML 2018 link
  • (Physical) “Fooling automated surveillance cameras: adversarial patches to attack person detection”. Thys et al. CVPR 2019 Workshop link
  • (Poisoning) “Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks”. Shafahi et al. NeurIPS 2018 link
  • (Poisoning) “Trojaning Attack on Neural Networks”. Liu et al. NDSS 2018 link
  • (Poisoning) “Hidden Trigger Backdoor Attacks”. Saha et al. AAAI 2020 link
  • (Poisoning) “Deep Partition Aggregation: Provable Defense against General Poisoning Attacks”. Levine et al. ICLR 2021 link

    • Mental Health: If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support.

    On-campus Resources

    Off-campus Resources

    Diversity: It is our intent that students from all diverse backgrounds and perspectives be well served by this course, and that students’ learning needs be addressed both in and out of class. We recognize the immense value of the diversity in identities, perspectives, and contributions that students bring, and the benefit it has on our educational environment. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally or for other students or student groups. In particular: