UBC CPEN 455 (2023 Winter Term 2): Deep Learning
Overview
Deep Learning has revolutionized many fields, e.g., computer vision, speech recognition, and natural language processing. It has become the vital pillar underpinning the modern machine learning and AI and is one of the most highly sought after skills in industries.
In this course, we embark on a comprehensive journey through both foundational and advanced topics in deep learning. You will gain hands-on experience with multilayer perceptrons, understand the nuances of batch normalization and back-propagation, and delve deep into popular neural network architectures, including convolutional, recurrent neural networks, transformers, and graph neural networks. The course places a significant emphasis on generative models, introducing a spectrum of approaches such as autoregressive models, Variational Autoencoders (VAEs), diffusion models, and Generative Adversarial Networks (GANs). This segment is pivotal for grasping how these models innovate in generating novel and diverse data sets. Moreover, the curriculum integrates deep reinforcement learning, skillfully blending deep learning methodologies with reinforcement learning tactics for practical, real-world scenarios. This course is meticulously crafted to offer a robust understanding of deep learning, equipping you with both theoretical insights and practical skills.
Course Information
| Instructor | Renjie Liao |
|---|---|
| TA | Qi Yan, Muchen Li, Jiahe Liu, Qihang Zhang |
| Time | 12:30pm to 2:00pm, Tue. and Thu. |
| Location | Hugh Dempster Pavilion 310 |
| Piazza | https://piazza.com/ubc.ca/winterterm22023/cpen455 |
| Canvas | CPEN 455 201 2023W2 |
| Tutorial | 1:00pm to 2:00pm Mon. Earth Sciences Building 1012 |
| Office Hour | 1:00pm to 2:00pm Wed. KAIS 3028 |
| rjliao@ece.ubc.ca |
Announcements
- Jan. 8 2024: We do not have the tutorial for the 1st week.
- Jan. 17 2024: We will have the lecture (Jan. 18) online due to the snow.
- Jan. 22 2024: We will the tutorial (Jan. 22) and the lecture (Jan. 23) online due to the Metro Vancouver transit strike.
Pre-requisites
- Basic knowledge in linear algebra, probability, and calculus.
- Proficiency in a programming language: preferably Python.
- Proficiency in a deep learning library: preferably PyTorch, JAX, Tensorflow, etc.
- Proficiency in LaTex.
Grading
- [20%] Homework (i.e., Written Assignments, 2X, each with 10%). Guideline & Policy
- [30%] Programming Assignments (3X, each with 10%). Guideline & Policy
- [10%] In-class Quiz (2X, each with 5%). Guideline & Policy
- [40%] Project (1X). Guideline & Policy
- [3% Extra Credits] Participation Guideline & Policy
Important Notes
-
All course-related questions should be sent and handled via Piazza. Canvas is only used for submitting homework, assignments, and projects. Try to avoid sending me emails directly as it is likely to be buried in my inbox.
-
All homework, assignments, and projects must be done individually. A 20% (non-hourly based) penalty is applied to any late submission unless special circumstances like illness. Any submission that is later than 3 days after the deadline will not be evaluated. E.g., if your homework is late but within 3 days after the deadline, your receive 80% of the grade for the homework. If it is beyond 3 days, then you get 0 grade.
-
You can use tools like ChatGPT in doing homework, assignments, and so on. If you use it, we request you to acknowledge it in your submitted materials and to submit your prompts (e.g., screenshots) for us to investigate the use cases. Ideally, you should restrict most of use cases to improving the english writing instead of getting answers directly.
-
UBC values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the Code of Student Conduct and Discipline.
Schedule
This is a tentative schedule, which will likely change as the course goes on. Changes will be announced on Piazza and this website.
| Dates | Topic | Slides | Suggested Readings | |
|---|---|---|---|---|
| Lecture 1 | Jan. 9 | Introduction | slides 1 | DL book: ch. 1 |
| Lecture 2 | Jan. 11 Jan. 16 |
Linear Models for Regression & Classification | slides 2 | DL book: ch. 5; PML1 book: ch. 10 & 11; PRML book: ch. 3 & 4 |
| Jan. 14 | Release Homework 1 (due Feb. 4) | |||
| Tutorial 1 | Jan. 15 | Tensor Operations Dataset & DataLoader | slides Colab |
Pytorch Official Tutorials |
| Lecture 3 | Jan. 18 Jan. 23 |
Multilayered Perceptron (MLP), Batch Normalization, Dropout | slides 3 | DL book: ch. 6 & 7; PML1 book: ch. 13; PML2 book: ch. 16; PRML book: ch. 5 |
| Tutorial 2 | Jan. 22 | Autograd Build your model | slides Colab |
|
| Lecture 4 | Jan. 25 Jan. 30 Feb. 1 Feb. 6 |
Back-Propagation, Optimization Methods w. Adaptive Learning Rate, Weight Initialization, Weight Decay, Early Stopping | slides 4.1 slides 4.2 |
DL book: ch. 7 & 8; PML1 book: ch. 13; PML2 book: ch. 6; PRML book: ch. 5 |
| Tutorial 3 | Jan. 29 | Training & Testing Your Model Hyper-parameter Tuning | slides Colab |
Google Tuning Playbook |
| Feb. 4 | Release Homework 2 (due Feb. 18) | |||
| Tutorial 4 | Feb. 5 | Recap on Matrix Calculus | slides | |
| Lecture 5 | Feb. 8 Feb. 13 Feb. 15 |
Invariance, Equivairance, Convolutions and Variants (Transposed, Dilated, Grouped, Seperable), Pooling, CNNs (UNet, ResNet, MobileNet) | slides 5.1 slides 5.2 |
DL book: ch. 9; PML1 book: ch. 14; PML2 book: ch. 16 |
| Tutorial 5 | Feb. 12 | Discussion on HW1 | N/A | |
| Lecture 6 | Feb. 27 Feb. 29 |
Recurrent Neural Networks | slides 6 | |
| Lecture 7 | Mar. 5 Mar. 7 Mar. 12 |
Transformers | slides 7 | |
| Feb. 18 | Release Programming Assignment 1 (due Mar. 3) | |||
| Tutorial 6 | Feb. 26 | Recap on Probability and Statistics | slides | |
| Lecture 8 | Mar. 14 Mar. 19 |
Graph Neural Networks | slides 8 | PML1 book: ch. 23; PML2 book: ch. 16 |
| Mar. 3 | Release Programming Assignment 2 (due Mar. 24) | |||
| Tutorial 7 | Mar. 4 | Discussion on HW2 | slides | |
| Lecture 9 | Mar. 26 Mar. 28 |
Autoregressive Models: PixelCNNs and Large Language Models (LLMs) | slides 9.1 slides 9.2 |
PML1 book: ch. 15; PML2 book: ch. 22 |
| Tutorial 8 | Mar. 11 | Discussion on Programming Assignment 1 | slides | |
| Lecture 10 | Mar. 21 Mar. 26 |
Autoencoders, Denoising Autoencoders, Variational Autoencoders (VAEs) | slides 10 | DL book: ch. 14; PML1: ch. 20; PML2 book: ch. 21 |
| Mar. 17 | Release Programming Assignment 3 (due Apr. 7) | |||
| Tutorial 9 | Mar. 18 | Hugging Face Transformers | slides | |
| Tutorial 10 | Mar. 25 | TBD | slides | |
| Tutorial 11 | Apr. 1 | TBD | slides | |
| Lecture 11 | Apr. 2 | Diffusion Models: DDPM & Score based Models | slides 11.1 slides 11.2 |
PML2 book: ch. 25 |
| Lecture 12 | Apr. 4 | Generative Advesarial Networks (GANs) | slides 12 | DL book: ch. 20; PML2 book: ch. 26 |
| Tutorial 12 | Apr. 8 | TBD | slides | |
| Apr. 21 (due) | Course Project |
FAQ
Can I audit or sit in?
I am very open to auditing guests if you are a member of the UBC community (registered student, staff, and/or faculty). I would appreciate that you first email me. If the in-person class is too full and running out of space, I would ask that you please allow registered students to attend.
Is there a textbook for this course?
While there is no required textbook, I recommend the following closely relevant ones for further reading:
- DL book: “Deep Learning” by Ian Goodfellow, Yoshua Bengio, Aaron Courville. Free online version
- PML1 book: “Probabilistic Machine Learning: An Introduction” by Kevin Murphy. Free online version
- PML2 book: “Probabilistic Machine Learning: Advanced Topics” by Kevin Murphy. Free online version
- PRML book: “Pattern Recognition and Machine Learning” by Christopher Bishop. Free online version
I also recommend students who are self-motivated to take a look at similar courses taught at other universities:
- UofT CSC413/2516, Winter 2022: Neural Networks and Deep Learning
- University of Amsterdam, 2022: Deep Learning
- EPFL EE559, Spring 2022: Deep Learning
- Stanford CS231n, Spring 2022: Deep Learning for Computer Vision