EECE 571F (2024 Winter Term 1): Advanced Topics in Deep Learning

Archived offering. Dates and assessment requirements below belong to this offering. View the current EECE 576L course.

Overview

Advanced deep learning techniques have revolutionized the field, enabling remarkable progress across various applications. This course will provide a comprehensive understanding of the latest models and methods that are shaping the future of deep learning, with a particular focus on probabilistic and geometric deep learning, and deep reinforcement learning.

We will cover following advanced topics:

Geometric Deep Learning: Graph Neural Networks (GNNs), Transformers, Group Equivariant Networks.

Probabilistic Deep Learning: Large Language Models (LLMs), VAEs, EBMs, Flow Models, Diffusion Models.

Course Information

Students should ask all course-related questions on Piazza. We will use Canvas to handle submission and evaluation of all reports and project related files.

Pre-requisites

• Basic knowledge about machine learning, linear algebra, probability, and calculus.
• Proficiency in a programming language: preferably Python.
• Proficiency in a deep learning libraries: PyTorch, JAX, Tensorflow, etc.

Announcements

• Sep. 3 2024: The first course will be held on Sep. 4.

Grading

Grades will be based on:

• [15%] One paper reading report. Guideline & Policy, due Sep. 27
• [15%] Project proposal. Guideline & Policy, due Oct. 11
• [15%] Project presentations. Guideline & Policy
• [15%] Peer-review report of project presentations. Guideline & Policy, due Dec. 6
• [40%] Project report and code. Guideline & Policy, due Dec. 11

Course Project

Students can work on projects individually, or in groups of up to four (group should be formed as early as possible). Students are strongly encouraged to form groups via, e.g., discussing on Piazza. However, a larger group would be expected to do more than a smaller one or individuals. All students in a group will receive the same grade. Students are allowed to undertake a research project that is related to their thesis or other external projects, but the work done for this course must represent substantial additional work and cannot be submitted for grades in another course.

The grade will depend on the quality of research ideas, how well you present them in the report, how clearly you position your work relative to prior literature, how illuminating and or convincing your experiments are, and well-supported your conclusions are. Full marks will require a novel contribution.

Each group of students will write a short (>=2 pages) research project proposal, which ideally will be structured similarly to a standard paper. You don’t have to do exactly what your proposal claims - the point of the proposal is mainly to have a plan for yourself and to allow me to give you feedback. Students will do a short presentation (roughly 5 minutes for individual, 10 to 15 minutes for a larger group) for their projects towards the end of the course. At the end of the class, every group needs to submit a project report (6~8 pages).

Important Notes

1. All reports (i.e., paper reading report, proposal, peer-review report, and final project report) must be written in NeurIPS conference format and must be submitted as PDF

2. Late work will be automatically subject to a 20% penalty and can be submitted up to 3 days after the deadline

3. UBC values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the Code of Student Conduct and Discipline.

4. It is the responsibility of each student to understand the policy for each course work, ask questions to the instructor if it is not clear, and carefully acknowledge all sources (papers, code, books, websites, individual communications) using appropriate referencing style when submitting work.

Schedule

This is a tentative schedule, which will likely change as the course goes on.

FAQ

Can I audit or sit in?

I am very open to auditing guests if you are a member of the UBC community (registered student, staff, and/or faculty). I would appreciate that you first email me. If the in-person class is too full and running out of space, I would ask that you please allow registered students to attend.

Is there a textbook for this course?

While there is no required textbook, I recommend the following closely relevant ones for further reading:

• DL book: “Deep Learning” by Ian Goodfellow, Yoshua Bengio, Aaron Courville. Free online version
• PML1 book: “Probabilistic Machine Learning: An Introduction” by Kevin Murphy. Free online version
• PML2 book: “Probabilistic Machine Learning: Advanced Topics” by Kevin Murphy. Free online version
• PRML book: “Pattern Recognition and Machine Learning” by Christopher Bishop. Free online version
• GNN book: “Graph Neural Networks: Foundations, Frontiers, and Applications” by Lingfei Wu, et al. Free online version
• GRL book: “Graph representation learning” by William Hamilton. Free online version

In particular, for diffusion models, I recommend the following textbook on the relevant background mathematical materials:

• SDE book: “Brownian Motion and Stochastic Calculus” by Karatzas, I. and Shreve, S., 2012.

I also recommend students who are self-motivated to take a look at similar courses taught at other universities:

• African Master’s in Machine Intelligence (AMMI 2021): Geometric Deep Learning
• Stanford CS224W, Winter 2021: Machine Learning with Graphs
• McGill Comp 766, Winter 2020: Graph Representation Learning
• UofT STA 4273 / CSC 2547, Spring 2018: Learning Discrete Latent Structure

Paper List

• Must-read papers on GNN
• Awesome Graph Neural Networks
• Awesome Equivariant Networks
• Awesome LLMs
• Awesome Diffusion Models
• Awesome Energy based Models
• Generative Models

Good Project Reports from 2022 Winter Term 1

• Node-element Hypergraph Message Passing for Mesh-based Simulations
• Benchmarking Attention-based Quantum State Tomography
• Heram: Multi-Magnification Graph-Structured Whole Slide Image Representation
• Vision Transformers for Classification in Small-Sized Chest X-Ray Datasets
• DiffuseDRAW: Structured Latent Variables Model with Discrete Diffusion Prior
• Graph-Guided Unsupervised Clustering for Source-Free Domain Adaptation
• EchoGNN with Contrastive Learning
• Deep Learning-Based Calibration for Millimeter-Wave Phased-Array Antennas

Good Project Reports from 2021 Winter Term 2

• Physics Aware Joint Inference for the Cryo-EM Inverse Problem: Normal Modes, Global 3D Pose and CTF Defocus
• V-DMGNN-GAN: Spatial Inpainting for Human Motion Prediction
• Improving Out-Of-Distribution Generalization of Neural Algorithmic Reasoning Tasks
• 3D Ultrasound Segmentation using Transformers
• Graph-enhanced Transformers for Referring Expressions Comprehension
• HiGNN: Hierarchical Left Ventricle Landmark Detection with Graph Neural Networks
• Towards Domain Generalized Segmentation with Transformer
• A Transformer-based Video Analysis Framework for Estimating Ejection Fraction from Echocardiograms

Previous Version:

• 2023 Winter Term 1
• 2022 Winter Term 1
• 2021 Winter Term 2