In this post, I will explain noise contrastive estimation.

In this post, I will explain score matching and its connection with denoising autoencoder. Suppose we have observed samples from an unknown data distribution $p_{0}(\mathbf{x})$ and want to learn an energy based model (EBM) as below,

Let us take a look at the softmax function which is frequently used in deep learning.

In this post, I will review several popular gradient estimators for discrete random variables. In machine learning, especially latent variable models and reinforcement learning (RL), we are often facing the following situation. We have a discrete random variable \(Z\) which takes values from \(K\) categories where \(K\) could be finite or countably infinite. Assuming the distribution associated with \(Z\) is \(q_{\phi}(Z)\), we would like to optimize the expected function as follows,

In this post, we will review a great paper which bounds the generalization error of a broad class of models, including a popular class of deep neural networks, using the PAC-Bayesian framework.

There are many scenarios in machine learning and statistics where we hope to compute the ratio of two density functions, e.g., KL-divergence, mutual information, importance sampling, and hypothesis testing. Specifically, given two density \(p(X)\) and \(q(X)\) on the same sample space, we’d like to compute