Denoising Deep Generative Models

Likelihood-based deep generative models exhibit pathological behaviour as a consequence of using high-dimensional densities to model data with low-dimensional structure. In this paper we propose two methodologies to remove the dimensionality mismatch during training. Our first approach is based on Tweedie’s formula, and the second on models which take the variance of added noise as a conditional input. We show that surprisingly, while well motivated, these approaches only sporadically improve performance over not adding noise, and that other methods of addressing the dimensionality mismatch are more empirically adequate.