Verifying the Union of Manifolds Hypothesis for Image Data

The manifold hypothesis states that data lies on an unknown manifold of low intrinsic dimension. We argue that this hypothesis does not properly capture the low-dimensional structure typically present in data, and we put forth the union of manifolds hypothesis, which accommodates the existence of non-constant intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, and show that classes with higher intrinsic dimensions are harder to classify.