About this paper

Analyzes why gradient-based methods struggle with high-frequency periodic functions and modular multiplication.

For these target classes, the gradient can carry very little information about the function being learned. The analysis relates this difficulty to gradient variance and the approximate orthogonality of candidate target functions.