Three illustrative periodic targets and multiplication by five modulo thirteen. The paper connects these kinds of oscillatory structure to target classes whose learning gradients can carry very little information under the analyzed assumptions. Research source
About this paper
Analyzes why gradient-based methods struggle with high-frequency periodic functions and modular multiplication.
For these target classes, the gradient can carry very little information about the function being learned. The analysis relates this difficulty to gradient variance and the approximate orthogonality of candidate target functions.
Citation
BibTeX
@article{takhanov2025gradientdescenthighfrequency,
title = {Gradient Descent Fails to Learn High-frequency Functions and Modular Arithmetic},
author = {Takhanov, Rustem and Tezekbayev, Maxat and Pak, Artur and Bolatov, Arman and Assylbekov, Zhenisbek},
year = {2025},
journal = {Machine Learning},
eprint = {2310.12660},
archivePrefix = {arXiv},
url = {https://arxiv.org/pdf/2310.12660}
}