SCRN separates a slowly evolving context state from a faster hidden state. This simplified unrolling shows the two interacting memories; the paper investigates how dropout and parameter sharing affect the model’s performance. Research source
About this paper
Reproduces the Structurally Constrained Recurrent Network and studies regularization for neural language modeling.
The experiments evaluate dropout, variational dropout, and weight tying. With suitable regularization and optimization, SCRN performs comparably to LSTM on English data and better on the tested non-English data, with results depending on the regularization setting.
Citation
BibTeX
@inproceedings{takhanov2018regularizingscrn,
title = {Reproducing and Regularizing the SCRN Model},
author = {Kabdolov, Olzhas and Assylbekov, Zhenisbek and Takhanov, Rustem},
year = {2018},
booktitle = {COLING 2018},
url = {https://aclanthology.org/C18-1145.pdf}
}