The Homunculus in Deep Learning

Abstract

Gated recurrent neural networks are commonly explained by their ability to create additive copy paths through time, which can preserve information and gradients over long sequences. This explanation is correct, but incomplete: useful gate values must themselves be learned, and this gate-learning process is also performed through recurrent computation. We study this missing learning step with controlled sequence classification tasks. We show that gated architectures do not solve long-range dependencies by architecture alone: when all training samples require long-range memory from the start, gated and non-gated recurrent models both fail. However, when training also contains short-dependency samples in which the same label relation can be learned over shorter temporal gaps, gated models can first learn a selective update behavior and then apply it to long-range samples. Diagnostic probes show that during successful training, larger gradients reach early recurrent states, and state updates depend more clearly on the input. A multi-class extension further shows that the learned behavior transfers partially beyond the subset of informative inputs that receives short-dependency samples. Overall, our results suggest that gates help not because they automatically solve long-range dependencies, but because they provide a mechanism that can be learned once the data makes the gate-learning problem simple enough. This reframes gated recurrence from an automatic solution to a learnable scaffold, and suggests that training data should be designed to expose gate-learning signals before relying on gates for long-range memory.