Learnable weight initialization in neural networks

A new method of initializing the weights in deep neural networks is proposed. The method follows two steps. First, consider each layer as a model and perform a linear regression to keep the mean of the layer output to zero and variance<br/>after the data is passed through the activation function to one. Once each layer converges to the target...