During training the weights are constantly changing and their final values uncertain. If you prune a connection to early you could destroy a synapse that would become important later on.
That's no different from a real-life network. Biology tends to solve this by growing connections as well as pruning them.
Another comment mentioned Song Han's work into NN compression. Well, I looked up a recent paper and look what it says:
> We discovered an interesting byproduct of model compression: re-densifying and retraining from a sparse model can improve the accuracy. That is, compared to a dense CNN baseline, dense → sparse → dense (DSD) training yielded higher accuracy
> We now explain our DSD training strategy. On top of the sparse SqueezeNet (pruned 3x), we let the killed weights recover, initializing them from zero. We let the survived weights keeping their value. We retrained the whole network using learning rate of 1e−4. After 20 epochs of training, we observed that the top-1 ImageNet accuracy improved by 4.3 percentage-points
> Sparsity is a powerful form of regularization. Our intuition is that, once the network arrives at a local minimum given the sparsity constraint, relaxing the constraint gives the network more freedom to escape the saddle point and arrive at a higher-accuracy local minimum. So far, we trained in just three stages of density (dense → sparse → dense), but regularizing models by intermittently pruning parameters throughout training would be an interesting area of future work
I wonder if you could also receive similar results by simply turning some connections "off" for a while, only training the rest of the connections. Then turn the missing connections on a gain and, randomly turning a few other connections off, and continue training.
Most deep NNs are already trained with a sparsity penalty called weight regularization or weight decay. That pushes most of the weights to be really close to zero unless larger values are necessary. The benefit of this is it's continuous and differentiable. So it can be trained with backpropagation. Binary on/off connections are much more complicated to optimize.
> That pushes most of the weights to be really close to zero unless larger values are necessary.
So why not have a certain epsilon, below which you can turn the connection off altogether? (meaning the back-propagation would only apply to the remaining connections) To avoid getting stuck in local minima you could occasionally re-initialise them with a random small value.
Again, zero background in machine learning here. It's a sincerely naive question to which I fully expect a "we've tried that with, methods X, Y and Z are most famous and this is how they work out in practice".
What do you gain by doing that? It isn't any cheaper to train with connections removed. It could really damage the training if a parameter gets stuck at 0 that shouldn't be. And the sparsity penalty has traditionally been considered to be enough.
> I wonder if you could also receive similar results by simply turning some connections "off" for a while, only training the rest of the connections. Then turn the missing connections on a gain and, randomly turning a few other connections off, and continue training.
Isn't that almost like Dropout? Dropout does what you say but it changes the dropped set of neurons for each individual example.
So this is basically dividing the network into two groups, and give each group a second weight: one for those that are on, zero for those that are off.
Has anyone tried more variants of this? Using random bundling + various levels of amplification of such bundles, so that backpropagation affects various networks differently?
