This is an interesting idea that may be combined with the article suggested idea.
In the article, if I understood correctly, what they propose is train your network once, remove the x% of smaller absolute magnitude weights, retrain your network fixing those smaller magnitude weights to 0 starting from the same initial starting point.
The idea behind is that your optimization process the first time is telling you that the solution is near a subspace where the weights are 0 but it can't really converge to it. So you project to this subspace by enforcing the weights to 0. Then you retrain again and the search will be easier because the space is smaller, but because you are starting from the same starting point you are kind of guaranteed that you will be able to reach the same optimum but projected.
The problem of the sparsity in the article is that while some weights are 0, they are 0 through masking, therefore you are still doing the computations, and you don't really benefit from the sparsity. If you have enough 0, you can benefit from the sparsity by using some sparse representation, but those are typically an order of magnitude slower than the dense representation.
Combining the idea of the article with the idea from OpenAI of block-sparse neural networks which reduce the operations done without suffering too much from the non-locality and indirection of a sparse representation.
After training normally (provided you have enough memory) (eventually with a sparse-block regularization term to help induce block-sparsity) you may try to prune in such a way that the least significant sparse-blocks are pruned, therefore you may expect both the boost in speed, and the better accuracy and convergence properties.
This is a kind of two phase search, first we look for a finer structure, then we restart to find the best weights for this finer structure.
Results from above suggest that you can leave 12.5% of your weights to be non-zero and get nice 4x speed up (1/8 operations done twice as slow). Accuracy start to drop at about 12.5% of weights remaining in both OP and OpenAI papers.
Based on what you described, it feels like the MIT paper and the openai paper are essentially the same thing. The only difference is the masking/pruning part, which I think is just an engineering detail.
Sorry If I mis-conveyed the ideas. They are quite different.
The openai paper is introducing operations which are a fast middle ground between dense and sparse operations. You still have to specify the sparsity structure you like. (Although often some random sparsity structure work well).
The MIT paper describe one way to choose a sparsity structure and starting point which will work well in the general case.
The OpenAI approach is more amenable to an obvious HW implementation with the block sparsity because the blocks are are GEMM operations are implemented in the first place.
There are obviously more available sparse solutions if the block sparsity constraint is relaxed therefore I wouldn't be surprised if the best results come from such a network.
Yes in the OpenAI paper you don't need to learn the sparsity structure. But afaiu the mit paper help suggest an appropriate sparsity structure.
Concerning the sparsity speed-up, I've tried the tensorflow sparse representation a while back, and it was kind of a high effort, low reward process. You had to drop like 90% of the weights (was reducing accuracy), change your ops (dense, convolutions,...) and use big enough layer size to get a feel you were getting something speed-wise.
The openai block-sparse kernels seems promising I'll give them a try.
They train block-sparse neural networks, where sparseness learned during training.