I find your comment a little strange. First you say something interesting about Deep Dream being an early attempt at better understanding of vision networks but that it's not enough. Then you say the end is in sight?
If you just take the visualization part we have a massive amount of work to do. Understanding how these networks solve problem in sophisticated ways will motivate new advances to mitigate shortcomings. It is a software problem that can have huge benefits. For example, see the paper Visualizing Loss Functions For Neural Networks as an example of the incredible insight to be gained with better visualization.
The intuition of loss functions hasn't changed much in the last several decades. If you have a local optimizer it needs to be able to find its way down further than it already is. Anything you can do to remove junk configurations from even being considered during optimization will help. But knowing when that is happening has been really hard because it's just so computationally intensive to generate plots like in that paper.
When they say the end is in sight, they aren't referring to the problems with neural networks. In fact, they mean the limits of what neural networks can do is in sight.
If you just take the visualization part we have a massive amount of work to do. Understanding how these networks solve problem in sophisticated ways will motivate new advances to mitigate shortcomings. It is a software problem that can have huge benefits. For example, see the paper Visualizing Loss Functions For Neural Networks as an example of the incredible insight to be gained with better visualization.
The intuition of loss functions hasn't changed much in the last several decades. If you have a local optimizer it needs to be able to find its way down further than it already is. Anything you can do to remove junk configurations from even being considered during optimization will help. But knowing when that is happening has been really hard because it's just so computationally intensive to generate plots like in that paper.
https://arxiv.org/abs/1712.09913