Well there are some fundamentals such as universal approximation.

And we may find that biology also exploits structures like deep nets