"the structure of the network itself would be represented in the silicon" -- ASICs then? perhaps even hybrid analog/digital one, where fixed coefficients are stored in digital memory, while input data is analog.

I believe there is a great value in being able to "snapshot" the state and later load exactly the same state into millions of devices. And I cannot see how this will easily work with memristors.