The O'Reilly book Understanding The Linux Kernel is fantastic. Starts you off in the deep end with memory management/addressing and fans out from there. Notably excludes networking, as the book is long enough without it.
I think it comes down to just understanding IP and TCP in general. Which I don't really (IP) or do sorta (TCP).
As far as kernel architecture, the rest of the system can work with the assumption "writes go into a buffer, reads come from a buffer, and the networking subsystem does the right thing in between", plus some stuff about DMA optimizations.
