Because of latency difference between RAM and on chip caches you don't gain too much by making wider data bus to RAM, because latency is still the most limiting factor. Modern solution is to have multiple full memory interfaces ("channels") which give you the ability to both behave as if the memory bus is wider as well as issue completely different memory transactions given that the accesses are suitably aligned (which is more significant performance boost).
Also the general trend of weird (i.e. PCI) and serial interfaces in 90's was motivated by per-unit costs at the expense of required engineering. Primary motivator for PCI's reflected wave switching and multiplexed address and data was to limit number of pins and number of required passive components on motherboard. Serial interfaces that came after that (SATA, PCI-E) were motivated by the fact that routing fast parallel synchronous buses for any significant distances is hard problem because of propagation times, which have to be roughly equal for all bus wires, which implies that PCB material parameters have to be known and reasonably consistent (and that means significant per-unit additional expense in PCB manufacture and testing cost).
Currently only interface that requires careful routing and design on PC motherboard is memory interface, which can be made short enough that manufacturing differences are negligible. (It's funny how TI has 20 page application note on correct routing of USB2 which includes rules as "no vias preferably, at most one", "no stubs", "controlled impedance" and "as short as possible" while Intel's layout recommendations for USB2 can be summarized as "it's differential pair, discontinuities do not matter much, use sensible routing")
Also, routing wide fast interfaces is hard, because of clock skew between the parallel lanes. Having independent parallel channels mitigates that somewhat, but if anything the trend is towards narrow serial interconnects.
That is more of an answer to routing and signal integrity problems of fast parallel interfaces, e.g. bandwidth. Most of the DRAM latency is inherent in the DRAM array itself (actually getting the data between dram array and sense amplifiers). Almost nobody cares about additional cycle of latency introduced by registered dimms, as it is essentially noise compared to precharge latency of the dram itself. This comes mostly from physical limits of what can be manufactured with reasonable power dissipation and reliability (see how "CAS Latency" grows at comparable rate to clock rate, ie. stays mostly comparable in wall-clock time)
Take a look at Micron's "HMC" stacked memory architecture. THIS is a move in the right direction. The short-term caveat is its focus on higher-performance niches like servers. It's a long time before you'll see this type of architecture in consumer PCs due to DDR3's dirt-cheap pricing.
> Modern solution is to have multiple full memory interfaces ("channels") which give you the ability to both behave as if the memory bus is wider as well as issue completely different memory transactions given that the accesses are suitably aligned (which is more significant performance boost).
In other words, RAID-0, for memory. :-)
I wonder if it would be interesting to have memory controllers managing striping/mirroring of memory modules and create read-optimized and write-optimized memory regions. 2 mirrored modules would give you half the latency for reads and the same latency as a single one for writes.
Has anyone already done this?
edit: and now I'm imagining an inter-memory-module bus to manage bank transitions to/from mirrored/striped without loading the processor bus.
>I wonder if it would be interesting to have memory controllers managing striping/mirroring of memory modules
This is already common. Log onto Dell.com and configure a high end server, you will see an extensive number of options regarding mirroring and advance ECC configurations.
its like two individual drives, one for /etc the other for /usr
and no, you cant have half the latency, latency is dictated by physical speed of actual ram inside chips, those are clocked at 200MHz for typical DDR3 1600Hz module, up to 300MHz for fasters DDR3 ones.
Actually it's more like raid0 than separate partitions, because anything upstream of memory controller does not have to care about this in any way. The amount of complexity between modern CPU and memory is quite fascinating (and huge amount of essentially invisible and relatively complex stuff is there at least since i486, i.e. with anything that does not care, in the works/does not work sense, about how you combine memory modules).
Also the general trend of weird (i.e. PCI) and serial interfaces in 90's was motivated by per-unit costs at the expense of required engineering. Primary motivator for PCI's reflected wave switching and multiplexed address and data was to limit number of pins and number of required passive components on motherboard. Serial interfaces that came after that (SATA, PCI-E) were motivated by the fact that routing fast parallel synchronous buses for any significant distances is hard problem because of propagation times, which have to be roughly equal for all bus wires, which implies that PCB material parameters have to be known and reasonably consistent (and that means significant per-unit additional expense in PCB manufacture and testing cost).
Currently only interface that requires careful routing and design on PC motherboard is memory interface, which can be made short enough that manufacturing differences are negligible. (It's funny how TI has 20 page application note on correct routing of USB2 which includes rules as "no vias preferably, at most one", "no stubs", "controlled impedance" and "as short as possible" while Intel's layout recommendations for USB2 can be summarized as "it's differential pair, discontinuities do not matter much, use sensible routing")
[edit: formatting + missing word]