Yep! The big thing people don't realize that it is memory operations that are the biggest cost in terms of both time and energy.
While it takes ~100 picojoules to do a double precision floating point operation on a Ivy Bridge Intel processor, it takes 4200 picojoules to move the 64 bits from DRAM to your registers. Most people assume that the huge power usage is because you need to move data from off the chip, but the reality (and surprising fact to most people) is that over 60% (~2500 picojoules) of the energy usage of moving the data is consumed by the on chip cache hierarchy. That doesn't mean the SRAM caches themselves, but all the additional logic that makes it hardware managed (TLBs, etc) that give you functionality like virtual memory translations, cache coherency, etc.
Getting rid of all of that cruft that has been added since the 80s to make programmers lives easier would actually reduce power consumption and latency significantly... My startup is working on that problem by removing all of that additional logic from the hardware and instead having it managed at compile time. The best thing though would be having programmers really think about locality when writing their programs though.
There are two (major) things that we are addressing with the compiler. The first is the traditional VLIW problem, which is the same problem Itanium faced, and the second is doing fully software managed memory, which no (non academic) architectures have attempted to solve with a compiler.
1) As for the traditional VLIW problem, the simplified explanation of why it is difficult for most systems is because it is difficult to know exactly when a functional unit will actually receive/have access to a piece of data (either due to data hazards, latency due to the memory system, or many other factors). We solve this at the hardware level by being the first architecture to be able to guarantee latency between any location in memory. Once you can guarantee this in hardware, your compiler has a lot more information to be able to make decisions with and does not need to needlessly insert nops that hurt performance.
2)When it comes to software memory management, we have some new proprietary techniques for determining memory usage at compile time plus runtime tools. For obvious reasons I can't go into too much detail on how they work, but we will be publishing on it in the near future.
To summarize, we think that the reason others have never made a "sufficiently smart compiler" is because the hardware never gave enough data to the compiler and vice versa. We decided to have virtual memory (which we think is unnecessary) and instead opted for having all of our cores have access to a shared memory space, which simplifies both the hardware and makes memory mapping easier for the compiler. Hardware features that guarantee the latency for both operating and moving data along with the entire system being non blocking is what really gives our compiler the information necessary to efficiently pipeline things.
Our initial target markets are those where programs would either be running on the bare metal (the basic program instructions running right on the cores, like an embedded system) or at most a pretty basic RTOS. From the bare metal standpoint, we can still have memory segmentation just like any other system... I would say it is even easier for the compiler to do that on our system due to the fact that all of the physical memory addresses are part of a single global memory map.
So the target market is .. high mips/watt microcontroller or DSP?
If you have fully software managed memory it sounds like any binary running on the system has full access to any other memory? This is kind of the opposite of ARM "TrustZone".
Edit: I'm just asking these questions because novel architectures tend to sink without trace and the small-system world is currently dominated by ARM. You need a real "wow" factor to get people to change their tooling.
Our focus has been on floating point performance. Originally we were targeting high performance computing, but have since expanded to high end DSP applications (Think mobile base station processing for LTE-Advanced and "5G").
For memory protection, the most traditional way would be leaving it up to a RTOS or microkernel. Something very small and verifiably secure like seL4 is something we want to port.
We have not made it a huge priority to start of with as our customers have a small number of applications that are being ported and are isolated on the system. As each application needs to be recompiled for our architecture, we think that memory segmentation done at compile time is good enough to start with (in these limited cases).
I'm not talking about microcontrollers in that case... Our chip has scratchpad for each core (128KB times 256 cores) with single cycle latency, along with attached DRAM. We have a whole lot more on chip memory that is 4x faster while using 1/5th of the power (on memory operations alone). Overall, we are aiming for a 10 to 25x energy efficiency improvement over CPU and GPUs for floating point heavy applications.
>managing scratch memory statically can only work for very high level (and very domain specific) languages. Hardly possible for C.
Well, we've solved that. Looking forward to sharing that in the near future.
If you really did, you've got much, much more than merely optimising a memory access. Consequences for the static analysis can be enormous. Looking forward to seeing your publication.
I hope you realize that managing memory latency has been the most fundamental motivation throughout Intel for 25 years. It isn't a problem that is a matter of a revelation about the problem itself.
I speed up programs all the time by reorganizing how memory is layed out and accessed. Many time by factors of 12x or more.
I would think that making SIMD, parallelism, and multiple simple loops instead of one bigger loop much easier to program around would be much more realistic. Something like a fusion of ISPC, Rust, C++11, and Julia.
First off, I was referring primarily to memory latency between L1 cache, which has improved over the past 2.5 decades only through the combination of Moore's law getting the wires shorter (which is going to end soon, at least for silicon) and increasing clockspeed (which really ended with the breakdown of Dennard scaling a little over 10 years ago). Intel's L1 cache latency has not improved in almost 10 years, with it still at 4 cycle latency (at best). The improvement has only been that there is more data you can access at L1, but the time to data hitting your registers has not improved at all.
Our scratchpad (the analogous term for software managed memory, in comparison to a traditional hardware managed L1/L2/L3 cache system) for instance has single cycle latency along with zero bus turnaround. Along with our ability to guarantee memory latencies between any locations in memory, our whole goal is to try to never have a wasted cycle.
>I would think that making SIMD, parallelism, and multiple simple loops instead of one bigger loop much easier to program around would be much more realistic. Something like a fusion of ISPC, Rust, C++11, and Julia.
What about functional programming? IMO the biggest benefit from programming without state is that order of execution does not matter. Thus programs can be parallelized trivially. Under the hood you end up with "multiple simple loops" without really even trying. I think when more people catch onto this, we're going to see a rise in functional language usage because of how easy it makes parallelism.
Those numbers are amazing, do you perhaps have a citation ? I'm a machine learning researcher and would love to include them in talks about data-aware algorithms.
This is also true for GPUs - in cryptocurrency mining every memory access equals more money spent on power. If you can make equivalent (in terms of throughput) compute/memory trade-offs, you always go compute.
While it takes ~100 picojoules to do a double precision floating point operation on a Ivy Bridge Intel processor, it takes 4200 picojoules to move the 64 bits from DRAM to your registers. Most people assume that the huge power usage is because you need to move data from off the chip, but the reality (and surprising fact to most people) is that over 60% (~2500 picojoules) of the energy usage of moving the data is consumed by the on chip cache hierarchy. That doesn't mean the SRAM caches themselves, but all the additional logic that makes it hardware managed (TLBs, etc) that give you functionality like virtual memory translations, cache coherency, etc.
Getting rid of all of that cruft that has been added since the 80s to make programmers lives easier would actually reduce power consumption and latency significantly... My startup is working on that problem by removing all of that additional logic from the hardware and instead having it managed at compile time. The best thing though would be having programmers really think about locality when writing their programs though.