As sad as this is, I find it very refreshing with a transparent and honest update like this. I've seen so many initially promising hardware projects just fizzle out in uncertainty and radio silence.
Especially that they are open with reasons why, what they feel they could have done better, and releasing blueprints allowing others to pick up.
No doubt 2020/2021 must have been the most challenging years in decades for an indie team to launch a new hardware project like this.
I was looking quite seriously at the Kobol64 last year; the only thing that eventually made me not preorder was realizing the RAM would not be sufficient for my purposes (ZFS-backed Glusterfs nodes).
I'm quite happy with ASROCK RACK's X570 boards. AMD-based, 10GBe, up to 8 SATA straight from the board, up to 128GB RAM, come in both mITX and mATX variants. And they have IPMI! And an integrated super basic GPU so you can run the non-APU Ryzens without a dGPU.
If you don't enjoy DIYing too much I'd recommend the mATX ones - the form factor and cooling alternatives for the mITX are... unorthodox.
But yeah, comparable non-x86 accessible for prosumers will be years away.
This is exactly what I've done, by the way. Same mITX motherboard you cite below with a Ryzen without integrated graphics (since the board itself supports VGA). 10 GbE and a whole bunch of SATA lanes. I didn't bother with trying to get it into a real chassis. I just grabbed a desktop case from Fractal Design that can hold 10 disks and it fits fine in my wall-mounted entertainment center cabinet below my television. It's in there with my router, a switch, and a few Minisforum small form factor Ryzen PCs. The only "gotcha" with cooling is you need to use a CPU cooler meant for Intel even though the chip is Ryzen, but the board documentation clearly tells you this. I'm just using simple Noctua low-profile fans. I also cut some holes in the cabinet and put in USB-powered cabinet fans, so I can keep the doors closed.
So sure, it's DIY and some work, but putting together the NAS itself and putting together the cabinet, including the cooling and wiring, only took one weekend.
It consumes more power compared to ARM, but I'm cooling a 4-story house with 18 foot ceilings in Texas anyway, so it's not a noticeable difference.
Besides some of the ARM options like the Honeycomb that have been mentioned, there are also the POWER9 systems from Raptor Computing Systems. Much more expensive than the x86 systems, but I feel compelled to mention them, since they have mATX boards with most of those yummy features (fewer SATA ports but higher RAM capacity, for instance).
I have looked into exactly those boards and they by far seem like the best available solution. I'd be really interested in hearing about your experience with them
Which type of case/chassis are you using for your set-up and how many nodes are you running?
I have 3x X570D4I-2T with Ryzen 3600. It's my understanding the new Zen3 should work just fine as well.
* I gave up on finding a chassis that wasn't either gigantic, too tight, or crazy expensive. They run naked resting on cork-boards on a very cheap DIY shelf that I've hung some 140mm fans on. Looks a bit cyberpunk and swapping drives is super easy. You may have better luck with chassis if you're in the US.
* ECC RAM supported, which is nice. I've understood you can't surveil the ECC, so I've seen it described as "semi-ECC". This is a Ryzen thing, not an ASROCK-RACK-thing.
* ECC DDR4 SODIMMs are expensive and rare. If you need ECC and lots of RAM but don't care about the small form factor, a bigger board will be a lot more cost-efficient. I got my sticks from Nemix and Samsung.
* Intially I had stability issues and errors in memtest. Completely disappeared after putting them behind a UPS; turned out the culprit was power fluctuations in my previous house.
* They need a PSU with EPS (my old leftover 15y/o PSU in the closet only booted the IPMI, not the board itself. Silverstone SFX 450W is great and cheap)
* (Specific for mITX) The CPU socket has bastard Intel fittings. I got Scythe Shoten and Noctua on there fine in the end, but it took some anxious-inducing creativity with M3 screws/nuts and DIY spacers,
* IPMI and monitoring are great for initial setup and if something would go wrong. Flashing UEFI/BIOS is a breeze.
* Apart from the on-board SATA I plopped in 4xM.2 PCIe bifurcator cards from 10Gtek - so besides the boot drive I can have 4xM2 NVMe drives on each board. Works great.
* I'd really really like to figure out something more compact PSU-wise - either Meanwell or Pico-style - but electricity is not my strong side and despite researching quit a bit I still haven't figured out exactly what I should buy and how I would hook that up.
Overall I'm happy. It's quite impressive how much IO you can get out of such a small board. But since I moved to a bigger house and the mATX model came out I'd have bought that instead if it was today.
In case you haven't checked out the servethehome forums already there are some people writing about their boards there as well.
Oh, and since you mentioned redundant storage: Glusterfs is great, except when it isn't. I have issues with it every now and then (files being stuck in heal without going in split-brain requiring low-level manual intervention, and fuse mounts on other nodes randomly disconnecting with "transport error" until I manually remount, despite always having min 2/3 nodes continuously online). I kind of wish I went with Ceph or LizardFS or something instead but I sunk so much time into Gluster already and have better things to do than start over with something new - and who knows if the grass would actually be greener.
That's a fantastic rundown - thank you so much for taking the time to write such a comprehensible explanation!
I live in a rather small apartment (56 square metres), so I'm trying to find an acceptable trade-off between noise and space. I'm in Europe, so sourcing parts can unfortunately be a bit of a challenge, so I'll probably have to be creative to solve some issues (which honestly is also fun)
I had found a couple of 2U racks that could hold 2 mITX boards, but the lack of cooling options made me nervous whether it would get too hot
With you now writing that you would have gone with the mATX board if you were making the build today, I think I'll probably go with that instead - especially since I'd like to use ECC ram, and you mention that ECC SODIMMs are expensive and difficult to find
I've been reading a lot of reviews on servethehome, but haven't gone through their forums yet. I'll make sure to check it out - thanks for the tip!
I haven't yet decided whether I'll go with ceph, gluster, or something else entirely. I'm leaning towards ceph, but I've read several warnings about running only 3 nodes in a ceph cluster, so it's (once again) a matter of figuring out a good trade-off
In any case - you've definitely helped me out here and I really appreciate that you took the time to write such a comprehensive explanation
The power connectors on that board are strange - it's almost ATX12VO but not quite. Does the motherboard come with the 24-to-4-pin adapter?
Assuming you need AC->DC an SFX power supply is probably your best option. It's hard to find generic power supplies that have power good, 5VSB, and remote PS_ON. The generic supplies of reasonable cost are almost certainly less efficient as well.
Wow, that actually sounds awesome and something that could finally enable me to retire my shitty QNAP NAS and my Proxmox "Server" running on an old i7 2nd gen desktop...
If you don't mind, how much power/watts does one of these draw continously?
Isn't the reason why people are interested in projects like Gnubee or Helios that they want to get away from closed source hardware with rootkits like the Intel ME?
I thought this was the main criteria why such projects based upon RISC-V even exist.
Especially that they are open with reasons why, what they feel they could have done better, and releasing blueprints allowing others to pick up.
No doubt 2020/2021 must have been the most challenging years in decades for an indie team to launch a new hardware project like this.
I was looking quite seriously at the Kobol64 last year; the only thing that eventually made me not preorder was realizing the RAM would not be sufficient for my purposes (ZFS-backed Glusterfs nodes).