Rackmount Modular Framework Server

@dashtesla the main issue with putting these in to a rack aside from not being ECC is really a cooling issue, servers are designed to push air front to back, the framework main board is top/bottom to the side or back if you mount width ways.

Now that being the case, that then essentially ties you to 1U height to fit the 4040 fans, although with the right design you could halve that I expect by mounting each board in the opposite orientation so 2 units Ying Yang together, that could in theory double the compute per U, and in theory allow 4 per 17” 1U.

In increasing the density you end up losing access to the onboard PCIe for other uses such as SATA expansion or 2.5G Ethernet off the WiFi slot, which is then why my initial design doesn’t use that approach… ideally I’d like to be able to fit in a full pcie slot to allow a standard 10G SFP+ card, but I need to do some more investigation in to thermals…

In terms of cost for a data centre, I’m not sure these boards would really stack up either, I think they make sense if you upgrade your laptop and then use it for a home server, but costs start to mount up by the time you factor power supplies and cost of thunderbolt expansion cards vs pcie.

However my main blocker right now is turning it on! The onboard power button is tiny and not really practical to put levers in place to get to it. So really I need someone to develop an interface board for the header so I can put in a more conventional switch… I have a couple of other projects nearing completion right now, once I have cleared those off I’ll start back on this and hopefully someone has released one by then.

If you have specific thoughts, requirements or have found some specific hardware that you think would be of interest for this project please do post up here!

2 Likes

Also for context of vertical clearance for PCIe, these cards are the prime examples, I’m still not sure the SATA will work without very expensive 90 degree cables.

3 Likes

@n3rding Those are fair assessments however I do believe that the cooling can be solved with either a custom heatsink that is more compatible with a front to back configuration, I don’t like the idea of 40mm fans they’re either very noisy of very unreliable and the way i see there is no need for fans if you rely on case cooling you just need a heatsink that allows the right airflow with thicker fins.

I’m not good at CAD otherwise i would love to draw it to make people see what i mean here but also 2.5G ethernet isn’t really that useful for server use, 2.5G is a niche residential standard that in my opinion should’ve never existed to begin with and we should’ve gone for 10G and 40G instead specially considering that most of the high bandwidth networking is used for iSCSI/SAN etc and 250mb/s is hardly fast enough to max out a sequential hard drive but not even a sata 3 ssd let alone SAS 3 or nvme/u.2.

The chassis is quite simple to make it’s a bunch of metal some cables and connections and existing hardware, for example all your thunderbolt to pcie problems you just need an egpu board i even have one that takes itx gpus that i can see with some minor modifications being able to fit into a slot module that would then take pcie cards.

As for power requiriments server psus are very affordable these days and so are breakout boards, 12V is all you need but wouldn’t be hard to get any higher or lower voltage either if needed, we’re talking hot swappable dual/triple psus here like every dell/hp/lenovo server has and there are plenty of such psus for next to nothing with 12V at the end and a simple board that combines the output of multiple psus into one line and sends a signal to the motherboard reporting it’s status.

It’s a lot of little things sure and I do agree it’s more of a framework project than a community one but with some modifications we can have what is basically hot swappable nodes and a whole new standard of server hardware and far less waste, considering the chassis wouldn’t be tied to a single generation of computer they can last easily a decade and so can these nodes allowing for easy upgrades and many custom configurations.

I sure would love to work on such a project with some bright minds like wendel from level1techs and actual people who use server hardware on a daily basis and understand what the industry is missing.

I also see a lot of potential for ARM and AMD here specially in the future and as for ECC that’s only a problem with intel because they refuse to get on with the times and allow unbuffered ecc for anything that’s not a xeon if they were to release a mobile workstation/multi-purpose cpu even lets call it a Xeon M… done problem solved or maybe just enable ECC support like AMD did and retire the old ways for good we can’t have this much ewaste in the world and things that can’t be reused for other purposes due to corporate negligence and remember it’s called “Rare Earth Elements” for reason.

1 Like

Ok, but a custom heat sink alone is an expensive step, and still then you need the 4040 server fans to push the air. Agreed it would be better with a heat sink in the right orientation, but then this adds additional cost and waste which isn’t really the purpose.

With regards to power, I was referring to powering the board, which isn’t as simple as applying 12 volts, as it needs to use something that can negotiate USB-C.

Regarding the 2.5G point, this is a limitation of of the port I was referring to, as I said later, ideally id want to be able to run 10G, but over thunderbolt thats insanely priced and very bulky, the only real option here is to take a full sized slot from the onboard PCIe.

Also to add with no ipmi it’s a less than ideal solution for a data centre.

At that point you’d probably be better going for a single server grade cpu with a lot more cores, which would take less physical space if that’s what you are looking to achieve, would likely be cheaper too.

I like the idea of the blade type principle, especially potentially having a backplane which I don’t have in my design currently (but might add this if I can sort an elegant but optional solution, but I don’t think it scales in this instance. From my design concept above, you can get multiple units side by side if you wanted to, but I’m fully expecting no one to do this.

1 Like

@n3rding You can also use some plastic pieces that can take air and channel it through the heatsink and then out the back in an S configuration, it requires a little more static pressure but should be a cheaper alternative while reusing the heatsink.

All you need is a little chip that will negotiate via USB-C that can be very cheaply integrated into the module and run off 12V as the entire case would.

As for thunderbolt stuff being expensive sure but it is an alternative you also have USB but in general you’d be looking at hot swappable slots that simply would take the usb-c/thunderbolt from the board and route to the back where it would then allow an expansion module to be slotted in that can be an affordable usb module or a more expensive thunderbolt module that would allow the pcie devices, now as with everything it gets cheaper with industries of scale so the problem is just taking the plane off once it’s up in the air it’s easy flying :slight_smile:

Case can have an IPMI that can also communicate with one of the USB-C ports allowing physical kvm and the IPMI can also be a module that takes something like a raspberry pi or a custom module that adds the video encode h265 plus audio keyboard and mouse allowing many other uses not just IPMI which would increase the interest on such a module.

Granted that you can go with a bigger server but it also creates other problems, it’s not a comparable solution here you’re looking at a highly customisable machine that you don’t need to turn off to upgrade a lot of things and that has many individual physical machines that can be rented to customers as bare metal and not virtual machines, and don’t forget redundancy and the ability to communicate with each other via thunderbolt as well that’s a 40gbps connection which would be very useful for failover/hypervisors.

As for 10G networking being bulky along with thunderbolt in general well it’s lack of engineering you can make everything fit in a lot less space thus why my module idea you can design these modules to be able to take single slot pcie cards and have the hardware and airflow it needs all inside the module making it a lot more manageable and practical but also 2.5G being able to run off USB makes it a lot cheaper and useful to some people that might even prefer 4x 2.5G vs 1x 10G etc

What I like the most is the idea of creating such a modular system that would for instance allow a small home user to run one of those system boards out of a little case you can put on your desk or somewhere out of the way or have a massive 4U chassis with several of those, the whole point is this ability that you can take something and put it somewhere else with very little effort just like you can with gpus and hot swappable hard drives that you can take from one server and put that into another from a similar brand but in this case an entire system, doesn’t have to be expensive or over-engineered just has to do the job and earn it’s worth.

I feel framework could work on such a system but there will be more of a market for something like this in the future vs today and there will be many more spare boards in the future vs today, also think framework will label their board sizes/designs with generation in mind after a few years they will likely have to change their design and layout to accommodate newer technology thus the current laptop chassis.

Everything I talked about is just ideas and a look at how a future could look like in terms of reusing parts and recycling things but also making everything easier to put together and highly scalable/customisable and hopefully open standards that other companies can also adopt, you know when you have something around like rack rails for an HP server and you’re ordering rack rails for a dell… sucks right :confused:

1 Like

Airflow isn’t an issue. You can turn fluid directions, then you can turn air flow direction. It’s all in the air pressure, fins and ducts. Every turn would incur a penalty in turbulence and energy loss, but it’s not a feasibility issue, more like an efficiency issue.

2 Likes

2.5 and 5gigabit were invented because the prices on 10gbit supposedly weren’t coming down fast enough for them to be viable for home use. As you said, they’re a waste of effort because most decent kit that supports them also supports 10gbit, though 10gbit copper nics/modules get very hot. 25 and 40gbit over copper also exists but I have yet to see any nics or kit for it.

1 Like

@Peter_Schofield I also don’t get why copper became standard pretty much every 10G connection I have here i use SFP+ and LC OM4 patch cords and every computer I build I always need to go out of my way to get an SFP+ pcie card because it’s always the same 1G or 2.5G copper which to me is useless other than for LAN/Internet but for SAN it’s ridiculously slow moving data from nvme to a NAS at 125-250mb/s is just painful specially if you’re dealing with 100GB-1TB of data just takes too long.

I feel like at some point SFP should’ve been made the standard even if it’s only 1gbps since it gives people the option of what to go for so you always have the option to go with SFP direct copper, fibre or rj45 etc and all that straight from your onboard nic.

I also know that many older cards that support 10G aren’t compatible with 2.5G and 5G which is again another reason why I always avoid them since you need a card that specifically supports it.

I also wish SAS became the standard it would’ve made a lot more sense back in the 2000s-2010s but that goes back to the days of SCSI/ATA now it’s all down to nvme/u.3 which will hopefully make this whole thing an obsolete discussion but if want to use any of my old SAS drives I also need to go out of my way use more pcie lanes and a controller just to use a hard drive and in this day and age this shouldn’t be a problem anymore it feels like companies just want to force people into using worse standards in general and make “enterprise” gear just ridiculously expensive for no reason other than greed.

I have an LTO 6 tape drive in a personal server and it obviously requires a sas controller, same for a sas expander to use multiple drives through the same hba I will also upgrade that to LTO 9 when I can since I do appreciate my offsite tape backups so I feel sas will live on for a while.

As for pricing sure 10 years ago 10G was expensive but at this point it’s really not that expensive anymore you get 10G SFP+ cards from 30-150 USD/EUR/GBP depending on where you get them from and how old they are, patch cords are also cheap and so are SFP+ modules only really expensive piece is a 10G SFP+ switch specially if you want more than 8 ports or a 40-100G uplink but it should also become a lot cheaper soon.

Something else I don’t appreciate is that in order to escape the usual 20-24 pcie lanes you have to go into workstation gear, so if you want to run dual gpus or multiple nvme or even have a sas controller, 10g ethernet and gpu at x16 anything that’s out of the ordinary at full speed you already need to get workstation hardware and I don’t see why people think having 20-24 pcie lanes is acceptable these days specially if you’re paying 300-500+ for a cpu.

Game is rigged :slight_smile:

because its cheaper, more resilient to abuse and simpler to install and maintain, especially when offices get reorganised. fibre to the desktop has been tried many times and it causes more problems than its advantages. It’s great for backhaul but not so good for endpoints at scale.

SFP/SFP+/SFP28/QSFP+ etc are not actually standards, they’re informal agreements which is why modules/switches/nics from different manufacturers are often deliberately incompatible with each other, manufacturers code their equipment to only work with their own kit. This vendor lock-in actively prevents more widespread use.

I agree about SAS and PCIE. SATA HDDs are interoperable with SAS controllers so the only reason its not done is because SATA is cheaper for the manufacturers. There’s no reason SAS controllers couldn’t be integrated into the CPU dies instead of SATA. CPUs used to have more PCIE lanes, (ivy bridge i7 had 44) but intel stopped that from haswell because the 3770k was such a good seller it was drawing people away from their much more expensive i9 ‘enthusiast’ parts. so they gimped the consumer line to 16 PCIE.

@Peter_Schofield I agree that SFP is another mess but there are ways around it and not every manufacturer will blacklist modules either.

If your average office had fibre installed back in the day 10-20 years later we would still have the same cabling future proofed for many decades to come, reality is copper from the 2000s is likely cat 5/5e and then again everything was replaced in the 2010s for cat 6 or cat 6a and that won’t do 10gbps for every office or every length or even every home depending on how big the home is specially for the uplink and in the 2020s faster than 1gbps internet is becoming more of a thing at least here in europe there are plenty of docsis and ftth gpon stuck on 1gbps but i see it slowly moving to 10gbps in locations where it’s viable to upgrade or more often than not facilitated by the city but if you consider 2030s 2040s then you’re also recabling your copper every 10-15 years and will people switch to that massive thick CAT 8 now? I don’t think so… I have one I got a few years ago for testing and fun and it’s so thick you can tow a car with it :slight_smile: If I was building a home or an office I would 100% go for fibre so I don’t ever have to worry about what goes through it if it’s 1gbps or 100gbps and I know it will last for a long time if done properly.

And yes I am rooting for u.3/nvme all we need is really ability to hot swap pcie devices which already exist and the end for sas/sata everything can just be straight pcie x1-x4 etc we just need more lanes.

I’m running 10G SFP+ where it matters and for much less than the cost of 2.5G, I agree the interim speeds before this are almost pointless, if the focus was on 10G copper from the start then we would have many different speeds that can’t play nice with each other, 2.5 didn’t really give a significant performance increase and it should have been a 10x jump the same as ever previous generation. However NVME has only really just started, I think if that came earlier then that may have pushed their hand.

In terms of this project though, the only viable option really is to piggy back off the mini pcie slot and allow a full sized card to allow SFP+ and then nvme storage over thunderbolt if you want to push 10G, 2.5G is possible using the WiFi slot

Can confirm. The SPF+ UniFi modules I have for 10GbE are very hot to the touch. Or only in the range of 70°C. You wouldn’t want to hold it for more than a few seconds. When disconnecting them I have to unlatch. Let it sit. Pull it out part way. Let it sit, then remove the rest of the way.

That’s why some people put a few heatsinks on to the outer cage on the NIC, however the NIC itself is also a challenge in some cases, the HP NC523SFP which is usually one of the cheapest NICs all over ebay gets incredibly hot, but going for a better card at not much more can drop this a fair bit, I’m running the Solarflare SF432 which performs well, both NIC and 10GTek module both running cool to the touch at normal workstation internet browsing usage. (Like 30 degrees C)

Both both parts will need to be considered if I end up making room for one in the design as I expect most people will want to print in PLA and that will start deforming if there isn’t some form of active cooling in place, (I think though I’m going to struggle with clearances unless I put the nic in a separate compartment and even then there’s a challenge of cable length and data degredation

how did I just find this thread.
I am built a 2 node version : PlexiCluster - Framework Laptop Boards for Rancher and Harvester - YouTube
And I am working on a 6 node version now.

Power is going to be the biggest enemy. I would love to find a way to power all the nodes with a single power supply. In fact the psu in the pictures are being returned. They can’t handle all the ports being used. I have new ones coming.

Heat is not currently an issue with the 2 node cluster. I am waiting to start assembling the 6 node to see.

8 Likes

I can imagine 2 would be fine as cooling is on both sides, I expect >2 might be a problem! I’ve had to put mine on hold for a few reasons, but now I can get expansion cards for a bit cheaper I’ll pick this back up again!

3 Likes

IT LIVES!

7 Likes

@Andy_Clemenko how’s it shaping up? I know there’s a lot of discussion above around cooling, and yours is pretty much worst case. I’m also curious about noise.

(I’m interested in building my own for home lab reasons–need a cluster test bed. Definitely thinking about including batteries, but I did read the above warning.)

The “BladeRunner” is great. There are ZERO heat issues. I was thinking about changing the orientation of the case. But since its working…

Do you have any designs for the case yet?

3 Likes

Blinking LEDs can be turned off in BIOS, just enable standalone operation.

Aside of that, have you considered connecting the nodes using Thunderbolt4 networking?

1 Like

Yeah, nrp responded here:

1 Like