Mainboard as Internet Router + Wireless AP + NAS Project

[D.H]

@Jeremiah_Jones

you can set resource limits,

My plan was to set resource limits via systemd slice groups. Not sure which is more efficient, that or containerization, but I suspect systemd might be.

@Peter_Schofield

you can’t readily (if at all) use an intel wifi card because they don’t support AP mode, they’re client only cards.

I don’t think this is 100% correct. My AX210 is advertising AP mode capability in iw list, but I believe that is available only on the 2.4ghz band, per this site: en:users:drivers:iwlwifi [Linux Wireless]

I will take some time to experiment and see what I come up with.

Many 5ghz and 6ghz seem to be a problem here as well, because the LAR is not hearing any transmitters on those frequencies, so they are defaulting to disabled. Going to be another head scratcher I think.

[Jeremiah_Jones]

@D.H

My plan was to set resource limits via systemd slice groups. Not sure which is more efficient, that or containerization, but I suspect systemd might be.

At least in LXD, resource limits are set via the kernel Cgroups features. This seems to be similar to how the systemd slices are implemented. The LXD resource limits are very nice in that they can intelligently kick in; from the first link:

When using a percentage value, the limit will only be applied when under load and will be used to calculate the scheduler priority for the instance, relative to any other instance which is using the same CPU(s).

My gushing love of LXD aside, there should be less overhead by doing everything natively. Although, I suspect resource usage will not start to play a role until surpassing the “1GbE with QoS and several vlans” level of routing, since the humble Raspberry Pi 4B can do it with only one of its cores. A real question is how you’re gonna hammer this thing!

[D.H]

Mainboard finally showed up a couple of days ago. Just a few “unboxing”… Err, unbagging pics for y’all:

Inner box removed from outer shipping wrap, which was expanded paper padding packed:

Inner box again packed with expanded paper padding:

Anti-static bag inside of the expanded paper padding:

Finally, the empty board itself:

One odd little thing, it did not come with the tiny screw to hold down the m.2 2280 drive. Erps…

[X_S]

Keeping an eye on this one too. I’m having router problems and am considering buying a Framework 11th gen i5 mainboard (once they become available here) and turning that into a router. I’m leaning towards pfSense, although I wouldn’t mind something Linux-based.

I only really need USB-C to RJ45 adapters for my WAN and LAN ports, as I have a dedicated WAP already. Still, it will be interesting to see your entire parts selection, as it may give me some new ideas.

[balupton]

I’m currently running a diy nas on gocryptfs on btrfs across 6 hdds attached to a raspberry pi 4 shared over samba, which works great. However I’m wanting to move to moosefs on xfs for the extra flexibility, however moosefs requires lots of ram, so I’m considering a framework mainboard as the moosefs master then all other devices (macos and raspberry pis) being chunk servers and clients.

[D.H]

Interesting, I’ll definitely check into those!

[Peter]

Is there an update on this ?

Now that 12th gen boards are available, this looks like cool use case for old boards.

[D.H]

So a partial update of my work so far:

Mainboard received with BIOS 3.07 installed. Flash to 3.10, initial setup, etc complete done using USB BIOS flash while following Getting started guide with a bare motherboard

Did some initial power testing, out of the box with arch linux installed:

• Draw is <0.5W when off
• ~4-5W idle at graphical desktop
• about ~40W max in stress test operations

Above numbers are including the power use of an nvm.e SSD, 2 sticks of RAM, a 3rd party USB hub with HDMI out, but no wifi card intalled yet. Started seeing voltage sag down to ~19.3V with an 87W USB-PD power supply, so I am really not interested in trying to push to 60W turbo.

Check combinations of RAM sticks, my 4GB stick can go to 3200MT/s, the 8GB tops out at 2133MT/s, and both together are limited by the slow at 2133MT/s. CL22 timings for each config. Plenty fast for a router, AP, NAS, etc.

I did confirm that AX210 is capable of ap mode on both 2.4GHz and 5GHz. I was using kernels 5.15 and 6.02 with iwlwifi module and intel-ucode 20220809-1 for testing. From my research, it appears that AX210 is still unable to ap mode on 6GHz due to Intel’s overly strict interpretation of regulations and locking it in their binary blob. If someone knows of a way around this, please let me know.

So I went searching for a better wifi card for ap mode on 6GHz. There is the Qualcomm QCN9074 and 9024, and possibly one other model, but as far as I can tell those are single band on 6GHz only, and USD$100-$200 each. Looks like the best candidates were from Mediatek, specifically:

• MT7921K, the pcie variant, used in AMD RZ608 and clones using m.2 a+e key form factor, or
• MT7921AU, the USB variant, or
• MT7922A, pcie, used in AMD RZ616

More details here: MediaTek - WikiDevi.Wi-Cat.RU

The former was significantly cheaper (~USD$19), so I ordered it, and it is on the way. If what I am reading about the mt76 kernel driver is correct, it should support AP mode on all three bands. Testing to confirm soon.

Oh and I ordered a USB-C powered ethernet switch from Cable Matters (model 201069, USD$29). It is here, has a Realtek Semiconductor Corp. RTL8153 chip in it, and seems to be supported by adding the r8152-dkms AUR package. I found some reports about it freezing up the entire LAN if the computer it is attached to goes de-powered, possibly even after a reboot, so it will be interesting to see how that turns out.

This project has definitely moved up my priority list as my R7000P has decided to completely stop making an AP on 5GHz, despite all my old tricks to getting it fully factory reset. Hopefully I can have my DIY 6E AP up soon instead.

Next to do items:

• Update BIOS to 3.10
• 3D print case
• Start network operations, nic bridging, etc testing

[D.H]

Cable matters switch with integrated USB-C ethernet nic is working as expected with the kernel module.

Using dnsmasq + iptables in initial testing for dhcp and packet forwarding, so far so good.

Relevant guides:

• Network bridge - ArchWiki
• Internet sharing - ArchWiki
• dnsmasq - ArchWiki

Next: Firewall configuration

[D.H]

Testing MT7921K wifi card as access point hit a brick wall, as the dev board Framework sent me appears to have a defective m.2 slot for the wifi card. I have swapped the MT7921K card into my main laptop and it seems to be working out of the box on arch 6.0.7 kernel, though. Looking for a different hardware dongle to make the AP with instead. Comfast CF-953AX, which uses an MT7921AU chip appears to be the next logical choice, but is not well stocked at the moment.

[D.H]

Status update:

Working:

• USB ethernet nic for WAN using either DHCP and Static IP options ( I have a small issue with my permanently assigned public static IP with my ISP that needs sorted )
• hostapd single wireless interface with WPA2 (the wifi dongle I have on hand doesn’t support WPA3)
• usb-c powered ethernet switch
• above 3 interfaces renamed and controlled by systemd (networkctl) and configured automatically at boot time
• bridged LAN switch and wireless AP interfaces
• dnsmasq for LAN IPs, local DNS, upstream DNS resolving
• wireguard server / client peer pairing acessible from WAN, and forwarding between LAN and VPN subnets

So the router is in active use now. The only USB wireless AP nic I have is 802.11n at best, so it is a bit slow. Still using old routers as additional 802.11 ac APs for the moment. Comfast CF-953AX on order from aliexpress, which should give me 802.11ax, 6GHz, and WPA3 availability. Supposedly ALFA is going to be coming out with MT7921(au) based dongles / adapters “any time now”. Will order some of those as well when they are available and integrate. That will let me simplify down to just this router and an additional RaspberryPi4 to make the APs I need for full coverage of the house.

ToDo:

• hostapd multiple wireless AP interfaces via systemd instancing
• WPA3 testing
• LAN DNS resolving for wireguard peers
• other items from the original post that are not on my mind yet lol
• Write up the recipe and options here of course

[Ian_Comings]

My 2 cents:

OPNSense for the OS. It is more up to date than PFSense and supports more hardware.

Ditch the wifi card and instead use a standalone WAP like a Ubiquiti AP connected to a switch. Use the board strictly for the router function.

[D.H]

Your Router Sucks. Build Your Own Instead!

Focuses on pfSense and OpnSense for those of you leaning that way…

[D.H]

I am starting research into the distributed, redundant filesystem step of the project. Up until now I have been keeping backups using systemd timers firing rsync actions. That is an okay solution, but I feel like it could be better. Important data to be backed up is personal files, pictures, and music recordings that would be next to impossible to re-acquire if lost. Secondary data are DVD and BR movies that I have ripped to my personal media collection. Maybe 4TB total at the moment, probably will grow over time.

Features I am looking for:

• presents a single unified filesystem / mount source
• posix-compliant with respect to file ownership and access security
• Must recover from split brain, loss of network comm, device powered off, etc automatically
• Able to utilize unequal sized partitions and disks highly preferred
• Low overhead in terms of memory, disk IO, network IO, etc
• No SPOF, either main controllers or any disk(s)
• Hot resize, add, remove disks to the cluster would be highly preferred, but not strictly required.

Just (re-)starting my research into the subject. I found this comparison article and read through it: https://computingforgeeks.com/ceph-vs-glusterfs-vs-moosefs-vs-hdfs-vs-drbd/

Curious to hear from your experience with any of the above distributed file systems, ease of setup, use, maintenance, etc.

Thanks

[Kyle_Chase1]

I’d give VyOS a try. I was going to try it myself, but I ended up using my main board as a server. If someone wanted to send me a dev board, I could fill it with 3 2.5G ethernet expansion modules and try VyOS on it.

[D.H]

Quick update:

• Replacement mainboard came from Framework, and I verified that PCI was working with the MT7921k m.2 card. Packaging was way different this time, a regular cardboard box but two inserts to keep the Mainboard suspended into the middle. Box arrived in perfect shape but the heat sink has one corner bashed in, and the whole Mainboard was warped a bit. Not enough to keep the screw holes from aligning, but it noticeably sags downwards in the middle.

• CableMatters USB powered nic + switch was having some reliability issues. That and the USB NIC I am using for the WAN port are both Realtek 8152/8153 chip based. The firmware for these in Linux seems to be far from perfect, and Realtek gets the blame. I’m getting better stability out of them by turning off things that require the NIC itself to do work. Checksums, forwarding, offloading, etc. Stability is definitely better but still seeing quite a few USB resets, errors in journalctl, etc. At this point I’m going to say r8152 based networking devices are to be avoided if you have any choice in the matter.

• CF-953AX (based on MT7921u) arrived today. Got as far as testing WPA3, 802.11ac with 80Mhz channels and 256QAM, so basically the same phy link rate as my Wifi 5 R7000P. The chip in this reports it is capable of 802.11ax (HE), 6Ghz, 1024QAM, quite a few other tweaks to get faster link rates. However, hostapd is not accepting the configuration line item for 802.11ax, which I think was not configured in when the arch package was compiled. Looking at doing a custom compile for hostapd possibly. Might try the linux-wifi-creator script also.

• 3d printed the back case plate from the Framework Tablet project, a 24 hour long print on my big dumb brute of a printer. Came out pretty good though. Mid layer of the case next.

• Framework still does not have an ETA for when batteries can be ordered from the Marketplace. Big sad face.

[D.H]

Confirmed that arch hostapd package 2.10 from earlier this month still does not seem to have 802.11ax compile flag enabled. There’s an open bug about this from 2 years ago that package maintainers seem to have missed now that 2.10 is out. Trying to get their attention about it, but fly spray (arch Linux bugs site) refuses to send my sign up email. Yay.

Compiled hostapd from git repository from w1.fi , set compile flags, got 802.11ax working on 5ghz on the CF-953AX. But, I can only get 20Mhz width channels working so far, so my phy link rate is still ~130mbps or so. The folks on the morrownr/usb-wifi GitHub issues queue have been having some ideas though.

Need to try on the MT7921K and AX210 m.2 cards soon.

All the 6 GHz channels are marked no initiate radiation, so I’ll have to figure out what regulatory domain setting will unlock those for me…

[D.H]

Small update from today’s testing:

• AX210 still refuses to even scan for or connect to 6Ghz APs due to flawed and out of date implementation of Location Aware Regulation in firmware.
• AX210 does NOT seem to be able to make a WPA3-capable AP
• MT7921K and MT7921U DO make WPA3 capable APs !
• MT7921K and MT7921U DO make 6 Ghz APs confirmed.
• I am still struggling to get more than 20Mhz channel size from either of the MT units, but I will post an example hostapd.conf file as soon as I figure this out.
• AX210s are being taken out of all of my Frameworks, and will be put up for sale because I refuse to put up with Intel’s poor firmware any longer. MT7921K’s are on sale for $14 currently.

[Jieren_Zheng]

It is interesting that Mediatek wifi firmware is better than Intel.

Out of curiosity, do you know of any decent m.2 (A-E key) ethernet cards? I saw Lattepanda has M.2(A+E Key) to Gigabit Ethernet Module Adapter for LattePanda Alpha and Delta - DFRobot, but not sure if there’s one with the port directly on the card itself.

[Fraoch]

@Jieren_Zheng

Out of curiosity, do you know of any decent m.2 (A-E key) ethernet cards? I saw Lattepanda has M.2(A+E Key) to Gigabit Ethernet Module Adapter for LattePanda Alpha and Delta - DFRobot, but not sure if there’s one with the port directly on the card itself.

There’s this:

http://www.commell.com.tw/Product/Peripheral/M.2%20(NGFF)%20card/M2-225.htm

but the port is on a PCI bracket and attached using a header cable.

Also I don’t know exactly where it can be purchased.