Framework for Storaxa replacement (Proxmox+NAS+Router)

Background:

I own an old server (i7-2700k+32Go RAM+5x16To in RAID5) for my homelab who serve multipurpose sus as NAS and hypervisor using proxmox.
I wanted to replace this server with the storaxa kickstarter campaign, but nothing go as planned.
So my target is to mimic the storaxa specs or even surpass it but on 1U server size.

The Old hardware:

- Intel i7-2700K
- 4x8Go 2133MHz CL10 DDR3
- PBZ77-V LX
- QNAP QM2-4P-384
- 1x10Gb SFP+ intel 82599ES
- 5x16To Seagate Exos X16
- 2x500Go NVMe Crusial CT500P5SSD8
- 1x500Go Samsung SSD 860 EVO
- 850w PSU Corsair RM850x

The New one (for the moment) :

- Intel i7-1280P
- 2x32Go 3200Mhz CL22 DDR4
- Framework Laptop 13 Mainboard (12th Gen Intel® Core™) BIOS beta 3.06
- QNAP QM2-4P-384 - Connected with:
    + TH3P4G3 (aliexpress /item/1005006130580555.html)
    + ELUTENG PCI-E 16x 3.0 Riser Cable Extension ( amazon /dp/B0B4DRHNC1) NEEDED to get full 4 NVMe
- 2x10GbE Intel X540-T2 (aliexpress /item/1005006268866821.html) - Connected with:
    + USB4 M.2 NVMe Enclosure 40Gbps ASM2464PD (aliexpress /item/1005006185746371.html)
    + M.2 to PCI-E x4 (aliexpress /item/1005004714035083.html)
- 10 ports SATA 6Gbps JMB575 + ASM1166 (aliexpress /item/1005005608262040.html) - Connected with:
    + M.2 A+E key to PCI-E x4 (aliexpress /item/4000207687691.html) (Working at PCI-E x1 speed)
- ORICO 6 in 1 USB 3.0 Hub with 4K HDMI, 2.5G Ethernet, PD 100W, 2 USB-A 3.0 Ports, 1 USB-C 3.0 Port (amazon /dp/B0BN7D63QD)
- 1x2.5GbE (aliexpress /item/1005003432672849.html)
- 5x16To ST16000NM001G
- 4x2To Kioxia Exceria Plus G2
- 1x2To Samsung SSD 990 PRO
- 1x500Go Samsung SSD 860 EVO
- 850w PSU Corsair RM850x

In testing:

- Power switch ATX-PS01 - Connected with :
    + Dual PSU Power Supply 24-Pin ATX Motherboard Adapter Cable (amazon /dp/B09B9GHFXJ) NEED to remove ATX power switch from TH3P4G3
- Simple power switch (amazon /dp/B0B2DKQH8Y) - Connected with :
    + Dual PSU Power Supply 24-Pin ATX Motherboard Adapter Cable (amazon /dp/B09B9GHFXJ)
- Radiator Fan Speed Controller 8x4-Pin PWM Fan (amazon /dp/B0C1CK3PV5)
- 5xARCTIC S4028-15K (amazon /dp/B09RK541T1)
- 350w PSU SilverStone FX350-G (amazon /dp/B07BVTT69K)
- Inter-Tech IPC 1U-10248 (amazon /dp/B00O234MNC)
- 10 ports SATA 6Gbps JMB575 + ASM1166 on :
    + 10Gbps M.2 NVMe SSD Enclosure Adapter (with M.2 to PCI-E x4)
    + 20Gbps M.2 NVMe SSD Enclosure Adapter (with M.2 to PCI-E x4)
- 2xUSB3.0 to 20pin (aliexpress /item/32436028406.html)
- ADT-Link R43UH 4.0 (aliexpress /item/1005006203494468.html)
- Carte Riser PCIe X16 to X8 + 2xM.2 (aliexpress /item/1005006666999472.html)
- Carte Riser PCIe X16 to 4xM.2 (aliexpress /item/1005006651677875.html)
- TH3P4G3 PCI bifucation support ?
- ASM2464PD PCI bifucation support ?

Untested (yet) but surely better afterway:

- TZT Graphics Card Dock External GPU Dock (aliexpress /item/1005005472156508.html) - Why :
    + Support power delivery up to 60w (if i'm understand corretly)
    + Free the 24-Pin ATX connector so i can plug my ATX-PS01 or directly a power switch.
- Car charger connected with CPU 12V (aliexpress /item/1005006149289861.html) - Why :
    + Using another USB4 M.2 NVMe Enclosure 40Gbps ASM2464PD to be least expensive
    + Free the 24-Pin ATX connector so i can plug my ATX-PS01 or directly a power switch.
- 2x10GbE Intel X550-T2 (aliexpress /item/1005006147612535.html) - Why :
    + Lower power comsummation : 8.16W instead of 14.28W (43% less)
    + PCI-E x4 instead of PCI-E x8 (limited? to x4 by Thunderbolt/USB4 connection)
    + unfortunatly more costy 144.94€ instead of 18.13€ (nearly 8 time more)
- 1x10Gb SFP+ TEG-10GECSFP - Why :
    + Lower power comsummation : 1.5W instead of 14.28W (89.5% less)
    + PCI-E x4 instead of PCI-E x8 (limited? to x4 by Thunderbolt/USB4 connection)
    + unfortunatly more costy 92.95€ instead of 18.13€ (nearly 5.13 time more)
    + only 1 port but if 2 aren't needed
    + Useful link : forums.servethehome /index.php?threads/best-low-power-sfp-cards.25824/
- 2x10Gb SFP+ Intel x710-DA2 - Why :
    + Lower power comsummation : 3.7W (Direct Attach) instead of 14.28W (74% less)
    + unfortunatly more costy 110€ (OEM & used) instead of 18.13€ (nearly 6.06 time more)
    + Useful link : forums.servethehome /index.php?threads/list-of-nics-and-their-equivalent-oem-parts.20974/
    + Useful link : reddit /r/homelab/comments/16wupb5/low_power_10gbe_nic/
    + Useful link : github /bibigon812/xl710-unlocker/tree/master
- SK hynix Gold P31 M.2 NVMe SSD - Why :
    + Lower power comsummation max : 6.3W instead of 11.09W [Kioxia Exceria Plus G2] (43.2% less)

Tested but no good for me:

- ORICO 40Gbps M.2 NVMe SSD Enclosure Adapter - Why :
    + Fan don't works well with open air and noisy
    + more expensive at the moment
- UT3G - Why :
    + Don't support power delivery the mainboard, only 15w
- M2 PCIe B+M Network Controller Card to RJ45 10GbE AQC107 - Why :
    + Unstable speed
    + seem very poorly made
    + expensive
- 6 ports SATA 6Gbps ASM1166 M.2 (aliexpress /item/1005005732198321.html) - Why :
    + don't work on 10Gbps M.2 NVMe SSD Enclosure Adapter
    + don't work on 20Gbps M.2 NVMe SSD Enclosure Adapter

Power consummation :

- For information :
    + 5x16To ST16000NM001G are in Standby_z power condition 99% of the time
        * Tuned with : /usr/bin/openSeaChest/openSeaChest_PowerControl -d /dev/sg4 --EPCfeature enable
        * Tuned with : /usr/bin/openSeaChest/openSeaChest_PowerControl -d /dev/sg4 --idle_b 15000 --idle_c 90000 --standby_z 1800000
        * Tuned with : /usr/bin/openSeaChest/openSeaChest_Configure -d /dev/sg4 --lowCurrentSpinup low # can be set to "ultra" if needed (need testing)
        * Tuned with : /usr/bin/openSeaChest/openSeaChest_Configure -d /dev/sg4 --puisFeature enable
        * Only used for seedbox, plex and backup
        * Useful link : forum.pine64 /showthread.php?tid=7193
    + 1x2To Samsung SSD 990 PRO is in idle 99% of the time (cache for HDD)
    + TH3P4G3 is power on 100% of the time
    + USB4 M.2 NVMe Enclosure 40Gbps ASM2464PD is power on 100% of the time
    + Intel i7-1280P is using powersave governor
    + powertop not in --auto-tune (SATA disk become unavailable if use)
    + From testing : QNAP QM2-4P-384 with 4x2To Kioxia Exceria Plus G2 use 8.5W in idle (on a standard motherboard)
    + Lower package C state : C2
- Max  consummation : 143.9W
- Mean consummation : 63.3W (4.809KwH / 3 days,4 hours)
- Min  consummation : 55.5W

To be improved :

- Lack of WOL due to poor designed TH3P4G3 who only power the mainboard if 5v powered or force powered:
    + Maybe the FBSL-5P connector can help but i don't even know what kind of connectique it is and how using it
- Fit all in a 1U rack 48cm dept (cf In testing part)
- Reduce the power consummation even more
- proper power on and off switch with or without power delivery :
    + Maybe one day this one will be purchasable (community.frame.work /t/atx-psu-controller-for-framework-mainboard-in-traditional-case/22352/1)
    + Maybe one day this one will get a full front panel connector (community.frame.work /t/input-connector-shim-for-power-switch-usb-2-0-and-leds/21430)

Help needed :

- Unable to reath lower than C2 power state in package
- How to connect TH3P4G3 FBSL-5P ?

For people who don’t own a framework, you may want considering others solutions:

- https://mattgadient.com/7-watts-idle-on-intel-12th-13th-gen-the-foundation-for-building-a-low-power-server-nas/
- https://nascompares.com/2024/02/23/updated-best-cpumotherboard-combo-for-your-m-itx-nas-build-ecc-pcie-gen-5-4x4-nvme-and-more/
- Minisforum MS-01 (store.minisforum /en/products/ms-01)
1 Like

Some photos of the beginning




3 Likes

In testing:

Power consummation :

- For information :
    + With 1x10Gb SFP+ TEG-10GECSFP
    + Without 2x10GbE Intel X540-T2
- Mean consummation : 53.85W (24.76KwH / 19 days,3.75 hours) (-9.45W ~ -15%)

Do you also have the bandwidth cap issue with that one?

Yes, max theoretical speed is 40Gb/s instead of 64Gb/s for PCIe4.0x4 : List of fastest USB4 ASM2464PD & TB3 JHL7440 SSD enclosures (January 2024) – Dan S. Charlton

Not that, the thing where it caps it to much less than the 40Gbit on linux (and doesn’t on windows).

Since i’m using the ASM2464PD for the 10Gb Ethernet part i don’t saturate 40Gb/s. My dmesg don’t flags it as “limited” and as you can read below lspci -vv don’t show any hint of a limit.

02:00.0 PCI bridge: ASMedia Technology Inc. Device 2463 (prog-if 00 [Normal decode])
        Subsystem: ASMedia Technology Inc. Device 2463
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 17
        Bus: primary=02, secondary=03, subordinate=04, sec-latency=0
        I/O behind bridge: [disabled] [32-bit]
        Memory behind bridge: 7c000000-7c4fffff [size=5M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Upstream Port, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 2463
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [160 v1] Latency Tolerance Reporting
                Max snoop latency: 204800ns
                Max no snoop latency: 204800ns
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=128us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [220 v1] Data Link Feature <?>
        Capabilities: [240 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [280 v1] Lane Margining at the Receiver <?>
        Kernel driver in use: pcieport

03:00.0 PCI bridge: ASMedia Technology Inc. Device 2463 (prog-if 00 [Normal decode])
        Subsystem: ASMedia Technology Inc. Device 2463
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 128
        IOMMU group: 18
        Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: [disabled] [32-bit]
        Memory behind bridge: 7c000000-7c4fffff [size=5M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00358  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 0W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet+ LinkState+
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- ARIFwd-
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 2463
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=128us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=128us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [220 v1] Data Link Feature <?>
        Capabilities: [240 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [280 v1] Lane Margining at the Receiver <?>
        Kernel driver in use: pcieport

04:00.0 Ethernet controller: Aquantia Corp. AQC100 10G Ethernet MAC controller [AQtion] (rev 02)
        Subsystem: TRENDnet AQC100 10G Ethernet MAC controller [AQtion]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 18
        Region 0: Memory at 7c400000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 7c410000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 7c000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 7c440000 [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink+ Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000200
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] Vital Product Data
                Product Name: Atlantic
                Read-only fields:
                        [PN] Part number: 3290495095
                        [EC] Engineering changes: 0
                        [FG] Unknown: 61 62 63
                        [LC] Unknown: 64 65 66
                        [MN] Manufacture ID: AFDSWEWEBSFD
                        [PG] Unknown: 49 49 49
                        [SN] Serial number: CPL5938TLKMY
                        [V0] Vendor specific: wfewfe
                        [V1] Vendor specific: fwewfe
                        [V2] Vendor specific: SDFWI
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [YA] Asset tag: 9495829
                        [V0] Vendor specific: f34ge4rsg
                        [V1] Vendor specific: ger35g5rthghgsa3
                        [Y0] System specific: bsdfvbxcz
                        [Y1] System specific: fwefewwfe
                        [RW] Read-write area: 11 byte(s) free
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [180 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: atlantic
        Kernel modules: atlantic

Maybe in least than one month, I’ll switch the TH3P4G3 and the ASM2464PD for testing the 40Gb/s limit.

Yeah never-mind you are on the intel platform. But just a 10gbit nic would not be a problem even in the bugged state.

In testing:

- TH3P4G3 PCI bifucation support => No
- 10 ports SATA 6Gbps JMB575 + ASM1166 on :
    + 10Gbps M.2 NVMe SSD Enclosure Adapter (with M.2 to PCI-E x4) => No
    + 20Gbps M.2 NVMe SSD Enclosure Adapter (with M.2 to PCI-E x4) => No
- ADT-Link R43UH 4.0 (aliexpress /item/1005006203494468.html) :
    + Useful only if PCI-E Gen 4 end device.
    + Else keep low costs M.2 to PCI-E x4 (aliexpress /item/1005004714035083.html)
- Switch TH3P4G3 <=> ASM2464PD :
    + Some stability issue but surely due to flashed firmware
    + No noticeable performance boost (can be caused by the firmware too)
    + If PCI-E Gen 4 end device => use only ASM2464PD for performance (TH3P4G3 limited to PCI-E Gen 3)

On Hold :

- Need ASM2464PDX Device :
    + Carte Riser PCIe X16 to X8 + 2xM.2 (aliexpress /item/1005006666999472.html)
    + Carte Riser PCIe X16 to 4xM.2 (aliexpress /item/1005006651677875.html)

Power consummation :

- For information :
    + With 1x10Gb SFP+ TEG-10GECSFP
    + Without 2x10GbE Intel X540-T2
    + Without 500Go Samsung SSD 860 EVO
    + 1x2To Samsung SSD 990 PRO is used as system drive and HDD cache
- Mean consummation : 52.21W (4.823KwH / 3 days, 20.5 hours)
So "500Go Samsung SSD 860 EVO" power consummation was around 1.5W on average.
1 Like

ASM2464PDX can create some miraculous board, but unfortunately I don’t see any creator using it.
If anyone knows a device like it please tell me

Why does that need the PDX an then use a pcie switch anyway? Wasn’t the whole point of the pdx bifurcation?

To duplicate a PCI-E device, i.e. without:

Ohhh direct pcie to pcie copy on that thing, that’s neat.

But the one before that didn’t need the pdx, the pd could do that allready