USB-C/HDMI/DP/Thunderbolt software-based display passthrough

Framework Laptop 16
1

So this is a fairly straightforward idea. Something like certain Macs’ Display Target Mode is a hardware feature. What I want to do is take display input (via USB-C/Thunderbolt, HDMI, DisplayPort, or whatever else is feasible) and put it on the laptop screen with as little delay as possible (so obviously, using OBS on top of some Linux distro is not it).

The fastest implementation I can think of is somehow piping the device through to a framebuffer in some way, but it surely can’t be that easy?

If this means I get to create a distribution solely for media routing, then so be it.

1 Like
2

I’ve played with a couple of different generations of software that attempted this:

DisplayLink which was like DisplayPort but optimized for slower USB2/early USB3 speeds. Unmitigated disaster, stay far away.

Scrcpy . Though this was specifically for mirroring an Android phone to a Linux desktop. Pain in the you know what to set up / restart a session, but once it was going it was amazingly good for as little bandwidth as it used.

Miracast. Haven’t tried the open source Linux tools for this in some time, but it was maybe a 50/50 chance of a session staying.

i haven’t used it, but I imagine Shairport has something like that you are taking about.

Of course there’s thousands of remote desktop solutions. RDP, Spice, VNC, x2Go, etc etc etc etc. Most of those are “pull” from a source to your client, not “push” a display into an input or onto a different client AFAIK, though.

3

So my goal is to use my Framework 16 as effectively a KVM for whatever computer I hook it up to (Raspberry Pi, FW13 enclosure, who knows). Failing that, simply a monitor for whatever I hook it up to (I have a good external keyboard; that’s not a problem).

My thought was that by far the most performant option is going to be writing a program from scratch, ran directly from the BIOS (UEFI?). Effectively precisely what we want (a display) and nothing we don’t (an OS).

How feasible is this really?

4

USB/HDMI/DP/Thunderbolt are all hardware; you say you want software-based passthrough. Can you elaborate?

There are plenty of options for video in (personally I have an Elgato CamLink 1080p HDMI → USB-A 3.0) which can work with OBS, but can also work with other software (it acts as a usb webcam, so any video chat software {eg. Zoom, Google Meet, Discord, etc} sees it as a camera). I know Meta recently released Meta Quest HDMI Link for Quest 2, 3, and Pro headsets allowing HDMI (and DP?) input into a virtual 2D display which also requires a “UVC (USB Video Class) compatible USB Capture cards which support 1080p capture and output resolution. USB 3.0 cards are recommended.”

Are you saying you want some software that turns the existing Framework dongles/adapters into display inputs? Or you have a USB capture card already and are looking for software to simply interface/display it?

5

Effectively a software replacement for a capture card. I want to use expansion cards for display inputs without needing any extra hardware.

6

That’s not a thing you can do. This is sort of like having a smoothie and trying to use a blender to get whole fruit out of it.

The video output ports on a Framework laptops (primarily through expansion ports) can only output video due to the circuitry in the laptop. Entirely different circuitry is required to input a video signal, encode it as USB data, and pass it into the laptop where it can be processed again (from USB traffic) back into a video stream.

If you want a portable monitor, there are plenty of options that are USB (or battery powered).

If you want to use a laptop as a portable display, the only (realistic) option is to use one that has video input. Generally, this is added via a capture card, but some laptops do offer it. In the case of Framework, you could try to fit a capture card in to an expansion card form factor. Or stream it over a network/wirelessly.

1 Like
7

So I can’t even take USB-C video input? Say I connected a FW13 mainboard to my FW16 laptop with a proper Thunderbolt cable. Dunno how power would work out (we’ll get there when we get there) but there’s absolutely no way I can take USB-C video input?

I’m fine with buying a capture card if I have to but if I can do it with software, I’d rather not blow upwards of $100 USD.

8

Again, you seem to be using a bunch of buzzwords.

Framework 16 is currently only available with AMD APUs/CPUs. Thunderbolt is an Intel/Apple co-developed and licensed technology/certification. While AMD laptops can get certified to use/support/offer Thunderbolt, in order to do so, they have to give money to Intel (one of their prime competitors) for Intel to thoroughly test and certify that AMD’s product meets Intel’s standards. It happens very rarely. As of the time of writing, Framework 16 is not* Thunderbolt certified.

Thunderbolt 3 uses a USB-C connector, but is not developed or maintained by the USB-IF (the group that does maintain and stupidly name USB standards).

Intel supports USB data traffic on Thunderbolt connections, as well as power passthrough, IP traffic, PCIe passthrough, and video signals.

USB 3.2 Gen2(x2) supports USB data traffic, power passthrough, and video signals.

The biggest difference is Intel requires that all Thunderbolt devices support all of the things, while USB says that pretty much everything is optional. It doesn’t have to support charging to be USB, it doesn’t have to support data to be USB (USB charging cables), it doesn’t have to support video, etc. All of this is on display with the Framework 16. Some ports don’t support video output, some don’t support charging, and if you use the port on the back of the dGPU, while it does support data, it’s limited to USB 2.0 speeds (even though the DP/HDMI signals being passed though it are likely 20-30Gbps bandwidth).

But as previously mentioned, certifying every laptop to support Thunderbolt is expensive, so there weren’t as many devices getting certified as Intel would like. Fewer certified laptops means fewer potential purchasers of Thunderbolt products like docks, screens, E-GPUs, or storage products (and Intel gets royalties from each product sold). So Intel cracked open Thunderbolt just a little and told the USB-IF that they could have/use Thunderbolt 3 for free right around the time that they were introducing Thunderbolt 4.

USB-IF then introduced USB4, which is mostly just a rebadge of Thunderbolt 3! But they did it the way they’ve introduced everything; by making nearly everything about Thunderbolt optional. (*So Framework 16 is NOT Thunderbolt certified, but some/many Thunderbolt 3 products are likely to work in the 2 ports closest to the screen.)

But this long, and probably boring story finally comes back to what you asked.

You can’t receive the video signals being output from an output device (like a Thunderbolt/USB4 enabled laptop) into another output device (like the video output on another laptop). While many things with Thunderbolt and USB-C are bidirectional, like USB data speeds or IP traffic, many others aren’t. Thunderbolt requires laptops are able to receive 100W (or up to 240W) to charge, they’re only required to output 15W. While they’re required to support HDMI/DP video content output, they aren’t required (and likely can’t) receive it.

That’s where that capture device comes in. While you can’t send the video signal back in to an output, a capture device is an input. Once the capture device receives the video stream data, it can encode it into a protocol/data stream that Thunderbolt/USB is capable of receiving; usually USB data traffic that gets treated as a camera, because that’s usually the only incoming video stream data that devices are made to expect.

There’s no technical reason I can think of that would prevent someone from sending the data over another supported bidirectional data type, like IP with Thunderbolt (many professional video production businesses send live video over dedicated, high-speed, but otherwise standard networks), but then there’s additional decoding overhead that I suspect would add latency.

As for USB-C, it’s just a connector and sadly does not enforce or require any specific capabilities (other than hopefully not being intentionally malicious). It can be used as a low-end 5v-only charging connector with no data connection, USB 2.0 data device (yay 480Mbps!), USB 3.0/3.1 Gen 1/3.2 Gen1x1/SuperSpeed at 5Gbps, USB 3.2 Gen 2x1 (SuperSpeed+)/USB 3.2 Gen 1×2(SuperSpeed) at 10Gbps, USB 3.2 Gen 2×2 (SuperSpeed+)/USB4 Gen 2x2 at 20Gbps, USB4 Gen3x2 at 40Gbps, video over DP alt-mode, video over HDMI alt-mode, support USB-PD at up to 60W/100W/240W, or PCIe passthrough/tunneling.

As for powering a Framework 13 mobo… I don’t know the minimum power draw requirements for the 13 (the 16 “requires” a 60W USB-PD charger minimum per Framework docs), but if the battery were connected, sort of as a built-in UPS, and you weren’t running a high draw compute load 24/7, you’d likely be able to charge it slowly. That being said, I’ve skimmed the thread of folks complaining about low USB power output on the Framework 16, and think Framework is likely following the USB standard’s minimum (7.5W) instead of the Thunderbolt standard’s minimum (15W). So a very slow charge.

1 Like
9

Well, this response was incredibly helpful and far from boring (and really just reinforces my preconceived notions that Thunderbolt as a spec was a silly idea: the short version is that standards should exist in one layer and one layer only, be it physical, transport, data, whatever. It doesn’t even have to be OIS layers, just that it’s one layer.)

So if I were so inclined, I could create an expansion card that acts as a halfway-decent capture card such that power delivery from a “master” framework (say a full FW16) to a “slave” system (say a FW13 mobo) and video signal from the slave to the master both happen? The idea being, one Thunderbolt cable to rule them all. I’d effectively be daisy-chaining power (outlet to master via power brick and USB-C, master to slave via magic card and Thunderbolt 4 cable) and video from slave over T4 to card to master, where it can then be processed at will.

In a perfect world, that One Cable would also carry keyboard/pointing device signals (effectively turning the master system into a KVM for itself and slave systems). Is this possible over Thunderbolt 4?

I have ideas, but ideas aren’t much use without real-world implementation, and implementation is pretty tough if it’s not feasible (kinda by definition).

10

There is a PiKVM project that does something like this for the raspberry pi, forwarding the KVM over IP. You could probably attach similar hardware to the framework 16, and hook it up to a framework 13 board over usb. The software that is used is open source and should run fine on the framework 16 on one of the supported Linux distro’s.

See here:

The hardware involved is quite substantial however, I don’t see a way to make this work in a expansion card or cable, but it would probably fit fine in a small external case.