USB passthrough rabbit hole

[retpolanne]

Description

QEMU usb passthrough rabbit hole

Okay, so I went down this rabbit hole to see how doable it is to implement USB passthrough.

Let's go step by step:

1. Lima talks to QEMU through QMP (QEMU Machine Protocol). It uses a lib made by DigitalOcean, go-qemu, which has a function DeviceAdd [1] that seems to be autogenerated. From the qemu qmp ref docs [2] we can see that it accepts device, bus and id. However, according to [3], it should accept other kinds of args as well. In our case, device would be usb-host, and then we would require vendorid and productid.

2. Trying on the qemu monitor, I see this error:

(qemu) device_add usb-host,vendorid=0x1235,productid=0x8211
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]

4. The whole bus could be connected, according to [4]

5. From my tests, if the device_add syntax is wrong or device is inexistent, it freezes the whole instance, seems to be a qemu bug.

More info at [5] and [6]

References

[1] https://pkg.go.dev/github.com/digitalocean/go-qemu@v0.0.0-20230711162256-2e3d0186973e/qmp/raw#Monitor.DeviceAdd

[2] https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2506

[3] https://qemu-project.gitlab.io/qemu/system/devices/usb.html

[4] #1317

[5] https://gitlab.com/qemu-project/qemu/-/issues/1951

[6] libusb/libusb#908

[retpolanne]

I wonder if libusb_get_device_descriptor is not returning, so qemu gets stuck.

[ 0.074566] [002c9212] libusb: debug [libusb_get_device_descriptor]  
0   libusb-1.0.0.dylib                  0x0000000102884cf8 libusb_get_device_descriptor + 88
1   qemu-system-arm                     0x0000000100ab3268 usb_host_auto_check + 200
2   qemu-system-arm                     0x0000000100ab074c usb_host_realize + 536
3   qemu-system-arm                     0x00000001009c4b48 usb_qdev_realize + 244
4   qemu-system-arm                     0x0000000100cb40d8 device_set_realized + 384
5   qemu-system-arm                     0x0000000100cbc28c property_set_bool + 100
6   qemu-system-arm                     0x0000000100cba1f8 object_property_set + 136
7   qemu-system-arm                     0x0000000100cbe5ec object_property_set_qobject + 60
8   qemu-system-arm                     0x0000000100cba6e0 object_property_set_bool + 60
9   qemu-system-arm                     0x0000000100a41b6c qdev_device_add_from_qdict + 2304
10  qemu-system-arm                     0x0000000100a42400 qmp_device_add + 104
11  qemu-system-arm                     0x0000000100e3aea4 do_qmp_dispatch_bh + 56
12  qemu-system-arm                     0x0000000100e5a1fc aio_bh_poll + 192
13  qemu-system-arm                     0x0000000100e45230 aio_dispatch + 40
14  qemu-system-arm                     0x0000000100e5afb8 aio_ctx_dispatch + 16
15  libglib-2.0.0.dylib                 0x000000010351db30 g_main_context_dispatch_unlocked + 236
16  libglib-2.0.0.dylib                 0x000000010351da34 g_main_context_dispatch + 44
17  qemu-system-arm                     0x0000000100e5b6bc main_loop_wait + 412
18  qemu-system-arm                     0x0000000100a477a8 qemu_main_loop + 104
19  qemu-system-arm                     0x0000000100cb0484 qemu_default_main + 16
20  dyld                                0x0000000186e750e0 start + 2360

This happened after I called:

╰→ telnet localhost 4444
Trying ::1...
Connected to localhost.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 8}, "package": "v8.2.0-1767-g91e3bf2e92-dirty"}, "capabilities": ["oob"]}}
{ "execute": "qmp_capabilities" }
{"return": {}}
{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x1235", "productid": "0x8211"} }

In other words

1. if we change go-qemu's raw monitor DeviceAdd call to receive any kind of args, then they'll be passed to QEMU and the device_add call will work!
2. since qemu + libusb are broken, then it fails, but that is due to qemu freezing.

In fact, I see errors from usb after I kill qemu!

{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x13fe", "productid": "0x3e00ffff"} }

{"timestamp": {"seconds": 1708801281, "microseconds": 231261}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-signal"}}
{"error": {"class": "GenericError", "desc": "productid out of range"}}
Connection closed by foreign host.

[retpolanne]

In the end, it seems to come down to this [1],

This is the correct payload that QEMU supports

{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x13fe", "productid": "0x3e00", "id": "usb-disk", "hostbus": "1", "hostaddr": "26"} }

in order to actually connect the USB device, we need to unload the kexts that talk to the device. If that doesn't happen, libusb_kernel_driver_active will fail.

[1] https://github.com/libusb/libusb/wiki/FAQ#how-can-i-run-libusb-applications-under-mac-os-x-if-there-is-already-a-kernel-extension-installed-for-the-device-and-claim-exclusive-access

[paschun]

fyi, podman 4.x has support for this on macos, which I think comes down to a couple qemu flags. From podman-machine-default.json:

  "-device",
  "qemu-xhci",
  "-device",
  "usb-host,vendorid=1234,productid=54321",

And looks like lima already uses the flag -device qemu-xhci,id=usb-bus

[AkihiroSuda]

fyi, podman 4.x has support for this on macos

Does it need the root on the host?

[savvn001]

Following. Would love to see this implemented 😃

[mschoenlaub]

I got an implementation working, based on https://github.com/SPICEorg/usbredir.

It does run the usbredirect daemon as root, one per device and this kinda works.

I just wanted to know about the expections for the config file structure.
I'm going to assume that:

1. You want to redirect/pass through one port/device to at maximum one running instance.
2. You want only max one instance running with some flag like --auto-add-usb.

How would you expect to configure this in the YAML structure?

What are the expections if you try to start an instance where the usb port/device is already in use by another instance?

Note: It does not use SPICE, but rather qemu's usb-redir function. A similar approach could be used with WSL.

Personally I feel like the implementation would be "driver specific", with VZ not offering any support currently.

[mschoenlaub]

Puh... working on a PR, but I'd need some guidance around where want to go with the "daemons". It feels like there's a ton of code tied into the networks package, that could be very well used for generic daemons.
But then, there's lots of slices only containg SocketVMNet related values. Mostly because the VDE stuff was removed.

i'm mainly writing this, because if there's another PR working in this area, rather large conflicts in code with low test coverage will arise.