mirror of
https://github.com/games-on-whales/wolf.git
synced 2026-10-15 21:12:08 +00:00
[GH-ISSUE #73] Running Wolf inside an LXC container #50
Labels
No labels
🧠 1 - 🤖 999
AMD
bug
bug
cannot reproduce
documentation
duplicate
easy
enhancement
enhancement
enhancement
fixed?
fixed?
good first issue
hard
help wanted
Intel
invalid
LXC
medium
moonlight-protocol
Multi GPU
nixos
not a bug
nvidia
pull-request
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
games-on-whales/wolf#50
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jefflewis on GitHub (Jun 4, 2024).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/73
I'm attempting to set up a wolf deployment on an LXC container in Proxmox. I have the v550 Nvidia drivers installed and all the required devices configured on the container. The system does run, but will often disconnect the moonlight client with either a termination error or a UDP port error. When the launch does succeed, I can install games and the steam app UI is very responsive. Sound and gamepad controls work just as expected.
When I launch a game, I only get a black screen. The GPU lists wolf as a process, and when I set WOLF_RENDER_NODE to use
/dev/dri/renderD129, it lists many other processes as well. (There is an integrated intel GPU available as well).In the container logs, I see many errors around inputs. Could this be part of the flakiness of the connection? I see no errors logged when the screen goes black. What can I do to further troubleshoot what is wrong with my setup?
docker-compose
lxc conf
nvidia-smi
@ABeltramo commented on GitHub (Jun 5, 2024):
Thanks for the detailed issue, running a Docker container inside an LXC container might get tricky, but I'll try to help where I can.
Wolf will definitely log something here, someone on Discord reported random warnings about binding the RTP port, could that be your case too?
Bear in mind that the very first time you launch a game that is expected since Docker is actually downloading the image (and we don't have an easy way to report that progress, yet). The stream should stay open with a black screen until the app is actually running.
From the few log lines that you've posted it seems that our custom compositor is not able to access the virtual devices (mouse and keyboard) that Wolf creates. I've noticed that you have a bunch of
lxc.mount.entryin your config; our virtual mouse and keyboard will have major13so maybe you should add something likeGenerally setting
WOLF_LOG_LEVEL=DEBUGis helpful in troubleshooting issues. You can post a full log run here, and I'll try to help you!@jefflewis commented on GitHub (Jun 5, 2024):
Thanks so much for the help! I know this is one of the more complex ways to run this. My goal is to be able to share this GPU with other services for when it's not used for game streaming (ML, etc).
Adding the extra conf for control group 13 seemed to let me actually start up into a game further than I had before! I still see the machine drop its connection and not be able to resume. When I've seen it launch to a black screen, it seems more than a "loading" state, as I've let it sit for 30 minutes and it just stays. I
I looked at the host machine's
/dev/inputdirectory, and it is in fact missing any event > 16, which are all events that fail in the wolf logs. I don't know what those events correlate to or why they are missing on the host.Here's an example launch session log with the connection being terminated before steam ever launches:
Here's a session log of when I receive a UDP port error on launch:
@ABeltramo commented on GitHub (Jun 5, 2024):
I think there are two different issues here:
UDP bind error
I've just pushed an update that should fix this. Make sure to update the Wolf Docker image by downloading the latest with:
Virtual devices
There's something in that LXC container that is blocking creating and/or accessing the virtual devices that Wolf needs to create in order to relay the remote inputs.
If they are missing from the host it probably means that Wolf is not able to access
/dev/uinput(notice theu): normally when you start a session from Moonlight, Wolf will create the necessary virtual devices (mouse, keyboard, joypad, ...) and they'll be visible under/dev/inputbut in your case you get the error message:Now one thing I noticed is that you've missed the LXC mount for
/run/udev, which is required, but I don't think that's going to fix this. We have to figure out whyuinputdoesn't work inside your container..Can I suggest trying out with a privileged LXC container first and then, if that works, trying to drop the privileges?
@jefflewis commented on GitHub (Jun 5, 2024):
The latest stable container image seems to have cleared up the UDP errors, thanks!
I am running a privileged container (I planned to try it unprivileged once I got a known container working 😅)
I may have mispoken for the "host". By that I meant physical Proxmox install is missing those inputs. Without them in the root Debian install, they are naturally missing from being passed into the container.
For missing
/dev/uinput... that is odd. That file is mounted in the container config and has the appropriate major allow group. I'll note that when running on an android tablet with a gamepad, the gamepad controls do control the steam library UI. I've not yet gotten any game past the launch screens, but it feels so close.I'd love to be able to add a setup wiki page for running wolf on proxmox, as if this can get to be working well feels very slick. I really love all that you've done with
wolfandgow❤️@jefflewis commented on GitHub (Jun 5, 2024):
Here's a log with the updated container and mounted
/run/udevdirectory. I started the container and immediately attempted to run with a connected moonlight client. The connection started and then terminated with code -1 on the client display, and with these associated server logs.@jefflewis commented on GitHub (Jun 6, 2024):
After a few config tweaks and lots of reboots, I was able to get multiple clients running and playing games. I had one playing Cyberpunk 2077 at near 60fps 😃. The original container config I had was woefully underpowered, so I suspect some of my issues were from the system trying to do too much. I gave it 26GB of RAM and 12 CPU cores and things seemed to start to be more reliable.
I've still seen lots of immediate disconnects when starting up the app for the first time. But continuing to retry does seem to eventually boot into a valid session. I'll get a good log file showing this soon.
I also haven't seen my keyboard and mouse work when connecting from a PC client (vs the mobile devices with gamepads). Could input events 17+ be those perhaps?
Thanks again for your help and hopefully this can be used to help others get a similar setup running.
@ABeltramo commented on GitHub (Jun 8, 2024):
If you could gather those I'd be happy to look into it!
That is very odd.. If gamepads works that means that Wolf is indeed able to create those devices using
uinput. They'll too will appear as additional/dev/input/event**and/dev/input/js**in your host, can you check if they are present?Btw, have you added the
/run/udevmount in the LXC container like I've mentioned before?@jefflewis commented on GitHub (Jun 10, 2024):
It's hard for me to get a good log set for these... I'll get that soon, but after more use, when this errors is when the app is quit and then restarted. Connections on restart can take quite a few connections to get started. I've not seen mid-session dropouts since boosting the container's resource specs.
Agreed! Here's the listing for
/devand/dev/inputfor the proxmox host and lxc containerIn the proxmox host (pve1):
In the lxc-container running the docker container (dockge):
Yep!
Here's the full lxc conf being used:
And here is the lxc container's
/rundirectory listing:@ABeltramo commented on GitHub (Jun 11, 2024):
I can see that the list of devices under
/dev/inputis the same in the host and in the LXC container but are thosemouse0andmouse1the virtual devices from Wolf? Are you checking this whilst running a stream (or at least with an app paused)? Is the same list available inside theWolfcontainer?(btw it might be worth continuing this on Discord so that we can go back and forth faster, if you'd like)
@jefflewis commented on GitHub (Jun 12, 2024):
Yea! I'll start a message in the moonlight discord #games-on-whales channel. Thanks again for all of your help!
@vinibutturi commented on GitHub (Jun 25, 2024):
Hi guys, I also managed to create a Privileged LXC on Proxmox running Wolf.
This is how I did:
On PVE Host
nvidia-smi,ls -l /dev/nividia*the output should look like the below images:nvidia-smils -l /dev/nividia*If you are missing some devices when listing
ls -l /dev/nividia*, you can follow the first part of this guide "Making Sure the NVIDIA GPU Kernel Modules Are Loaded in Proxmox VE 8 Automatically"cat /sys/module/nvidia_drm/parameters/modeset, the expected result is YIf you got an "N", you can enable it by changing the grub config
nano /etc/default/gruband adding the "nvidia-drm.modeset=1" to theGRUB_CMDLINE_LINUX_DEFAULT line
Then
update-grubandreboot.nano /etc/udev/rules.d/85-wolf-virtual-inputs.rulesand paste the below content (extracted from Wolf's quickstart )Create your LXC container with docker, I've used tteck's script to create a privileged docker LXC with Debian 12, make sure you select Y when asked to install docker-compose.
Edit you LXC config file
nano /etc/pve/lxc/108.confand remove the following lines (They are for usb passthrough but make no sense to keep them as we'll pass /dev/input to the LXC)This is what it should look like in the end
We are done working on the PVE host, start (or restart your LXC).
On LXC
curl https://raw.githubusercontent.com/games-on-whales/gow/master/images/nvidia-driver/Dockerfile | docker build -t gow/nvidia-driver:latest -f - --build-arg NV_VERSION=$(cat /sys/module/nvidia/version) .and then
docker create --rm --mount source=nvidia-driver-vol,destination=/usr/nvidia gow/nvidia-driver:latest shnano docker-compose.ymland paste the followingpay attention to the highlighted lines, as they differ from the one on the quickstart
now a
docker compose upshould be all that's left for you to enjoy Wolf 😁Next step is trying to do the same but inside an Unprivileged LXC 🤪
@jefflewis commented on GitHub (Jul 14, 2024):
@vinibutturi Were you also able to get the mouse and keyboard input working? I was able to run this fine with only joypad support, but anything requiring the other inputs was failing.
@vinibutturi commented on GitHub (Jul 14, 2024):
I mainly use Retroarch, so I cannot say about playing games with mouse and keyboard, but they do work on the Retroarch menu. So I would say, yes.
Maybe you're running into permission issues, i have set my LXC to allow acess to all shared devices by adding the below to mey LXC config file:
lxc.cgroup2.devices.allow: alxc.cap.drop:with this you don't need to add individually permissions to the devices by using major and minor ids, etc...
If you still can't figure it out, shout out on the discord group. There's me a another member toying with LXC there and he also had similar behaviour (joypad or but no keyboard/mouse) and we'll try to help you.
@Cronvs commented on GitHub (Jul 19, 2025):
Why does it need write access to udev, and input, isn't write to uinput enough?
Also, isn't giving specific devices in /dev/dri enough?
@ABeltramo commented on GitHub (Jul 19, 2025):
I should write this better in the docs, but here's the rundown:
/dev/uinputand/dev/input/access/run/udev/dev/uhidwhich will end up creating/dev/input/event*devices and/dev/hidraw*at runtime.If you are interested in the low level details of how all this works under the hood I've written a bunch of blog posts: Faking udev for containerized apps, When uinput Isn’t Enough: Virtualizing a DualSense controller, Creating a Virtual DualSense Controller via UHID and Beyond USB: Improving Virtual Controller Support in Linux Games