mirror of
https://github.com/games-on-whales/wolf.git
synced 2026-10-15 21:12:08 +00:00
[GH-ISSUE #233] Gstreamer Nvidia encoders are not used when multiple Nvidia GPUs are present #157
Labels
No labels
🧠 1 - 🤖 999
AMD
bug
bug
cannot reproduce
documentation
duplicate
easy
enhancement
enhancement
enhancement
fixed?
fixed?
good first issue
hard
help wanted
Intel
invalid
LXC
medium
moonlight-protocol
Multi GPU
nixos
not a bug
nvidia
pull-request
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
games-on-whales/wolf#157
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zexi on GitHub (Jul 28, 2025).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/233
In an environment with 8 Nvidia GPUs, I can successfully utilize a specific GPU for rendering by passing
-e WOLF_RENDER_NODE=/dev/dri/renderD129and--device /dev/nvidia1parameters via the script below.However, Gstreamer's output stream always uses software encoding instead of hardware nvcodec hardware encoding. What troubleshooting steps should I take to enable nvcodec hardware encoding?
Logs
@ABeltramo commented on GitHub (Jul 28, 2025):
First off, I recognise that we don't have great support for multiple GPUs of the same vendor. It's something that I'd like to improve in the near future, but I have a few other important PRs in the work right now..
The specific issue in your case is most likely above:
you might be able to get more info by lowering the Gstreamer log level with the env variable
GST_DEBUG=5which will be much more verbose, if you could post the logs again I might be able to help you better.The first thing to check here is if the Nvidia driver volume is correctly setup and it matches the host driver version. My suggestion is to start from scratch:
Assuming that wasn't the issue, if I have to take a guess, I'd say that you aren't passing some specific Nvidia device like
/dev/nvidiaX.Once you'll fix that and Gstreamer can use CUDA to instantiate at least one HW encoder plugin, then you'll quickly have the issue of selecting which specific GPU to use for encoding. There are multiple ways to do that, the easiest is probably to edit the
config.tomlto use a specificnv{encoder}device{index}encwhereindexis the cuda device id (you should be able to get that easily vianvidia-smi) .For
HEVC(the most common encoder in Moonlight) you'll have to edit this line replacingnvh265encwithnvh265device1encfor the device with ID 1.Ideally, that part will be automatically managed by Wolf to match the selected
WOLF_RENDER_NODE@zexi commented on GitHub (Jul 29, 2025):
@ABeltramo Thanks for your response. I followed your guidance to perform troubleshooting, but the issue persists.
1. The Gstreamer log level with the env variable GST_DEBUG=5:
debug.log
2. The Nvidia driver volume:
The nvidia driver volume is the right version matching host version.
3. Passing the right /dev/nvidiaX
I first located the PCI address of renderD129 as 0000:41:00.0 from the
/dev/dri/by-path/directory.Then, by examining the file
cat /proc/driver/nvidia/gpus/0000:41:00.0/information, I found that the Device Minor is 1.Does this confirm that passing /dev/nvidia1 is correct? However, the nvidia-smi command shows the index of the GPU at 41:00.0 as 2.
Below is the output of the relevant commands:
Selecting specific GPU to use for encoding
After I modified this line to nvh265device1enc, software encoding was still being used.
Are there any other details I should provide? This machine is equipped with 8 GPUs, and I'm available to collaborate on testing multi-GPU scenarios.
@ABeltramo commented on GitHub (Jul 29, 2025):
Thanks for the logs, here are the relevant bits:
Good, Gstreamer can see your GPU, but then you get a lot of errors like:
Here's an example output from my dev machine with a 3070 plugged:
See the differences? It can correctly instantiate and use the HW encoders, ex:
Following the Gstreamer source code it seems that you are stuck in this call:
In your case it's returning
100which meansCUDA_ERROR_NO_DEVICE, from the docs:Not sure what's causing that on your machine, but I suppose you've got something wrong with the Nvidia drivers. I recommend removing them and re-install from the official sources https://www.nvidia.com/en-us/drivers/unix/ then reboot and re-create the Wolf Nvidia driver volume as explained in the previous comment.
FYI the latest "stable" version of the driver is currently 570.172.08 whilst you have
570.133.07.Until we can fix this, the rest of the points outlined don't really apply: there's no capable HW encoder found.
@DatCaptainHorse commented on GitHub (Jul 29, 2025):
@zexi Make sure your host system drivers are not "server" drivers, they don't come with encoding/decoding features for some reason. I.E. some Debian/Ubuntu systems default to
nvidia-driver-xxx-serverwhich will not work, whilenvidia-driver-xxxwould.@zexi commented on GitHub (Jul 30, 2025):
@DatCaptainHorse Thanks for your hint. I used the
NVIDIA-Linux-x86_64-*.rundriver, so the encoding/decoding features is working.@zexi commented on GitHub (Jul 30, 2025):
@ABeltramo I installed the latest 570.172.08 nvidia driver, but the same issue and error persist.
However, I discovered a solution: by mapping all
/dev/nvidia*devices into the container via the--deviceparameter, GStreamer can then recognize the nvcodec hardware encoder.Then exec into the wolf container and runs
gst-inspect-1.0 nvcodeccommand, I can see the features:With this, I can modify the /etc/wolf/cfg/config.toml file to specify which GPU to use for encoding by setting
nvh265device*enc.But a new issue arises: if I set the
WOLF_RENDER_NODE=/dev/dri/renderD130environment variable, the ZERO COPY feature throws the following error, whereas specifying /dev/dri/renderD128 works without issues:This is the log file with
GST_DEBUG=5:debug-zero-cp.log.zip
If I disable ZERO COPY feautre by setting
WOLF_USE_ZERO_COPY=FALSEand use the legacy streaming method, the error disappears, and the system runs smoothly when using any specified /dev/dri/renderDxxx device.Is it possible that there's a bug in the ZERO COPY implementation, causing it to work only with renderD128?
@DatCaptainHorse commented on GitHub (Jul 30, 2025):
@zexi zero-copy requires that both rendering and encoding are done on same GPU, so if you try set different GPU for
WOLF_RENDER_NODEthan what is used for encoding, it will error out.@zexi commented on GitHub (Jul 31, 2025):
@DatCaptainHorse When setting WOLF_RENDER_NODE=/dev/dri/renderD128, I can successfully specify different GPUs for streaming by configuring the nvh265device*enc parameter in /etc/wolf/cfg/config.toml by this line.
For example, with nvh265device1enc, the nvidia-smi command confirms the streaming process runs on GPU1 while other processes reside on GPU3.
However, if I set WOLF_RENDER_NODE=/dev/dri/renderD130 (or any non-renderD128 device), the system throws an error described in the previous comment. This inconsistency leads me to suspect that the zero-copy functionality may only operate correctly with the renderD128 device.
@Gylfkxjyjdll commented on GitHub (Jul 31, 2025):
i had the same issue back then.
my solution was:
purging all nvidia/cuda driver and reinstalling everything. reinstalled nvidia with the open-source kernel module (very important), the cuda driver and nvidia-docker2. didnt had any issues with the driver 570.172.08 and 570.133.07. Atm i use 575.64.05
checked if all kernel parameters where loaded right (had once an issue here after restarting, dont know why)
deleting all gow related stuff
checked all read and write permissions
adding WOLF_RENDER_NODE=/dev/dri/renderD129 and WOLF_ENCODER_NODE=/dev/dri/renderD129 as env parameter (otherwise i had the same issue as you)
add devices = [ '/dev/nvidia0:/dev/nvidia0:mrw', '/dev/nvidia1:/dev/nvidia1:mrw', '/dev/nvidia2:/dev/nvidia2:mrw', '/dev/nvidia3:/dev/nvidia3:mrw' ] to the config.toml
for debugging, i changed "Privileged": true" in the config.toml
here you can read through my debugging process.
https://discord.com/channels/856434175455133727/1368484261937676341
@zexi commented on GitHub (Aug 1, 2025):
@Gylfkxjyjdll Thank you for sharing the solution. I think the most critical step is passing all
/dev/nvidia*devices into the container.Additionally, have you tested the latest
ghcr.io/games-on-whales/wolf:stableimage? This version enables zero-copy feature by default, but when I specify /dev/dri/renderD129 in a multi-GPU environment, it throws errors. Have you encountered the same issue?@Gylfkxjyjdll commented on GitHub (Aug 2, 2025):
add the conatainer in the config.toml?
i tried the latest stable release. everything workes as expected.
@ABeltramo commented on GitHub (Aug 3, 2025):
@zexi It's one thing to pass the nvidia devices to the Wolf container and another one to actually share them with the running app containers (Steam, Firefox, ...). Have you added the additional
/dev/nvidiamounts in each app that requires them inconfig.tomllike @Gylfkxjyjdll showed?@zexi commented on GitHub (Aug 4, 2025):
@Gylfkxjyjdll @ABeltramo I've added the additional
/dev/nvidia*devices in the config.toml but still got the same issue.The output of wolf service:
The device parts of
docker inspect WolfSteam_xxxoutput:@zexi commented on GitHub (Sep 28, 2025):
@DatCaptainHorse @ABeltramo After forking the gst-wayland-display and adding some debugging code (link: https://github.com/zexi/gst-wayland-display/blob/test-stable/wayland-display-core/src/utils/device/gpu.rs#L48-L74), I noticed that the
enumerate_gpu_devicesfunction retrieves drm_path values like/dev/dri/card%din my NVIDIA GPU environment.These paths do not match the device I specified via the environment variable
WOLF_RENDER_NODE=/dev/dri/renderD129. Is this a potential issue?Log output is as follows and the
Error during GetRenderDevice: No GPU device for given DRM nodemessage is at last line:@ABeltramo commented on GitHub (Sep 29, 2025):
Nope, ultimately we just use that code to add a workaround specific only for some Intel GPUs here, but it wouldn't do anything if we hit that warning.
Lots of users report that warning, so there's definitely something wrong with the logic, and I'll fix it, but rest assured that it wouldn't cause any kind of issue with a Nvidia GPU.
@zexi commented on GitHub (Oct 20, 2025):
@ABeltramo For reference only, I guess the problem with the additional
/dev/nvidia*devices is related to the issue: https://github.com/NVIDIA/nvidia-container-toolkit/issues/1249, which forces the requirement to pass all GPUs for Gstreamer to recognize the nvcodec hardware encoder.@ABeltramo commented on GitHub (Oct 30, 2025):
This should be fixed in the latest
ghcr.io/games-on-whales/wolf:wolf-uiwe should now correctly pick up the right GPU for both encoding and playing games in a multi-GPU setup with theWOLF_RENDER_NODEenv variable.Can you please try it out and let me know if it works for you?
@zexi commented on GitHub (Oct 31, 2025):
@ABeltramo Thanks for your reply. The following error occurred while testing
ghcr.io/games-on-whales/wolf:wolf-ui:Logs
@ABeltramo commented on GitHub (Oct 31, 2025):
Thanks for testing the latest, from the logs you've posted it seems that the call to cuGraphicsEGLRegisterImage (called here for reference) fails with
CUDA_ERROR_INVALID_DEVICEwhich from the docs means:Since we aren't creating a CUDA context in that code path, I suppose that we should interpret that as the action requested is invalid for the specified device, whatever that means.. So one simple test could be to try using the other devices, what about
/dev/dri/renderD129and so on? Do you get different results there?Ultimately, if all fails, I would suggest trying out ditching the Nvidia container toolkit and give it a go with the manual nvidia driver module from the quickstart guide. If you are looking for more clues, it might be helpful to also set the env variable
RUST_LOG=DEBUGto get some more info from our custom Wayland compositor@Gylfkxjyjdll commented on GitHub (Oct 31, 2025):
If I try a different GPU as /dev/dri/renderD129, I receive the same error with the new wolf-ui build. i tried the nvidia container toolkit and the manual driver module. both logs here.
gamesonwhale-23.log
gamesonwhale-22.log
@ABeltramo commented on GitHub (Oct 31, 2025):
@Gylfkxjyjdll thanks for testing latest!
I suppose that means that if you use
/dev/dri/renderD128it works, right?Can you share the logs of running Wolf with
renderD128withWOLF_LOG_LEVEL=DEBUGandRUST_LOG=INFOand then the same logs forrenderD129?Which exact GPUs do you have btw? I can see from the logs there's also an iGPU available..
@Gylfkxjyjdll commented on GitHub (Oct 31, 2025):
im sorry i was in hurry.
D128 is igpu from intel. never tried this with wolf
D129 is my "default" gaming gpu <- this works, always worked
D130-D132 i recive this error.
D130 log:
gamesonwhale-24.log
D129 log:
gamesonwhale-25.log
@zexi commented on GitHub (Oct 31, 2025):
@ABeltramo I've tried the other devices by container toolkit and manual way, still got the same results. The following logs with
RUST_LOG=DEBUG:wolf-ui.log
@zexi commented on GitHub (Nov 16, 2025):
@ABeltramo Could you help review this PR: https://github.com/games-on-whales/wolf/pull/287 and see if it's ready to merge? This should resolve the nvidia multi-GPU issue in my environment.
@ABeltramo commented on GitHub (Nov 16, 2025):
This has been merged to
wolf:stable, can you guys try it out on your setups and let me know if it's working for you too?@zexi commented on GitHub (Nov 17, 2025):
@ABeltramo The latest
wolf:stableimage is working in my environment.The other issue I found is that the encoder for gstreamer always uses the 1st card, even though the
WOLF_RENDER_NODEis specified to the other card.Is this related to this section https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/state/configTOML.cpp#L107-L109 of code?
@zexi commented on GitHub (Nov 23, 2025):
The issue of being unable to use the NVIDIA Encoder in a multi-GPU environment has been resolved.
I’ll open a new issue https://github.com/games-on-whales/wolf/issues/298 to record this problem https://github.com/games-on-whales/wolf/issues/233#issuecomment-3540074283.
@zexi commented on GitHub (Nov 23, 2025):
Another noteworthy issue is that we currently have to pass all NVIDIA GPUs to the container for GStreamer’s nvcodec to work properly.
This is related to NVIDIA’s drivers — for reference, here’s the link: https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-driver-565/347361 .
Once NVIDIA’s drivers fix this issue, we should be able to pass only one GPU to the Wolf container. This will make the GPU selection logic simpler and improve isolation.