games-on-whales/wolf
1
0
Fork 0
mirror of https://github.com/games-on-whales/wolf.git synced 2026-10-15 13:02:03 +00:00
Code Issues 119 Projects Packages Wiki Activity

[GH-ISSUE #298] [Nvidia] Gstreamer encoder always uses the 1st card with multiple nvidia GPUs #209

New issue
Open
opened 2026-10-15 12:58:14 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-15 12:58:14 +00:00
Owner
Copy link

Originally created by @zexi on GitHub (Nov 23, 2025).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/298

Problem Description

I found that the GStreamer encoder always uses the first card, even though WOLF_RENDER_NODE is set to the other card, e.g., the render path is /dev/dri/renderD129 and the CUDA device ID is 2.

Here’s the VRAM screenshot via nvidia-smi:

Image

Is this related to this section https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/state/configTOML.cpp#L107-L109 of code?

Possible Solutions

If I manually set the environment variable CUDA_VISIBLE_DEVICES=2, Wolf can use the specified GPU and everything works properly.

However, there is a warning Failed to create CUDA context for device in the log, and you can refer to the screenshot below:

Image

My question is: since the CUDA context creation has failed here, why does everything still work?

Originally created by @zexi on GitHub (Nov 23, 2025). Original GitHub issue: https://github.com/games-on-whales/wolf/issues/298 # Problem Description I found that the GStreamer encoder always uses the first card, even though WOLF_RENDER_NODE is set to the other card, e.g., the render path is /dev/dri/renderD129 and the CUDA device ID is 2. Here’s the VRAM screenshot via nvidia-smi: <img width="831" height="174" alt="Image" src="https://github.com/user-attachments/assets/eef13c8f-055b-40ca-98af-ec6ade2a04f8" /> Is this related to this section https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/state/configTOML.cpp#L107-L109 of code? # Possible Solutions If I manually set the environment variable `CUDA_VISIBLE_DEVICES=2`, Wolf can use the specified GPU and everything works properly. However, there is a warning `Failed to create CUDA context for device` in the log, and you can refer to the screenshot below: <img width="1508" height="393" alt="Image" src="https://github.com/user-attachments/assets/dfd7fb84-a86f-4580-a595-8d0603bfd802" /> My question is: since the CUDA context creation has failed here, why does everything still work?
bsakaguchi commented 2026-10-15 12:58:15 +00:00
Author
Owner
Copy link

@n0deserve commented on GitHub (May 21, 2026):

I ran into the same issue where it was always defaulting to the first GPU. When Wolf creates the CUDA context, NVML can see both cards and it must default to the first card (GPU 0).

My workaround was to pin it using the card's UID instead:

docker exec wolf nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5070 Ti (UUID: GPU-your-UID-GPU0)
GPU 1: NVIDIA GeForce RTX 5070 Ti (UUID: GPU-your-UID-GPU1)

And in compose.yml:

      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=GPU-your-UID-GPU1
      - CUDA_VISIBLE_DEVICES=GPU-your-UID-GPU1
      - WOLF_RENDER_NODE=/dev/dri/renderD130
<!-- gh-comment-id:4504043634 --> @n0deserve commented on GitHub (May 21, 2026): I ran into the same issue where it was always defaulting to the first GPU. When Wolf creates the CUDA context, NVML can see both cards and it must default to the first card (GPU 0). My workaround was to pin it using the card's UID instead: ```bash docker exec wolf nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5070 Ti (UUID: GPU-your-UID-GPU0) GPU 1: NVIDIA GeForce RTX 5070 Ti (UUID: GPU-your-UID-GPU1) ``` And in `compose.yml`: ```yaml - NVIDIA_DRIVER_CAPABILITIES=all - NVIDIA_VISIBLE_DEVICES=GPU-your-UID-GPU1 - CUDA_VISIBLE_DEVICES=GPU-your-UID-GPU1 - WOLF_RENDER_NODE=/dev/dri/renderD130 ```
bsakaguchi commented 2026-10-15 12:58:15 +00:00
Author
Owner
Copy link

@ABeltramo commented on GitHub (May 30, 2026):

@n0deserve since this was first opened we've added proper support for multiple Nvidia GPUs; setting the WOLF_RENDER_NODE=/dev/dri/renderD130 environment variable should be the only required step as specified in the docs.
Can you please verify if that's the case for your setup too or do you actually need the Nvidia container toolkit additional env variables?

<!-- gh-comment-id:4582335218 --> @ABeltramo commented on GitHub (May 30, 2026): @n0deserve since this was first opened we've added proper support for multiple Nvidia GPUs; setting the `WOLF_RENDER_NODE=/dev/dri/renderD130` environment variable should be the only required step as specified in the [docs](https://games-on-whales.github.io/wolf/stable/user/configuration.html#_multiple_gpu). Can you please verify if that's the case for your setup too or do you actually need the Nvidia container toolkit additional env variables?
bsakaguchi commented 2026-10-15 12:58:15 +00:00
Author
Owner
Copy link

@tjacoby2006 commented on GitHub (Sep 15, 2026):

I am having the same or similar issue, basically I have two GOW services running, each pinned to a GPU. The first should be GPU 0, the second should be pinned to GPU 1. For some reason the compute for the second service is running on GPU 0, while the encode is running on GPU 1.

version: "3"

services:
  wolf-gpu0:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-0
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=0
      - WOLF_RENDER_NODE=/dev/dri/renderD128
    volumes:
      - ~/sambashare/GamesOnWhales/0:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
    device_cgroup_rules:
      - 'c 13:* rmw'
    devices:
      - /dev/dri/card0
      - /dev/dri/renderD128
      - /dev/uinput
      - /dev/uhid
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.4
    restart: unless-stopped

  wolf-gpu1:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-1
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=1
      - WOLF_RENDER_NODE=/dev/dri/renderD129
      - CUDA_VISIBLE_DEVICES=1
    volumes:
      - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
    device_cgroup_rules:
      - 'c 13:* rmw'
    devices:
      - /dev/dri/card1
      - /dev/dri/renderD129
      - /dev/uinput
      - /dev/uhid
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.3
    restart: unless-stopped

networks:
  wolf-macvlan:
    driver: macvlan
    driver_opts:
      parent: enp4s0
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

This is with two games running on the second service-

Image
<!-- gh-comment-id:5680425117 --> @tjacoby2006 commented on GitHub (Sep 15, 2026): I am having the same or similar issue, basically I have two GOW services running, each pinned to a GPU. The first should be GPU 0, the second should be pinned to GPU 1. For some reason the compute for the second service is running on GPU 0, while the encode is running on GPU 1. ```yml version: "3" services: wolf-gpu0: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-0 environment: - NVIDIA_DRIVER_CAPABILITIES=all - NVIDIA_VISIBLE_DEVICES=0 - WOLF_RENDER_NODE=/dev/dri/renderD128 volumes: - ~/sambashare/GamesOnWhales/0:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw device_cgroup_rules: - 'c 13:* rmw' devices: - /dev/dri/card0 - /dev/dri/renderD128 - /dev/uinput - /dev/uhid deploy: resources: reservations: devices: - driver: nvidia device_ids: ['0'] capabilities: [gpu] networks: wolf-macvlan: ipv4_address: 192.168.1.4 restart: unless-stopped wolf-gpu1: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-1 environment: - NVIDIA_DRIVER_CAPABILITIES=all - NVIDIA_VISIBLE_DEVICES=1 - WOLF_RENDER_NODE=/dev/dri/renderD129 - CUDA_VISIBLE_DEVICES=1 volumes: - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw device_cgroup_rules: - 'c 13:* rmw' devices: - /dev/dri/card1 - /dev/dri/renderD129 - /dev/uinput - /dev/uhid deploy: resources: reservations: devices: - driver: nvidia device_ids: ['1'] capabilities: [gpu] networks: wolf-macvlan: ipv4_address: 192.168.1.3 restart: unless-stopped networks: wolf-macvlan: driver: macvlan driver_opts: parent: enp4s0 ipam: config: - subnet: 192.168.1.0/24 gateway: 192.168.1.1 ``` This is with two games running on the second service- <img width="1681" height="856" alt="Image" src="https://github.com/user-attachments/assets/4bf70c1a-9fcc-40eb-b84e-edce92618f0d" />
bsakaguchi commented 2026-10-15 12:58:15 +00:00
Author
Owner
Copy link

@ABeltramo commented on GitHub (Sep 15, 2026):

@tjacoby2006 mixing up Nvidia container toolkit and manual /dev/dri mounts would most likely cause issues. You should use only one method to separate the GPUs, if done right, it should work.

The real fix would be support multi-GPU load balancing in Wolf though. We've discussed this multiple times but we never got around to actually implement it.

<!-- gh-comment-id:5685065205 --> @ABeltramo commented on GitHub (Sep 15, 2026): @tjacoby2006 mixing up Nvidia container toolkit and manual `/dev/dri` mounts would most likely cause issues. You should use only one method to separate the GPUs, if done right, it should work. The real fix would be support multi-GPU load balancing in Wolf though. We've discussed this multiple times but we never got around to actually implement it.
bsakaguchi commented 2026-10-15 12:58:16 +00:00
Author
Owner
Copy link

@tjacoby2006 commented on GitHub (Sep 15, 2026):

I pivoted to using the manual method, but am still seeing the same behavior.

version: "3"

services:
  wolf-gpu0:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-0
    environment:
      - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol
      - WOLF_RENDER_NODE=/dev/dri/renderD128
    volumes:
      - ~/sambashare/GamesOnWhales/0:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
      - nvidia-driver-vol:/usr/nvidia:rw
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
      - /dev/nvidiactl
      - /dev/nvidia0
      - /dev/nvidia1
      - /dev/nvidia-modeset
    device_cgroup_rules:
      - 'c 13:* rmw'
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.4
    restart: unless-stopped

  wolf-gpu1:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-1
    environment:
      - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol
      - WOLF_RENDER_NODE=/dev/dri/renderD129
    volumes:
      - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
      - nvidia-driver-vol:/usr/nvidia:rw
    device_cgroup_rules:
      - 'c 13:* rmw'
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
      - /dev/nvidiactl
      - /dev/nvidia0
      - /dev/nvidia1
      - /dev/nvidia-modeset
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.3
    restart: unless-stopped

volumes:
  nvidia-driver-vol:
    external: true

networks:
  wolf-macvlan:
    driver: macvlan
    driver_opts:
      parent: enp4s0
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

We still see encoding on device 0 and rendering on device 1.
Image

Thank you for your help on this.

<!-- gh-comment-id:5686102081 --> @tjacoby2006 commented on GitHub (Sep 15, 2026): I pivoted to using the manual method, but am still seeing the same behavior. ```yml version: "3" services: wolf-gpu0: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-0 environment: - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol - WOLF_RENDER_NODE=/dev/dri/renderD128 volumes: - ~/sambashare/GamesOnWhales/0:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw - nvidia-driver-vol:/usr/nvidia:rw devices: - /dev/dri - /dev/uinput - /dev/uhid - /dev/nvidia-uvm - /dev/nvidia-uvm-tools - /dev/nvidia-caps/nvidia-cap1 - /dev/nvidia-caps/nvidia-cap2 - /dev/nvidiactl - /dev/nvidia0 - /dev/nvidia1 - /dev/nvidia-modeset device_cgroup_rules: - 'c 13:* rmw' networks: wolf-macvlan: ipv4_address: 192.168.1.4 restart: unless-stopped wolf-gpu1: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-1 environment: - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol - WOLF_RENDER_NODE=/dev/dri/renderD129 volumes: - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw - nvidia-driver-vol:/usr/nvidia:rw device_cgroup_rules: - 'c 13:* rmw' devices: - /dev/dri - /dev/uinput - /dev/uhid - /dev/nvidia-uvm - /dev/nvidia-uvm-tools - /dev/nvidia-caps/nvidia-cap1 - /dev/nvidia-caps/nvidia-cap2 - /dev/nvidiactl - /dev/nvidia0 - /dev/nvidia1 - /dev/nvidia-modeset networks: wolf-macvlan: ipv4_address: 192.168.1.3 restart: unless-stopped volumes: nvidia-driver-vol: external: true networks: wolf-macvlan: driver: macvlan driver_opts: parent: enp4s0 ipam: config: - subnet: 192.168.1.0/24 gateway: 192.168.1.1 ``` We still see encoding on device 0 and rendering on device 1. <img width="922" height="984" alt="Image" src="https://github.com/user-attachments/assets/f1b660d3-727c-4918-ba44-fc280650c18b" /> Thank you for your help on this.
bsakaguchi commented 2026-10-15 12:58:16 +00:00
Author
Owner
Copy link

@tjacoby2006 commented on GitHub (Sep 15, 2026):

I may have gotten it. Fix yml-

version: "3"

services:
  wolf-gpu0:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-0
    environment:
      - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol
      - WOLF_RENDER_NODE=/dev/dri/renderD128
    volumes:
      - ~/sambashare/GamesOnWhales/0:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
      - nvidia-driver-vol:/usr/nvidia:rw
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
      - /dev/nvidiactl
      - /dev/nvidia0
      - /dev/nvidia-modeset
    device_cgroup_rules:
      - 'c 13:* rmw'
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.4
    restart: unless-stopped

  wolf-gpu1:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf-1
    environment:
      - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol
      - WOLF_RENDER_NODE=/dev/dri/renderD129
    volumes:
      - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
      - nvidia-driver-vol:/usr/nvidia:rw
    device_cgroup_rules:
      - 'c 13:* rmw'
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
      - /dev/nvidiactl
      - /dev/nvidia1
      - /dev/nvidia-modeset
    networks:
      wolf-macvlan:
        ipv4_address: 192.168.1.3
    restart: unless-stopped

volumes:
  nvidia-driver-vol:
    external: true

networks:
  wolf-macvlan:
    driver: macvlan
    driver_opts:
      parent: enp4s0
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

The key was differentiating the /dev/nvidia* devices being passed through.

<!-- gh-comment-id:5686693558 --> @tjacoby2006 commented on GitHub (Sep 15, 2026): I may have gotten it. Fix yml- ```yml version: "3" services: wolf-gpu0: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-0 environment: - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol - WOLF_RENDER_NODE=/dev/dri/renderD128 volumes: - ~/sambashare/GamesOnWhales/0:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw - nvidia-driver-vol:/usr/nvidia:rw devices: - /dev/dri - /dev/uinput - /dev/uhid - /dev/nvidia-uvm - /dev/nvidia-uvm-tools - /dev/nvidia-caps/nvidia-cap1 - /dev/nvidia-caps/nvidia-cap2 - /dev/nvidiactl - /dev/nvidia0 - /dev/nvidia-modeset device_cgroup_rules: - 'c 13:* rmw' networks: wolf-macvlan: ipv4_address: 192.168.1.4 restart: unless-stopped wolf-gpu1: image: ghcr.io/games-on-whales/wolf:stable container_name: wolf-1 environment: - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol - WOLF_RENDER_NODE=/dev/dri/renderD129 volumes: - ~/sambashare/docker-data/GamesOnWhales/1:/etc/wolf - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw - nvidia-driver-vol:/usr/nvidia:rw device_cgroup_rules: - 'c 13:* rmw' devices: - /dev/dri - /dev/uinput - /dev/uhid - /dev/nvidia-uvm - /dev/nvidia-uvm-tools - /dev/nvidia-caps/nvidia-cap1 - /dev/nvidia-caps/nvidia-cap2 - /dev/nvidiactl - /dev/nvidia1 - /dev/nvidia-modeset networks: wolf-macvlan: ipv4_address: 192.168.1.3 restart: unless-stopped volumes: nvidia-driver-vol: external: true networks: wolf-macvlan: driver: macvlan driver_opts: parent: enp4s0 ipam: config: - subnet: 192.168.1.0/24 gateway: 192.168.1.1 ``` The key was differentiating the `/dev/nvidia*` devices being passed through.
bsakaguchi commented 2026-10-15 12:58:16 +00:00
Author
Owner
Copy link

@ABeltramo commented on GitHub (Sep 21, 2026):

The key was differentiating the /dev/nvidia* devices being passed through.

Good point, forgot to mention that. We should really fix this properly so I'm going to keep this one open..

<!-- gh-comment-id:5762303314 --> @ABeltramo commented on GitHub (Sep 21, 2026): > The key was differentiating the /dev/nvidia* devices being passed through. Good point, forgot to mention that. We should really fix this properly so I'm going to keep this one open..
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
beta
feat/fedora-base-images
stable
axolotlitehotfix/randomly-generated-client-id
feat/vulkan-image
ci/image-smoke-harness
nv12-producer-pin
fix/nvidia-zero-copy
fix/amd-dma-buf-negotiation
fix/joypad-input-leak
codex/party-mode-split-render
ci-fedora-images
test/gst-wld-block-linear
fix/nvidia-blackwell-zero-copy
feat/embed-wolf-den
fix/fake-udev-write-db
fix/session-runner-user
fedora-dockerfile
feat/comprehensive-display-modes
aimee/quickstart-scripts
JBailes-patch-1
dev-nix
dev-fake-uinput
dev-tracy
fix/gst_vaapi
v2024.07
v2023.06
v2023.02
v2022.11
Labels
Clear labels   
🧠 1 - 🤖 999

   
AMD

   
bug

   
bug

   
cannot reproduce

   
documentation

   
duplicate

   
easy

   
enhancement

   
enhancement

   
enhancement

   
fixed?

   
fixed?

   
good first issue

   
hard

   
help wanted

   
Intel

   
invalid

   
LXC

   
medium

   
moonlight-protocol

   
Multi GPU

   
nixos

   
not a bug

   
nvidia

   
pull-request
Mirrored from GitHub Pull Request

  
question

   
wontfix
No labels
🧠 1 - 🤖 999
AMD
bug
bug
cannot reproduce
documentation
duplicate
easy
enhancement
enhancement
enhancement
fixed?
fixed?
good first issue
hard
help wanted
Intel
invalid
LXC
medium
moonlight-protocol
Multi GPU
nixos
not a bug
nvidia
pull-request
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
games-on-whales/wolf#209
No description provided.