games-on-whales/LXC2Docker

Fork 0

mirror of https://github.com/games-on-whales/LXC2Docker.git synced 2026-10-16 05:12:04 +00:00

[PR #76] [MERGED] feat(gpu): inject NVIDIA driver userspace into GPU containers #75

New issue

Closed

opened 2026-10-15 12:51:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:51:49 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/LXC2Docker/pull/76
Author: @JBailes
Created: 6/8/2026
Status: ✅ Merged
Merged: 6/14/2026
Merged by: @JBailes

Base: main ← Head: feat/nvidia-gpu-toolkit

📝 Commits (2)

d5a3fcc feat(gpu): inject NVIDIA driver userspace into GPU containers
e68a4ea fix(gpu): tolerate nvidia-container-cli ldcache EPERM in the mount hook

📊 Changes

4 files changed (+216 additions, -9 deletions)

View changed files

📝 Makefile (+3 -0)
📝 internal/lxc/config.go (+53 -0)
📝 internal/lxc/config_test.go (+53 -9)
➕ packaging/nvidia-hook.sh (+107 -0)

📄 Description

Problem

A container that requests the GPU the games-on-whales / --gpus way — by setting NVIDIA_VISIBLE_DEVICES (which --gpus/compose deploy.devices[driver:nvidia] set alongside the libnvidia env) — only got the bound device nodes (/dev/dri, /dev/nvidia*). LXC2Docker has no notion of a GPU request: HostConfig.Runtime is accepted but ignored, and there's no DeviceRequests, so the NVIDIA driver userspace (libcuda, libnvidia-encode, libEGL_nvidia, libGLX_nvidia, GLVND/EGL configs) was never injected.

Result: the kernel render node opens, but CUDA/EGL init fails and GPU apps black-screen. Concretely with Wolf, the Wayland compositor panics:

Failed to open drm node /dev/dri/renderD128
CUDA initialization failed: Failed to load CUDA library
Supported DMA formats: []
panicked at wayland-display-core/.../renderer/mod.rs: unwrap() on None
→ video producer aborts → Moonlight connects but gets no video

Fix

When a container opts in (NVIDIA_VISIBLE_DEVICES set, not void/empty), emit a single lxc.hook.mount that runs nvidia-container-cli configure against the freshly-mounted rootfs — the same mechanism Docker's nvidia-container-runtime uses. Rather than hardcode a driver-version-specific file list, this defers to the NVIDIA container toolkit, which:

copies the matching driver libraries + device nodes into the rootfs,
runs ldconfig to create the SONAME symlinks,
installs the GLVND/EGL vendor configs,

all matched to the running host driver, so it keeps working across driver upgrades with no daemon change.

The hook reads the device/capability selection back from the container's LXC config (defaulting to all), maps NVIDIA_DRIVER_CAPABILITIES to toolkit flags, and self-skips when the toolkit is absent or the selection is void — so it's a no-op on non-NVIDIA hosts and for containers that didn't ask for a GPU.

Changes

internal/lxc/config.go: nvidiaGPUItems / nvidiaGPURequested helpers; emit the hook from both buildItems (raw LXC) and buildPVEItems (Proxmox CT), right after the env block.
packaging/nvidia-hook.sh: the mount hook (POSIX sh).
Makefile: ship the hook at /usr/libexec/docker-lxc-daemon/nvidia-hook.sh via install and deb (+ uninstall cleanup).
internal/lxc/config_test.go: tests for opt-in (both builders) and no-hook-without-opt-in (incl. empty/void).

Notes / scope

Detection keys on NVIDIA_VISIBLE_DEVICES because that env is present in every GOW NVIDIA form (docker run -e … --gpus, compose env, compose deploy.devices). Parsing Docker DeviceRequests/--gpus directly could be a follow-up for callers that set only --gpus with no env.
Device cgroup access relies on the container already permitting the device majors (privileged GOW containers use lxc.cgroup2.devices.allow: a); the hook passes --no-cgroups and only touches the rootfs. An unprivileged path (emitting per-major cgroup allows) could be a follow-up.

go test ./internal/... passes; hook validated with sh -n.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/LXC2Docker/pull/76 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 6/8/2026 **Status:** ✅ Merged **Merged:** 6/14/2026 **Merged by:** [@JBailes](https://github.com/JBailes) **Base:** `main` ← **Head:** `feat/nvidia-gpu-toolkit` --- ### 📝 Commits (2) - [`d5a3fcc`](https://github.com/games-on-whales/LXC2Docker/commit/d5a3fcc1ccdac1d31dc62e4015b2736fa1728a85) feat(gpu): inject NVIDIA driver userspace into GPU containers - [`e68a4ea`](https://github.com/games-on-whales/LXC2Docker/commit/e68a4ea21f3aa5c2af413f6755af17f21fd1a29f) fix(gpu): tolerate nvidia-container-cli ldcache EPERM in the mount hook ### 📊 Changes **4 files changed** (+216 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `Makefile` (+3 -0) 📝 `internal/lxc/config.go` (+53 -0) 📝 `internal/lxc/config_test.go` (+53 -9) ➕ `packaging/nvidia-hook.sh` (+107 -0) </details> ### 📄 Description ## Problem A container that requests the GPU the games-on-whales / `--gpus` way — by setting `NVIDIA_VISIBLE_DEVICES` (which `--gpus`/compose `deploy.devices[driver:nvidia]` set alongside the libnvidia env) — only got the bound device nodes (`/dev/dri`, `/dev/nvidia*`). LXC2Docker has no notion of a *GPU request*: `HostConfig.Runtime` is accepted but ignored, and there's no `DeviceRequests`, so the NVIDIA driver **userspace** (`libcuda`, `libnvidia-encode`, `libEGL_nvidia`, `libGLX_nvidia`, GLVND/EGL configs) was never injected. Result: the kernel render node opens, but CUDA/EGL init fails and GPU apps black-screen. Concretely with **Wolf**, the Wayland compositor panics: ``` Failed to open drm node /dev/dri/renderD128 CUDA initialization failed: Failed to load CUDA library Supported DMA formats: [] panicked at wayland-display-core/.../renderer/mod.rs: unwrap() on None → video producer aborts → Moonlight connects but gets no video ``` ## Fix When a container opts in (`NVIDIA_VISIBLE_DEVICES` set, not `void`/empty), emit a single `lxc.hook.mount` that runs `nvidia-container-cli configure` against the freshly-mounted rootfs — the **same mechanism Docker's `nvidia-container-runtime` uses**. Rather than hardcode a driver-version-specific file list, this defers to the NVIDIA container toolkit, which: - copies the matching driver libraries + device nodes into the rootfs, - runs `ldconfig` to create the SONAME symlinks, - installs the GLVND/EGL vendor configs, all matched to the **running host driver**, so it keeps working across driver upgrades with no daemon change. The hook reads the device/capability selection back from the container's LXC config (defaulting to `all`), maps `NVIDIA_DRIVER_CAPABILITIES` to toolkit flags, and **self-skips** when the toolkit is absent or the selection is `void` — so it's a no-op on non-NVIDIA hosts and for containers that didn't ask for a GPU. ## Changes - `internal/lxc/config.go`: `nvidiaGPUItems` / `nvidiaGPURequested` helpers; emit the hook from both `buildItems` (raw LXC) and `buildPVEItems` (Proxmox CT), right after the env block. - `packaging/nvidia-hook.sh`: the mount hook (POSIX `sh`). - `Makefile`: ship the hook at `/usr/libexec/docker-lxc-daemon/nvidia-hook.sh` via `install` and `deb` (+ `uninstall` cleanup). - `internal/lxc/config_test.go`: tests for opt-in (both builders) and no-hook-without-opt-in (incl. empty/`void`). ## Notes / scope - Detection keys on `NVIDIA_VISIBLE_DEVICES` because that env is present in every GOW NVIDIA form (`docker run -e … --gpus`, compose env, compose `deploy.devices`). Parsing Docker `DeviceRequests`/`--gpus` directly could be a follow-up for callers that set *only* `--gpus` with no env. - Device cgroup access relies on the container already permitting the device majors (privileged GOW containers use `lxc.cgroup2.devices.allow: a`); the hook passes `--no-cgroups` and only touches the rootfs. An unprivileged path (emitting per-major cgroup allows) could be a follow-up. `go test ./internal/...` passes; hook validated with `sh -n`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:51:49 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

games-on-whales/LXC2Docker#75

No description provided.

Rows
Columns