mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #11] Installing issue with gdrdrv #12
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#12
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @kevinhuangxf on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/11
Hi~
Thanks for the great work!
I'm installing the GDRCopy dependencies but I encouter the following issue.
The README file indicate a 2.4-4 version while my installation appears a 2.5-1 version:
The make.log file is:
However, I can not find the nv-p2p.h in the lib:
Any clue for solving this issue?
Thanks.
@Jackhally520 commented on GitHub (Feb 25, 2025):
Looking forward to answers
@daneren commented on GitHub (Feb 25, 2025):
Looking forward to answers
@haswelliris commented on GitHub (Feb 25, 2025):
I regret the oversight regarding the upstream update of GDRCopy. I will now specify the version of GDRCopy for clarity.
For instance, when using Ubuntu 22.04 and CUDA 12.3 as an example, the installation steps are as follows:
@Baibaifan commented on GitHub (Feb 25, 2025):
sudo ./insmod.sh ->sudo ../insmod.sh? @haswelliris
@xibosun commented on GitHub (Feb 25, 2025):
Hi, the instruction asks us to install packages in the docker without rebuilding modules. However, in my case, the host is a centos machine while the docker is a ubuntu system. So, even though I have successfully installed gdrdr on the host, I do not have pre-built packages (gdrcopy, libgdrapi, and gdrcopy-tests) for the ubuntu environment. May I kindly ask how to install these packages inside the docker?
@haswelliris commented on GitHub (Feb 25, 2025):
It is advisable to use the same distribution across the environment.
If this condition proves difficult to achieve, consider utilizing a privileged container. In this scenario, mount the host directory
/usr/src/nvidia-${NVIDIA_DRIVER_VERSION}into the container, as these driver headers are necessary for a successful build. Following this, proceed with the corresponding compilation and installation.@xibosun commented on GitHub (Feb 25, 2025):
Thanks a lot!
@Sun1Plus commented on GitHub (Mar 12, 2025):
I encountered the same situation,
there's no nvidia-* under /usr/src, does anyone know how to solve it? Thanks
@Sun1Plus commented on GitHub (Mar 13, 2025):
I encoutered the same question, and my question happened because i exec the command in a container(k8s pod), I solved it by exec "install gdrdrv" in real env rather than container.
hope it helpful~
@BestDreamy commented on GitHub (Mar 21, 2025):
Hi, Could you tell me how to solve this problem, please?
I also use centos with the ubuntu docker.
Just use "--priviledged=true -v /usr/src/nvidia-${NVIDIA_DRIVER_VERSION}:/usr/src/nvidia-${NVIDIA_DRIVER_VERSION}" is ok?
@galeselee commented on GitHub (Apr 6, 2025):
The compliation of GDRCopy needs nv driver. You may compile this on your compute-node. (this is the cause for me)
@MarsMeng1994 commented on GitHub (Apr 23, 2025):
$ sudo ./insmod.sh
insmod: ERROR: could not insert module src/gdrdrv/gdrdrv.ko: Operation not permitted
INFO: driver major is
INFO: creating /dev/gdrdrv inode
mknod: missing operand after '0'
Try 'mknod --help' for more information.
chmod: cannot access '/dev/gdrdrv': No such file or directory
how to slove?
@Sun1Plus commented on GitHub (Apr 23, 2025):
@MarsMeng1994, I encountered the same error, these .ko files could be found in "/lib/modules/5.15.0-138-generic/updates/dkms/" after you installed gdrdrv.
I change the insmod.sh line 28, pointer gdrdrv.ko to above path, then it runs ok.
@MarsMeng1994 commented on GitHub (Apr 24, 2025):
thk, i try but not work, may it a permision error?
@Sun1Plus commented on GitHub (Apr 24, 2025):
could you show the complete output include "dpkg -I gdrdrv-xxx" & "./insmod.sh"?
@MarsMeng1994 commented on GitHub (Apr 24, 2025):
$ sudo NVIDIA_SRC_DIR=/sharedata/msm/workspace/DeepEP/third-party/NVIDIA-Linux-x86_64-550.144.03/kernel/nvidia/ dpkg -i gdrdrv-dkms_2.4.4_amd64.Ubuntu22_04.deb
(Reading database ... 61771 files and directories currently installed.)
Preparing to unpack gdrdrv-dkms_2.4.4_amd64.Ubuntu22_04.deb ...
Module gdrdrv-2.4.4 for kernel 5.15.0-124-generic (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.
gdrdrv.ko:
depmod...
Deleting module gdrdrv-2.4.4 completely from the DKMS tree.
Unpacking gdrdrv-dkms:amd64 (2.4.4) over (2.4.4) ...
Setting up gdrdrv-dkms:amd64 (2.4.4) ...
Loading new gdrdrv-2.4.4 DKMS files...
Building for 5.15.0-124-generic
Building for architecture x86_64
Building initial module for 5.15.0-124-generic
Done.
gdrdrv.ko:
Running module version sanity check.
depmod...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
$ sudo ./insmod.sh
insmod: ERROR: could not insert module /lib/modules/5.15.0-124-generic/updates/dkms/gdrdrv.ko: Operation not permitted
INFO: driver major is
INFO: creating /dev/gdrdrv inode
mknod: missing operand after '0'
Try 'mknod --help' for more information.
chmod: cannot access '/dev/gdrdrv': No such file or directory
$ cat ./insmod.sh
THIS_DIR=$(dirname $0)
grep gdrdrv /proc/devices >/dev/null && sudo /sbin/rmmod gdrdrv
sudo /sbin/insmod /lib/modules/5.15.0-124-generic/updates/dkms/gdrdrv.ko dbg_enabled=0 info_enabled=0 use_persistent_mapping=0
major=
fgrep gdrdrv /proc/devices | cut -b 1-4echo "INFO: driver major is $major"
if [ -e /dev/gdrdrv ]; then
sudo rm /dev/gdrdrv
fi
echo "INFO: creating /dev/gdrdrv inode"
sudo mknod /dev/gdrdrv c $major 0
sudo chmod a+w+r /dev/gdrdrv
@Sun1Plus commented on GitHub (Apr 24, 2025):
following is my output after executing "dpkg -i gdrdrv-dkms_2.4.4_amd64.Ubuntu22_04.deb"
And your output is:
it seems like your env didn't install gdrdrv successfully, but I didn't encounter it, I don't know how to solve it.
@MarsMeng1994 commented on GitHub (Apr 24, 2025):
I asked our company's operation and maintenance, and he said that the problem was that the privileged mode was not turned on. Then he operated it and the error was gone. Professionals do professional things.
@sphish commented on GitHub (Jun 25, 2025):
After https://github.com/deepseek-ai/DeepEP/pull/201, DeepEP can be built without depending on gdrcopy.