[GH-ISSUE #484] Questions about the Necessity of NVIDIA Driver Config Parameters #311

New issue

Open

opened 2026-10-26 09:58:25 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 09:58:25 +00:00

Owner

Originally created by @michaelchen1996 on GitHub (Nov 6, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/484

I have a question regarding the necessity of specific NVIDIA configuration parameters in the current version of DeepEP.

options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"

However, in our tests, after removing these two parameters from initramfs, we still observe the following outputs:

NIC buffer will be on GPU memory
NIC handler will be GPU

From my observations, the version of cuStreamWaitValue64 used in nvshmem is 11070_v2, as indicated by:

LOAD_SYM(table, cuStreamWriteValue64, 11070, _v2, 1);
LOAD_SYM(table, cuStreamWaitValue64, 11070, _v2, 1);

According to the NVIDIA CUDA Driver API documentation, the NVreg_EnableStreamMemOPs=1 parameter is only required for v1 of these functions.

Additionally, does the PeerMappingOverride parameter only take effect in the case of non-root users?

So, are the parameters NVreg_EnableStreamMemOPs=1 and NVreg_RegistryDwords="PeerMappingOverride=1;" still necessary in the current DeepEP?

What impact (if any) would there be without these parameters—for instance, on performance?

Thank you!

Originally created by @michaelchen1996 on GitHub (Nov 6, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/484 I have a question regarding the necessity of specific NVIDIA configuration parameters in the current version of DeepEP. `options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"` However, in our tests, after removing these two parameters from initramfs, we still observe the following outputs: ``` NIC buffer will be on GPU memory NIC handler will be GPU ``` From my observations, the version of `cuStreamWaitValue64` used in `nvshmem` is `11070_v2`, as indicated by: ``` LOAD_SYM(table, cuStreamWriteValue64, 11070, _v2, 1); LOAD_SYM(table, cuStreamWaitValue64, 11070, _v2, 1); ``` According to the NVIDIA CUDA Driver API documentation, the `NVreg_EnableStreamMemOPs=1` parameter is only required for `v1` of these functions. Additionally, does the `PeerMappingOverride` parameter only take effect in the case of non-root users? So, are the parameters `NVreg_EnableStreamMemOPs=1` and `NVreg_RegistryDwords="PeerMappingOverride=1;"` still necessary in the current DeepEP? What impact (if any) would there be without these parameters—for instance, on performance? Thank you!

bsakaguchi commented

2026-10-26 09:58:26 +00:00

Author

Owner

@sphish commented on GitHub (Nov 6, 2025):

I’m not an expert on CUDA drivers, but I suspect this might be related to recent updates in NVSHMEM. You could try asking about it in the NVSHMEM repository.

@sphish commented on GitHub (Nov 6, 2025): I’m not an expert on CUDA drivers, but I suspect this might be related to recent updates in NVSHMEM. You could try asking about it in the NVSHMEM repository.

bsakaguchi commented

2026-10-26 09:58:26 +00:00

Author

Owner

@michaelchen1996 commented on GitHub (Dec 19, 2025):

I got a reply from in NVSHMEM repo.

https://github.com/NVIDIA/nvshmem/issues/23#issuecomment-3672239968

Starting with 3.4.5, you do not need them anymore, for CPU-assisted IBGDA. You will still need them for fully-device-assisted IBGDA. See the release notes: https://docs.nvidia.com/nvshmem/release-notes-install-guide/release-notes/release-3405.html#key-features-and-enhancements

@michaelchen1996 commented on GitHub (Dec 19, 2025): I got a reply from in NVSHMEM repo. https://github.com/NVIDIA/nvshmem/issues/23#issuecomment-3672239968 > Starting with 3.4.5, you do not need them anymore, for CPU-assisted IBGDA. You will still need them for fully-device-assisted IBGDA. See the release notes: https://docs.nvidia.com/nvshmem/release-notes-install-guide/release-notes/release-3405.html#key-features-and-enhancements

bsakaguchi commented

2026-10-26 09:58:26 +00:00

Author

Owner

@cxz66666 commented on GitHub (Dec 24, 2025):

I find a nvidia config explanation from https://gist.github.com/denji/52b9b0980ef3dadde0ff3d3ccf74a2a6, seem that these two params both are provided for Non-root users and not necessary for root user

NVreg_EnableStreamMemOPs=1 (Default: 0)
Enables CUDA Stream Memory Operations in user-mode applications.
Requires nvidia.ko version 510.40.3 or later and must be loaded with:
NVreg_RegistryDwords="PeerMappingOverride=1;"
More details: https://docs.nvidia.com/nvshmem/release-notes-install-guide/install-guide/abstract.html

@cxz66666 commented on GitHub (Dec 24, 2025): I find a nvidia config explanation from https://gist.github.com/denji/52b9b0980ef3dadde0ff3d3ccf74a2a6, seem that these two params both are provided for Non-root users and not necessary for root user > **NVreg_EnableStreamMemOPs**=1 (Default: 0) Enables CUDA Stream Memory Operations in user-mode applications. Requires nvidia.ko version 510.40.3 or later and must be loaded with: NVreg_RegistryDwords="**PeerMappingOverride**=1;" More details: https://docs.nvidia.com/nvshmem/release-notes-install-guide/install-guide/abstract.html

bsakaguchi referenced this issue

2026-10-27 21:03:31 +00:00

[PR #311] [MERGED] Add diagnosis module for efficient and precise location of slow rank #513