565.57.01

560.35.03
560.31.02
2026-01-27 19:49:47 +00:00 · 2024-10-22 17:38:58 +02:00 · 2024-08-19 10:46:21 -07:00 · 2024-07-31 11:27:06 -07:00 · 2024-07-19 15:45:15 -07:00 · 2024-07-19 15:38:11 -07:00
2802 changed files with 957581 additions and 174681 deletions
--- a/.github/ISSUE_TEMPLATE/10_functional_bug.yml
+++ b/.github/ISSUE_TEMPLATE/10_functional_bug.yml
@@ -1,5 +1,8 @@
 name: Report a functional bug 🐛
-description: Functional bugs affect operation or stability of the driver and/or hardware.
+description: |
+  Functional bugs affect operation or stability of the driver or hardware.
+
+  Bugs with the closed source driver must be reported on the forums (see link on New Issue page below).
 labels:
  - "bug"
 body:
@@ -18,14 +21,12 @@ body:
    description: "Which open-gpu-kernel-modules version are you running? Be as specific as possible: SHA is best when built from specific commit."
  validations:
    required: true
- type: dropdown
+- type: checkboxes
  id: sw_driver_proprietary
  attributes:
-    label: "Does this happen with the proprietary driver (of the same version) as well?"
+    label: "Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver."
    options:
-    - "Yes"
-    - "No"
-    - "I cannot test this"
+    - label: "I confirm that this does not happen with the proprietary driver package."
  validations:
    required: true
 - type: input
@@ -42,6 +43,14 @@ body:
    description: "Which kernel are you running? (output of `uname -a`, say if you built it yourself)"
  validations:
    required: true
+- type: checkboxes
+  id: sw_host_kernel_stable
+  attributes:
+    label: "Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels."
+    options:
+    - label: "I am running on a stable kernel release."
+  validations:
+    required: true
 - type: input
  id: hw_gpu_type
  attributes:
@@ -78,7 +87,10 @@ body:
  id: bug_report_gz
  attributes:
    label: nvidia-bug-report.log.gz
-    description: "Please reproduce the problem, after that run `nvidia-bug-report.sh`, and attach the resulting nvidia-bug-report.log.gz here."
+    description: |
+      Please reproduce the problem, after that run `nvidia-bug-report.sh`, and attach the resulting nvidia-bug-report.log.gz here.
+
+      Reports without this file will be closed.
    placeholder: You can usually just drag & drop the file into this textbox.
  validations:
    required: true
--- a/.github/ISSUE_TEMPLATE/20_build_bug.yml
+++ b/.github/ISSUE_TEMPLATE/20_build_bug.yml
@@ -32,6 +32,14 @@ body:
    description: "Which kernel are you running? (output of `uname -a`, say if you built it yourself)."
  validations:
    required: true
+- type: checkboxes
+  id: sw_host_kernel_stable
+  attributes:
+    label: "Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels."
+    options:
+    - label: "I am running on a stable kernel release."
+  validations:
+    required: true
 - type: textarea
  id: bug_description
  attributes:
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -1,14 +1,14 @@
 blank_issues_enabled: false
 contact_links:
+  - name: Report a bug with the proprietary driver
+    url: https://forums.developer.nvidia.com/c/gpu-graphics/linux/148
+    about: Bugs that aren't specific to the open source driver in this repository must be reported with the linked forums instead.
  - name: Report a cosmetic issue
    url: https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/categories/general
    about: We are not currently accepting cosmetic-only changes such as whitespace, typos, or simple renames. You can still discuss and collect them on the boards.
  - name: Ask a question
    url: https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/categories/q-a
    about: Unsure of what to click, where to go, what the process for your thing is? We're happy to help. Click to visit the discussion board and say hello!
-  - name: Report a bug with the proprietary driver
-    url: https://forums.developer.nvidia.com/c/gpu-graphics/linux/148
-    about: Bugs that aren't specific to the open source driver in this repository should be reported with the linked forums instead. If you are unsure on what kind of bug you have, feel free to open a thread in Discussions. We're here to help!
  - name: Suggest a feature
    url: https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/categories/ideas
    about: Please do not open Issues for feature requests; instead, suggest and discuss new features on the Github discussion board. If you have a feature you worked on and want to PR it, please also open a discussion before doing so.
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,44 +0,0 @@
-# Changelog
-
-## Release 515 Entries
-
-### [515.65.01] 2022-08-02
-
-#### Fixed
-
- Collection of minor fixes to issues, [#6](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/61) by @Joshua-Ashton
- Remove unnecessary use of acpi_bus_get_device().
-
-### [515.57] 2022-06-28
-
-#### Fixed
-
- Backtick is deprecated, [#273](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/273) by @arch-user-france1
-
-### [515.48.07] 2022-05-31
-
-#### Added
-
- List of compatible GPUs in README.md.
-
-#### Fixed
-
- Fix various README capitalizations, [#8](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/8) by @27lx 
- Automatically tag bug report issues, [#15](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/15) by @thebeanogamer
- Improve conftest.sh Script, [#37](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/37) by @Nitepone
- Update HTTP link to HTTPS, [#101](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/101) by @alcaparra
- moved array sanity check to before the array access, [#117](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/117) by @RealAstolfo
- Fixed some typos, [#122](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/122) by @FEDOyt
- Fixed capitalization, [#123](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/123) by @keroeslux
- Fix typos in NVDEC Engine Descriptor, [#126](https://github.com/NVIDIA/open-gpu-kernel-modules/pull/126) from @TrickyDmitriy
- Extranous apostrohpes in a makefile script [sic], [#14](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/14) by @kiroma
- HDMI no audio @ 4K above 60Hz, [#75](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/75) by @adolfotregosa
- dp_configcaps.cpp:405: array index sanity check in wrong place?, [#110](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/110) by @dcb314
- NVRM kgspInitRm_IMPL: missing NVDEC0 engine, cannot initialize GSP-RM, [#116](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/116) by @kfazz
- ERROR: modpost: "backlight_device_register" [...nvidia-modeset.ko] undefined, [#135](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/135) by @sndirsch
- aarch64 build fails, [#151](https://github.com/NVIDIA/open-gpu-kernel-modules/issues/151) by @frezbo
-
-### [515.43.04] 2022-05-11
-
- Initial release.
-
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,3 +1,21 @@
 Thank you for all the enthusiasm around open-gpu-kernel-modules.

-For the next while, we are going to focus review on pull requests of functional changes.  Please refrain from further cosmetic pull requests until we publish our style guide.
+## Non-functional (cosmetic) changes
+
+While we appreciate your enthusiasm, we have decided not to accept non-functional changes such as
+non-code typo fixes, comment and language adjustments, whitespace changes, and similar.
+
+Changes going into this codebase incur significant overhead. As such, we want to focus our resources
+on executable code improvements for now.
+
+If you have questions, or are unsure about the nature of your desired change, please ask us on the
+[Discussion boards](https://github.com/NVIDIA/open-gpu-kernel-modules/discussions)!
+
+## Code style
+
+We currently do not publish a code style guide, as we have many different components coming together.
+Please read the existing code in the repository, especially the one surrounding your proposed change,
+to get a feel for what you should aim for.
+
+Don't worry too much about it! We are happy to guide you through any neccessary style changes through
+code review of your PR.
--- a/26
+++ b/26
@@ -6,9 +6,9 @@
 # To install the build kernel modules: run (as root) `make modules_install`
 ###########################################################################

-include utils.mk
-
-all: modules
+###########################################################################
+# variables
+###########################################################################

 nv_kernel_o                = src/nvidia/$(OUTPUTDIR)/nv-kernel.o
 nv_kernel_o_binary         = kernel-open/nvidia/nv-kernel.o_binary
@@ -16,13 +16,20 @@ nv_kernel_o_binary         = kernel-open/nvidia/nv-kernel.o_binary
 nv_modeset_kernel_o        = src/nvidia-modeset/$(OUTPUTDIR)/nv-modeset-kernel.o
 nv_modeset_kernel_o_binary = kernel-open/nvidia-modeset/nv-modeset-kernel.o_binary

-.PHONY: $(nv_kernel_o) $(nv_modeset_kernel_o) modules modules_install
+###########################################################################
+# rules
+###########################################################################

+include utils.mk
+
+.PHONY: all
+all: modules

 ###########################################################################
 # nv-kernel.o is the OS agnostic portion of nvidia.ko
 ###########################################################################

+.PHONY: $(nv_kernel_o)
 $(nv_kernel_o):
 	$(MAKE) -C src/nvidia

@@ -34,6 +41,7 @@ $(nv_kernel_o_binary): $(nv_kernel_o)
 # nv-modeset-kernel.o is the OS agnostic portion of nvidia-modeset.ko
 ###########################################################################

+.PHONY: $(nv_modeset_kernel_o)
 $(nv_modeset_kernel_o):
 	$(MAKE) -C src/nvidia-modeset

@@ -46,31 +54,33 @@ $(nv_modeset_kernel_o_binary): $(nv_modeset_kernel_o)
 # the kernel modules with kbuild.
 ###########################################################################

+.PHONY: modules
 modules: $(nv_kernel_o_binary) $(nv_modeset_kernel_o_binary)
 	$(MAKE) -C kernel-open modules

-
 ###########################################################################
 # Install the built kernel modules using kbuild.
 ###########################################################################

+.PHONY: modules_install
 modules_install:
 	$(MAKE) -C kernel-open modules_install

-
 ###########################################################################
 # clean
 ###########################################################################

-.PHONY: clean nvidia.clean nvidia-modeset.clean kernel-open.clean
-
+.PHONY: clean
 clean: nvidia.clean nvidia-modeset.clean kernel-open.clean

+.PHONY: nvidia.clean
 nvidia.clean:
 	$(MAKE) -C src/nvidia clean

+.PHONY: nvidia-modeset.clean
 nvidia-modeset.clean:
 	$(MAKE) -C src/nvidia-modeset clean

+.PHONY: kernel-open.clean
 kernel-open.clean:
 	$(MAKE) -C kernel-open clean
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
 # NVIDIA Linux Open GPU Kernel Module Source

 This is the source release of the NVIDIA Linux open GPU kernel modules,
-version 515.49.14.
+version 565.57.01.


 ## How to Build
@@ -15,9 +15,9 @@ as root:

    make modules_install -j$(nproc)

-Note that the kernel modules built here must be used with gsp.bin
+Note that the kernel modules built here must be used with GSP
 firmware and user-space NVIDIA GPU driver components from a corresponding
-515.49.14 driver release.  This can be achieved by installing
+565.57.01 driver release.  This can be achieved by installing
 the NVIDIA GPU driver from the .run file using the `--no-kernel-modules`
 option.  E.g.,

@@ -74,7 +74,7 @@ kernel.

 The NVIDIA open kernel modules support the same range of Linux kernel
 versions that are supported with the proprietary NVIDIA kernel modules.
-This is currently Linux kernel 3.10 or newer.
+This is currently Linux kernel 4.15 or newer.


 ## How to Contribute
@@ -162,20 +162,33 @@ for the target kernel.
 - `src/nvidia/`                 The OS-agnostic code for nvidia.ko
 - `src/nvidia-modeset/`         The OS-agnostic code for nvidia-modeset.ko
 - `src/common/`                 Utility code used by one or more of nvidia.ko and nvidia-modeset.ko
+- `nouveau/`                    Tools for integration with the Nouveau device driver
+
+
+## Nouveau device driver integration
+
+The Python script in the 'nouveau' directory is used to extract some of the
+firmware binary images (and related data) encoded in the source code and
+store them as distinct files.  These files are used by the Nouveau device
+driver to load and communicate with the GSP firmware.
+
+The layout of the binary files is described in nouveau_firmware_layout.ods,
+which is an OpenDocument Spreadsheet file, compatible with most spreadsheet
+software applications.


 ## Compatible GPUs

-The open-gpu-kernel-modules can be used on any Turing or later GPU
-(see the table below). However, in the 515.49.14 release,
-GeForce and Workstation support is still considered alpha-quality.
+The NVIDIA open kernel modules can be used on any Turing or later GPU (see the
+table below).

-To enable use of the open kernel modules on GeForce and Workstation GPUs,
-set the "NVreg_OpenRmEnableUnsupportedGpus" nvidia.ko kernel module
-parameter to 1. For more details, see the NVIDIA GPU driver end user
-README here:
+For details on feature support and limitations, see the NVIDIA GPU driver
+end user README here:

-https://us.download.nvidia.com/XFree86/Linux-x86_64/515.49.14/README/kernel_open.html
+https://us.download.nvidia.com/XFree86/Linux-x86_64/565.57.01/README/kernel_open.html
+
+For vGPU support, please refer to the README.vgpu packaged in the vGPU Host
+Package for more details.

 In the below table, if three IDs are listed, the first is the PCI Device 
 ID, the second is the PCI Subsystem Vendor ID, and the third is the PCI
@@ -186,6 +199,7 @@ Subsystem Device ID.
 | NVIDIA TITAN RTX                                | 1E02           |
 | NVIDIA GeForce RTX 2080 Ti                      | 1E04           |
 | NVIDIA GeForce RTX 2080 Ti                      | 1E07           |
+| NVIDIA CMP 50HX                                 | 1E09           |
 | Quadro RTX 6000                                 | 1E30           |
 | Quadro RTX 8000                                 | 1E30 1028 129E |
 | Quadro RTX 8000                                 | 1E30 103C 129E |
@@ -378,6 +392,7 @@ Subsystem Device ID.
 | NVIDIA GeForce RTX 2070                         | 1F07           |
 | NVIDIA GeForce RTX 2060                         | 1F08           |
 | NVIDIA GeForce GTX 1650                         | 1F0A           |
+| NVIDIA CMP 40HX                                 | 1F0B           |
 | NVIDIA GeForce RTX 2070                         | 1F10           |
 | NVIDIA GeForce RTX 2070 with Max-Q Design       | 1F10 1025 132D |
 | NVIDIA GeForce RTX 2070 with Max-Q Design       | 1F10 1025 1342 |
@@ -635,25 +650,50 @@ Subsystem Device ID.
 | NVIDIA T1000 8GB                                | 1FF0 17AA 1612 |
 | NVIDIA T400 4GB                                 | 1FF2 1028 1613 |
 | NVIDIA T400 4GB                                 | 1FF2 103C 1613 |
+| NVIDIA T400E                                    | 1FF2 103C 18FF |
 | NVIDIA T400 4GB                                 | 1FF2 103C 8A80 |
 | NVIDIA T400 4GB                                 | 1FF2 10DE 1613 |
+| NVIDIA T400E                                    | 1FF2 10DE 18FF |
 | NVIDIA T400 4GB                                 | 1FF2 17AA 1613 |
+| NVIDIA T400E                                    | 1FF2 17AA 18FF |
 | Quadro T1000                                    | 1FF9           |
 | NVIDIA A100-SXM4-40GB                           | 20B0           |
 | NVIDIA A100-PG509-200                           | 20B0 10DE 1450 |
 | NVIDIA A100-SXM4-80GB                           | 20B2 10DE 1463 |
 | NVIDIA A100-SXM4-80GB                           | 20B2 10DE 147F |
-| NVIDIA PG506-242                                | 20B3 10DE 14A7 |
-| NVIDIA PG506-243                                | 20B3 10DE 14A8 |
+| NVIDIA A100-SXM4-80GB                           | 20B2 10DE 1622 |
+| NVIDIA A100-SXM4-80GB                           | 20B2 10DE 1623 |
+| NVIDIA PG509-210                                | 20B2 10DE 1625 |
+| NVIDIA A100-SXM-64GB                            | 20B3 10DE 14A7 |
+| NVIDIA A100-SXM-64GB                            | 20B3 10DE 14A8 |
 | NVIDIA A100 80GB PCIe                           | 20B5 10DE 1533 |
 | NVIDIA A100 80GB PCIe                           | 20B5 10DE 1642 |
 | NVIDIA PG506-232                                | 20B6 10DE 1492 |
 | NVIDIA A30                                      | 20B7 10DE 1532 |
+| NVIDIA A30                                      | 20B7 10DE 1804 |
+| NVIDIA A30                                      | 20B7 10DE 1852 |
+| NVIDIA A800-SXM4-40GB                           | 20BD 10DE 17F4 |
 | NVIDIA A100-PCIE-40GB                           | 20F1 10DE 145F |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 179B |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 179C |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 179D |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 179E |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 179F |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 17A0 |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 17A1 |
+| NVIDIA A800-SXM4-80GB                           | 20F3 10DE 17A2 |
+| NVIDIA A800 80GB PCIe                           | 20F5 10DE 1799 |
+| NVIDIA A800 80GB PCIe LC                        | 20F5 10DE 179A |
+| NVIDIA A800 40GB Active                         | 20F6 1028 180A |
+| NVIDIA A800 40GB Active                         | 20F6 103C 180A |
+| NVIDIA A800 40GB Active                         | 20F6 10DE 180A |
+| NVIDIA A800 40GB Active                         | 20F6 17AA 180A |
+| NVIDIA AX800                                    | 20FD 10DE 17F8 |
 | NVIDIA GeForce GTX 1660 Ti                      | 2182           |
 | NVIDIA GeForce GTX 1660                         | 2184           |
 | NVIDIA GeForce GTX 1650 SUPER                   | 2187           |
 | NVIDIA GeForce GTX 1650                         | 2188           |
+| NVIDIA CMP 30HX                                 | 2189           |
 | NVIDIA GeForce GTX 1660 Ti                      | 2191           |
 | NVIDIA GeForce GTX 1660 Ti with Max-Q Design    | 2191 1028 0949 |
 | NVIDIA GeForce GTX 1660 Ti with Max-Q Design    | 2191 103C 85FB |
@@ -683,6 +723,7 @@ Subsystem Device ID.
 | NVIDIA GeForce RTX 3090 Ti                      | 2203           |
 | NVIDIA GeForce RTX 3090                         | 2204           |
 | NVIDIA GeForce RTX 3080                         | 2206           |
+| NVIDIA GeForce RTX 3070 Ti                      | 2207           |
 | NVIDIA GeForce RTX 3080 Ti                      | 2208           |
 | NVIDIA GeForce RTX 3080                         | 220A           |
 | NVIDIA CMP 90HX                                 | 220D           |
@@ -707,6 +748,24 @@ Subsystem Device ID.
 | NVIDIA A10                                      | 2236 10DE 1482 |
 | NVIDIA A10G                                     | 2237 10DE 152F |
 | NVIDIA A10M                                     | 2238 10DE 1677 |
+| NVIDIA H100 NVL                                 | 2321 10DE 1839 |
+| NVIDIA H800 PCIe                                | 2322 10DE 17A4 |
+| NVIDIA H800                                     | 2324 10DE 17A6 |
+| NVIDIA H800                                     | 2324 10DE 17A8 |
+| NVIDIA H20                                      | 2329 10DE 198B |
+| NVIDIA H20                                      | 2329 10DE 198C |
+| NVIDIA H100 80GB HBM3                           | 2330 10DE 16C0 |
+| NVIDIA H100 80GB HBM3                           | 2330 10DE 16C1 |
+| NVIDIA H100 PCIe                                | 2331 10DE 1626 |
+| NVIDIA H200                                     | 2335 10DE 18BE |
+| NVIDIA H200                                     | 2335 10DE 18BF |
+| NVIDIA H100                                     | 2339 10DE 17FC |
+| NVIDIA H800 NVL                                 | 233A 10DE 183A |
+| NVIDIA H200 NVL                                 | 233B 10DE 1996 |
+| NVIDIA GH200 120GB                              | 2342 10DE 16EB |
+| NVIDIA GH200 120GB                              | 2342 10DE 1805 |
+| NVIDIA GH200 480GB                              | 2342 10DE 1809 |
+| NVIDIA GH200 144G HBM3e                         | 2348 10DE 18D2 |
 | NVIDIA GeForce RTX 3060 Ti                      | 2414           |
 | NVIDIA GeForce RTX 3080 Ti Laptop GPU           | 2420           |
 | NVIDIA RTX A5500 Laptop GPU                     | 2438           |
@@ -734,6 +793,8 @@ Subsystem Device ID.
 | NVIDIA RTX A3000 12GB Laptop GPU                | 24B9           |
 | NVIDIA RTX A4500 Laptop GPU                     | 24BA           |
 | NVIDIA RTX A3000 12GB Laptop GPU                | 24BB           |
+| NVIDIA GeForce RTX 3060                         | 24C7           |
+| NVIDIA GeForce RTX 3060 Ti                      | 24C9           |
 | NVIDIA GeForce RTX 3080 Laptop GPU              | 24DC           |
 | NVIDIA GeForce RTX 3070 Laptop GPU              | 24DD           |
 | NVIDIA GeForce RTX 3070 Ti Laptop GPU           | 24E0           |
@@ -743,17 +804,21 @@ Subsystem Device ID.
 | NVIDIA GeForce RTX 3050                         | 2507           |
 | NVIDIA GeForce RTX 3050 OEM                     | 2508           |
 | NVIDIA GeForce RTX 3060 Laptop GPU              | 2520           |
+| NVIDIA GeForce RTX 3060 Laptop GPU              | 2521           |
 | NVIDIA GeForce RTX 3050 Ti Laptop GPU           | 2523           |
 | NVIDIA RTX A2000                                | 2531 1028 151D |
 | NVIDIA RTX A2000                                | 2531 103C 151D |
 | NVIDIA RTX A2000                                | 2531 10DE 151D |
 | NVIDIA RTX A2000                                | 2531 17AA 151D |
+| NVIDIA GeForce RTX 3060                         | 2544           |
 | NVIDIA GeForce RTX 3060 Laptop GPU              | 2560           |
 | NVIDIA GeForce RTX 3050 Ti Laptop GPU           | 2563           |
 | NVIDIA RTX A2000 12GB                           | 2571 1028 1611 |
 | NVIDIA RTX A2000 12GB                           | 2571 103C 1611 |
 | NVIDIA RTX A2000 12GB                           | 2571 10DE 1611 |
 | NVIDIA RTX A2000 12GB                           | 2571 17AA 1611 |
+| NVIDIA GeForce RTX 3050                         | 2582           |
+| NVIDIA GeForce RTX 3050                         | 2584           |
 | NVIDIA GeForce RTX 3050 Ti Laptop GPU           | 25A0           |
 | NVIDIA GeForce RTX 3050Ti Laptop GPU            | 25A0 103C 8928 |
 | NVIDIA GeForce RTX 3050Ti Laptop GPU            | 25A0 103C 89F9 |
@@ -766,14 +831,107 @@ Subsystem Device ID.
 | NVIDIA GeForce RTX 2050                         | 25A7           |
 | NVIDIA GeForce RTX 2050                         | 25A9           |
 | NVIDIA GeForce MX570 A                          | 25AA           |
+| NVIDIA GeForce RTX 3050 4GB Laptop GPU          | 25AB           |
+| NVIDIA GeForce RTX 3050 6GB Laptop GPU          | 25AC           |
+| NVIDIA GeForce RTX 2050                         | 25AD           |
+| NVIDIA RTX A1000                                | 25B0 1028 1878 |
+| NVIDIA RTX A1000                                | 25B0 103C 1878 |
+| NVIDIA RTX A1000                                | 25B0 10DE 1878 |
+| NVIDIA RTX A1000                                | 25B0 17AA 1878 |
+| NVIDIA RTX A400                                 | 25B2 1028 1879 |
+| NVIDIA RTX A400                                 | 25B2 103C 1879 |
+| NVIDIA RTX A400                                 | 25B2 10DE 1879 |
+| NVIDIA RTX A400                                 | 25B2 17AA 1879 |
 | NVIDIA A16                                      | 25B6 10DE 14A9 |
 | NVIDIA A2                                       | 25B6 10DE 157E |
 | NVIDIA RTX A2000 Laptop GPU                     | 25B8           |
 | NVIDIA RTX A1000 Laptop GPU                     | 25B9           |
 | NVIDIA RTX A2000 8GB Laptop GPU                 | 25BA           |
 | NVIDIA RTX A500 Laptop GPU                      | 25BB           |
+| NVIDIA RTX A1000 6GB Laptop GPU                 | 25BC           |
+| NVIDIA RTX A500 Laptop GPU                      | 25BD           |
 | NVIDIA GeForce RTX 3050 Ti Laptop GPU           | 25E0           |
 | NVIDIA GeForce RTX 3050 Laptop GPU              | 25E2           |
 | NVIDIA GeForce RTX 3050 Laptop GPU              | 25E5           |
+| NVIDIA GeForce RTX 3050 6GB Laptop GPU          | 25EC           |
+| NVIDIA GeForce RTX 2050                         | 25ED           |
 | NVIDIA RTX A1000 Embedded GPU                   | 25F9           |
 | NVIDIA RTX A2000 Embedded GPU                   | 25FA           |
+| NVIDIA RTX A500 Embedded GPU                    | 25FB           |
+| NVIDIA GeForce RTX 4090                         | 2684           |
+| NVIDIA GeForce RTX 4090 D                       | 2685           |
+| NVIDIA GeForce RTX 4070 Ti SUPER                | 2689           |
+| NVIDIA RTX 6000 Ada Generation                  | 26B1 1028 16A1 |
+| NVIDIA RTX 6000 Ada Generation                  | 26B1 103C 16A1 |
+| NVIDIA RTX 6000 Ada Generation                  | 26B1 10DE 16A1 |
+| NVIDIA RTX 6000 Ada Generation                  | 26B1 17AA 16A1 |
+| NVIDIA RTX 5000 Ada Generation                  | 26B2 1028 17FA |
+| NVIDIA RTX 5000 Ada Generation                  | 26B2 103C 17FA |
+| NVIDIA RTX 5000 Ada Generation                  | 26B2 10DE 17FA |
+| NVIDIA RTX 5000 Ada Generation                  | 26B2 17AA 17FA |
+| NVIDIA RTX 5880 Ada Generation                  | 26B3 1028 1934 |
+| NVIDIA RTX 5880 Ada Generation                  | 26B3 103C 1934 |
+| NVIDIA RTX 5880 Ada Generation                  | 26B3 10DE 1934 |
+| NVIDIA RTX 5880 Ada Generation                  | 26B3 17AA 1934 |
+| NVIDIA L40                                      | 26B5 10DE 169D |
+| NVIDIA L40                                      | 26B5 10DE 17DA |
+| NVIDIA L40S                                     | 26B9 10DE 1851 |
+| NVIDIA L40S                                     | 26B9 10DE 18CF |
+| NVIDIA L20                                      | 26BA 10DE 1957 |
+| NVIDIA L20                                      | 26BA 10DE 1990 |
+| NVIDIA GeForce RTX 4080 SUPER                   | 2702           |
+| NVIDIA GeForce RTX 4080                         | 2704           |
+| NVIDIA GeForce RTX 4070 Ti SUPER                | 2705           |
+| NVIDIA GeForce RTX 4070                         | 2709           |
+| NVIDIA GeForce RTX 4090 Laptop GPU              | 2717           |
+| NVIDIA RTX 5000 Ada Generation Laptop GPU       | 2730           |
+| NVIDIA GeForce RTX 4090 Laptop GPU              | 2757           |
+| NVIDIA RTX 5000 Ada Generation Embedded GPU     | 2770           |
+| NVIDIA GeForce RTX 4070 Ti                      | 2782           |
+| NVIDIA GeForce RTX 4070 SUPER                   | 2783           |
+| NVIDIA GeForce RTX 4070                         | 2786           |
+| NVIDIA GeForce RTX 4060 Ti                      | 2788           |
+| NVIDIA GeForce RTX 4080 Laptop GPU              | 27A0           |
+| NVIDIA RTX 4000 SFF Ada Generation              | 27B0 1028 16FA |
+| NVIDIA RTX 4000 SFF Ada Generation              | 27B0 103C 16FA |
+| NVIDIA RTX 4000 SFF Ada Generation              | 27B0 10DE 16FA |
+| NVIDIA RTX 4000 SFF Ada Generation              | 27B0 17AA 16FA |
+| NVIDIA RTX 4500 Ada Generation                  | 27B1 1028 180C |
+| NVIDIA RTX 4500 Ada Generation                  | 27B1 103C 180C |
+| NVIDIA RTX 4500 Ada Generation                  | 27B1 10DE 180C |
+| NVIDIA RTX 4500 Ada Generation                  | 27B1 17AA 180C |
+| NVIDIA RTX 4000 Ada Generation                  | 27B2 1028 181B |
+| NVIDIA RTX 4000 Ada Generation                  | 27B2 103C 181B |
+| NVIDIA RTX 4000 Ada Generation                  | 27B2 10DE 181B |
+| NVIDIA RTX 4000 Ada Generation                  | 27B2 17AA 181B |
+| NVIDIA L2                                       | 27B6 10DE 1933 |
+| NVIDIA L4                                       | 27B8 10DE 16CA |
+| NVIDIA L4                                       | 27B8 10DE 16EE |
+| NVIDIA RTX 4000 Ada Generation Laptop GPU       | 27BA           |
+| NVIDIA RTX 3500 Ada Generation Laptop GPU       | 27BB           |
+| NVIDIA GeForce RTX 4080 Laptop GPU              | 27E0           |
+| NVIDIA RTX 3500 Ada Generation Embedded GPU     | 27FB           |
+| NVIDIA GeForce RTX 4060 Ti                      | 2803           |
+| NVIDIA GeForce RTX 4060 Ti                      | 2805           |
+| NVIDIA GeForce RTX 4060                         | 2808           |
+| NVIDIA GeForce RTX 4070 Laptop GPU              | 2820           |
+| NVIDIA GeForce RTX 3050 A Laptop GPU            | 2822           |
+| NVIDIA RTX 3000 Ada Generation Laptop GPU       | 2838           |
+| NVIDIA GeForce RTX 4070 Laptop GPU              | 2860           |
+| NVIDIA GeForce RTX 4060                         | 2882           |
+| NVIDIA GeForce RTX 4060 Laptop GPU              | 28A0           |
+| NVIDIA GeForce RTX 4050 Laptop GPU              | 28A1           |
+| NVIDIA RTX 2000 Ada Generation                  | 28B0 1028 1870 |
+| NVIDIA RTX 2000 Ada Generation                  | 28B0 103C 1870 |
+| NVIDIA RTX 2000E Ada Generation                 | 28B0 103C 1871 |
+| NVIDIA RTX 2000 Ada Generation                  | 28B0 10DE 1870 |
+| NVIDIA RTX 2000E Ada Generation                 | 28B0 10DE 1871 |
+| NVIDIA RTX 2000 Ada Generation                  | 28B0 17AA 1870 |
+| NVIDIA RTX 2000E Ada Generation                 | 28B0 17AA 1871 |
+| NVIDIA RTX 2000 Ada Generation Laptop GPU       | 28B8           |
+| NVIDIA RTX 1000 Ada Generation Laptop GPU       | 28B9           |
+| NVIDIA RTX 500 Ada Generation Laptop GPU        | 28BA           |
+| NVIDIA RTX 500 Ada Generation Laptop GPU        | 28BB           |
+| NVIDIA GeForce RTX 4060 Laptop GPU              | 28E0           |
+| NVIDIA GeForce RTX 4050 Laptop GPU              | 28E1           |
+| NVIDIA RTX 2000 Ada Generation Embedded GPU     | 28F8           |
--- a/kernel-open/Kbuild
+++ b/kernel-open/Kbuild
@@ -70,9 +70,25 @@ $(foreach _module, $(NV_KERNEL_MODULES), \

 EXTRA_CFLAGS += -I$(src)/common/inc
 EXTRA_CFLAGS += -I$(src)
-EXTRA_CFLAGS += -Wall -MD $(DEFINES) $(INCLUDES) -Wno-cast-qual -Wno-error -Wno-format-extra-args
+EXTRA_CFLAGS += -Wall $(DEFINES) $(INCLUDES) -Wno-cast-qual -Wno-format-extra-args
 EXTRA_CFLAGS += -D__KERNEL__ -DMODULE -DNVRM
-EXTRA_CFLAGS += -DNV_VERSION_STRING=\"515.49.14\"
+EXTRA_CFLAGS += -DNV_VERSION_STRING=\"565.57.01\"
+
+ifneq ($(SYSSRCHOST1X),)
+ EXTRA_CFLAGS += -I$(SYSSRCHOST1X)
+endif
+
+# Some Android kernels prohibit driver use of filesystem functions like
+# filp_open() and kernel_read(). Disable the NV_FILESYSTEM_ACCESS_AVAILABLE
+# functionality that uses those functions when building for Android.
+
+PLATFORM_IS_ANDROID ?= 0
+
+ifeq ($(PLATFORM_IS_ANDROID),1)
+ EXTRA_CFLAGS += -DNV_FILESYSTEM_ACCESS_AVAILABLE=0
+else
+ EXTRA_CFLAGS += -DNV_FILESYSTEM_ACCESS_AVAILABLE=1
+endif

 EXTRA_CFLAGS += -Wno-unused-function

@@ -87,7 +103,7 @@ ifeq ($(ARCH),arm64)
 endif

 ifeq ($(NV_BUILD_TYPE),debug)
- EXTRA_CFLAGS += -g -gsplit-dwarf
+ EXTRA_CFLAGS += -g
 endif

 EXTRA_CFLAGS += -ffreestanding
@@ -102,7 +118,7 @@ ifeq ($(ARCH),x86_64)
 endif

 ifeq ($(ARCH),powerpc)
- EXTRA_CFLAGS += -mlittle-endian -mno-strict-align -mno-altivec
+ EXTRA_CFLAGS += -mlittle-endian -mno-strict-align
 endif

 EXTRA_CFLAGS += -DNV_UVM_ENABLE
@@ -118,6 +134,16 @@ ifneq ($(wildcard /proc/sgi_uv),)
 EXTRA_CFLAGS += -DNV_CONFIG_X86_UV
 endif

+ifdef VGX_FORCE_VFIO_PCI_CORE
+ EXTRA_CFLAGS += -DNV_VGPU_FORCE_VFIO_PCI_CORE
+endif
+
+WARNINGS_AS_ERRORS ?=
+ifeq ($(WARNINGS_AS_ERRORS),1)
+ ccflags-y += -Werror
+else
+ ccflags-y += -Wno-error
+endif

 #
 # The conftest.sh script tests various aspects of the target kernel.
@@ -144,6 +170,10 @@ NV_CONFTEST_CMD := /bin/sh $(NV_CONFTEST_SCRIPT) \
 NV_CFLAGS_FROM_CONFTEST := $(shell $(NV_CONFTEST_CMD) build_cflags)

 NV_CONFTEST_CFLAGS = $(NV_CFLAGS_FROM_CONFTEST) $(EXTRA_CFLAGS) -fno-pie
+NV_CONFTEST_CFLAGS += $(call cc-disable-warning,pointer-sign)
+NV_CONFTEST_CFLAGS += $(call cc-option,-fshort-wchar,)
+NV_CONFTEST_CFLAGS += $(call cc-option,-Werror=incompatible-pointer-types,)
+NV_CONFTEST_CFLAGS += -Wno-error

 NV_CONFTEST_COMPILE_TEST_HEADERS := $(obj)/conftest/macros.h
 NV_CONFTEST_COMPILE_TEST_HEADERS += $(obj)/conftest/functions.h
@@ -203,9 +233,25 @@ $(obj)/conftest/patches.h: $(NV_CONFTEST_SCRIPT)
 	@mkdir -p $(obj)/conftest
 	@$(NV_CONFTEST_CMD) patch_check > $@

-$(obj)/conftest/headers.h: $(NV_CONFTEST_SCRIPT)
-	@mkdir -p $(obj)/conftest
-	@$(NV_CONFTEST_CMD) test_kernel_headers '$(NV_CONFTEST_CFLAGS)' > $@
+include $(src)/header-presence-tests.mk
+
+# Filename to store the define for the header in $(1); this is only consumed by
+# the rule below that concatenates all of these together.
+NV_HEADER_PRESENCE_PART = $(addprefix $(obj)/conftest/header_presence/,$(addsuffix .part,$(1)))
+
+# Define a rule to check the header $(1).
+define NV_HEADER_PRESENCE_CHECK
+ $$(call NV_HEADER_PRESENCE_PART,$(1)): $$(NV_CONFTEST_SCRIPT) $(obj)/conftest/uts_release
+	@mkdir -p $$(dir $$@)
+	@$$(NV_CONFTEST_CMD) test_kernel_header '$$(NV_CONFTEST_CFLAGS)' '$(1)' > $$@
+endef
+
+# Evaluate the rule above for each header in the list.
+$(foreach header,$(NV_HEADER_PRESENCE_TESTS),$(eval $(call NV_HEADER_PRESENCE_CHECK,$(header))))
+
+# Concatenate all of the parts into headers.h.
+$(obj)/conftest/headers.h: $(call NV_HEADER_PRESENCE_PART,$(NV_HEADER_PRESENCE_TESTS))
+	@cat $^ > $@

 clean-dirs := $(obj)/conftest

--- a/kernel-open/Makefile
+++ b/kernel-open/Makefile
@@ -28,7 +28,7 @@ else
  else
    KERNEL_UNAME ?= $(shell uname -r)
    KERNEL_MODLIB := /lib/modules/$(KERNEL_UNAME)
-    KERNEL_SOURCES := $(shell test -d $(KERNEL_MODLIB)/source && echo $(KERNEL_MODLIB)/source || echo $(KERNEL_MODLIB)/build)
+    KERNEL_SOURCES := $(shell ((test -d $(KERNEL_MODLIB)/source && echo $(KERNEL_MODLIB)/source) || (test -d $(KERNEL_MODLIB)/build/source && echo $(KERNEL_MODLIB)/build/source)) || echo $(KERNEL_MODLIB)/build)
  endif

  KERNEL_OUTPUT := $(KERNEL_SOURCES)
@@ -42,7 +42,11 @@ else
  else
    KERNEL_UNAME ?= $(shell uname -r)
    KERNEL_MODLIB := /lib/modules/$(KERNEL_UNAME)
-    ifeq ($(KERNEL_SOURCES), $(KERNEL_MODLIB)/source)
+    # $(filter patter...,text) - Returns all whitespace-separated words in text that
+    # do match any of the pattern words, removing any words that do not match.
+    # Set the KERNEL_OUTPUT only if either $(KERNEL_MODLIB)/source or
+    # $(KERNEL_MODLIB)/build/source path matches the KERNEL_SOURCES.
+    ifneq ($(filter $(KERNEL_SOURCES),$(KERNEL_MODLIB)/source $(KERNEL_MODLIB)/build/source),)
      KERNEL_OUTPUT := $(KERNEL_MODLIB)/build
      KBUILD_PARAMS := KBUILD_OUTPUT=$(KERNEL_OUTPUT)
    endif
@@ -57,12 +61,15 @@ else
      -e 's/armv[0-7]\w\+/arm/' \
      -e 's/aarch64/arm64/' \
      -e 's/ppc64le/powerpc/' \
+      -e 's/riscv64/riscv/' \
    )
  endif

  NV_KERNEL_MODULES ?= $(wildcard nvidia nvidia-uvm nvidia-vgpu-vfio nvidia-modeset nvidia-drm nvidia-peermem)
  NV_KERNEL_MODULES := $(filter-out $(NV_EXCLUDE_KERNEL_MODULES), \
                                    $(NV_KERNEL_MODULES))
+  INSTALL_MOD_DIR ?= kernel/drivers/video
+
  NV_VERBOSE ?=
  SPECTRE_V2_RETPOLINE ?= 0

@@ -74,7 +81,7 @@ else
  KBUILD_PARAMS += NV_KERNEL_SOURCES=$(KERNEL_SOURCES)
  KBUILD_PARAMS += NV_KERNEL_OUTPUT=$(KERNEL_OUTPUT)
  KBUILD_PARAMS += NV_KERNEL_MODULES="$(NV_KERNEL_MODULES)"
-  KBUILD_PARAMS += INSTALL_MOD_DIR=kernel/drivers/video
+  KBUILD_PARAMS += INSTALL_MOD_DIR="$(INSTALL_MOD_DIR)"
  KBUILD_PARAMS += NV_SPECTRE_V2=$(SPECTRE_V2_RETPOLINE)

  .PHONY: modules module clean clean_conftest modules_install
--- a/kernel-open/common/inc/cpuopsys.h
+++ b/kernel-open/common/inc/cpuopsys.h
@@ -101,13 +101,6 @@
 #   define NV_ANDROID
 #endif

-
-
-
-
-
-
-
 #if defined(DceCore) && !defined(NV_DCECORE)
 #    define NV_DCECORE
 #endif
@@ -249,7 +242,7 @@
 #endif

 /* For verification-only features not intended to be included in normal drivers */
-#if (defined(NV_MODS) || defined(NV_GSP_MODS)) && defined(DEBUG) && !defined(DISABLE_VERIF_FEATURES)
+#if defined(ENABLE_VERIF_FEATURES)
 #define NV_VERIF_FEATURES
 #endif

@@ -283,12 +276,6 @@
 #define NV_IS_MODS 0
 #endif

-#if defined(NV_GSP_MODS)
-#define NV_IS_GSP_MODS 1
-#else
-#define NV_IS_GSP_MODS 0
-#endif
-
 #if defined(NV_WINDOWS)
 #define NVOS_IS_WINDOWS 1
 #else
@@ -355,15 +342,6 @@
 #define NVOS_IS_INTEGRITY 0
 #endif

-
-
-
-
-
-
-
-
-
 #if defined(NVCPU_X86)
 #define NVCPU_IS_X86 1
 #else
--- a/kernel-open/common/inc/nv-chardev-numbers.h
+++ b/kernel-open/common/inc/nv-chardev-numbers.h
@@ -0,0 +1,43 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+
+#ifndef _NV_CHARDEV_NUMBERS_H_
+#define _NV_CHARDEV_NUMBERS_H_
+
+// NVIDIA's reserved major character device number (Linux).
+#define NV_MAJOR_DEVICE_NUMBER  195
+
+// Minor numbers 0 to 247 reserved for regular devices
+#define NV_MINOR_DEVICE_NUMBER_REGULAR_MAX         247
+
+// Minor numbers 248 to 253 currently unused
+
+// Minor number 254 reserved for the modeset device (provided by NVKMS)
+#define NV_MINOR_DEVICE_NUMBER_MODESET_DEVICE      254
+
+// Minor number 255 reserved for the control device
+#define NV_MINOR_DEVICE_NUMBER_CONTROL_DEVICE      255
+
+#endif  // _NV_CHARDEV_NUMBERS_H_
+
--- a/kernel-open/common/inc/nv-firmware-registry.h
+++ b/kernel-open/common/inc/nv-firmware-registry.h
@@ -0,0 +1,83 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+//
+// This file holds GPU firmware related registry key definitions that are
+// shared between Windows and Unix
+//
+
+#ifndef NV_FIRMWARE_REGISTRY_H
+#define NV_FIRMWARE_REGISTRY_H
+
+//
+// Registry key that when enabled, will enable use of GPU firmware.
+//
+// Possible mode values:
+//  0 - Do not enable GPU firmware
+//  1 - Enable GPU firmware
+//  2 - (Default) Use the default enablement policy for GPU firmware
+//
+// Setting this to anything other than 2 will alter driver firmware-
+// enablement policies, possibly disabling GPU firmware where it would
+// have otherwise been enabled by default.
+//
+// Policy bits:
+//
+// POLICY_ALLOW_FALLBACK:
+//  As the normal behavior is to fail GPU initialization if this registry
+//  entry is set in such a way that results in an invalid configuration, if
+//  instead the user would like the driver to automatically try to fallback
+//  to initializing the failing GPU with firmware disabled, then this bit can
+//  be set (ex: 0x11 means try to enable GPU firmware but fall back if needed).
+//  Note that this can result in a mixed mode configuration (ex: GPU0 has
+//  firmware enabled, but GPU1 does not).
+//
+#define NV_REG_STR_ENABLE_GPU_FIRMWARE                   "EnableGpuFirmware"
+
+#define NV_REG_ENABLE_GPU_FIRMWARE_MODE_MASK              0x0000000F
+#define NV_REG_ENABLE_GPU_FIRMWARE_MODE_DISABLED          0x00000000
+#define NV_REG_ENABLE_GPU_FIRMWARE_MODE_ENABLED           0x00000001
+#define NV_REG_ENABLE_GPU_FIRMWARE_MODE_DEFAULT           0x00000002
+
+#define NV_REG_ENABLE_GPU_FIRMWARE_POLICY_MASK            0x000000F0
+#define NV_REG_ENABLE_GPU_FIRMWARE_POLICY_ALLOW_FALLBACK  0x00000010
+
+#define NV_REG_ENABLE_GPU_FIRMWARE_DEFAULT_VALUE          0x00000012
+
+//
+// Registry key that when enabled,  will send GPU firmware logs
+// to the system log, when possible.
+//
+// Possible values:
+//  0 - Do not send GPU firmware logs to the system log
+//  1 - Enable sending of GPU firmware logs to the system log
+//  2 - (Default) Enable sending of GPU firmware logs to the system log for
+//      the debug kernel driver build only
+//
+#define NV_REG_STR_ENABLE_GPU_FIRMWARE_LOGS                "EnableGpuFirmwareLogs"
+
+#define NV_REG_ENABLE_GPU_FIRMWARE_LOGS_DISABLE            0x00000000
+#define NV_REG_ENABLE_GPU_FIRMWARE_LOGS_ENABLE             0x00000001
+#define NV_REG_ENABLE_GPU_FIRMWARE_LOGS_ENABLE_ON_DEBUG    0x00000002
+
+#endif  // NV_FIRMWARE_REGISTRY_H
--- a/kernel-open/common/inc/nv-firmware.h
+++ b/kernel-open/common/inc/nv-firmware.h
@@ -0,0 +1,136 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef NV_FIRMWARE_H
+#define NV_FIRMWARE_H
+
+
+
+#include <nvtypes.h>
+#include <nvmisc.h>
+
+typedef enum
+{
+    NV_FIRMWARE_TYPE_GSP,
+    NV_FIRMWARE_TYPE_GSP_LOG
+} nv_firmware_type_t;
+
+typedef enum
+{
+    NV_FIRMWARE_CHIP_FAMILY_NULL = 0,
+    NV_FIRMWARE_CHIP_FAMILY_TU10X = 1,
+    NV_FIRMWARE_CHIP_FAMILY_TU11X = 2,
+    NV_FIRMWARE_CHIP_FAMILY_GA100 = 3,
+    NV_FIRMWARE_CHIP_FAMILY_GA10X = 4,
+    NV_FIRMWARE_CHIP_FAMILY_AD10X = 5,
+    NV_FIRMWARE_CHIP_FAMILY_GH100 = 6,
+    NV_FIRMWARE_CHIP_FAMILY_GB10X = 8,
+    NV_FIRMWARE_CHIP_FAMILY_END,
+} nv_firmware_chip_family_t;
+
+static inline const char *nv_firmware_chip_family_to_string(
+    nv_firmware_chip_family_t fw_chip_family
+)
+{
+    switch (fw_chip_family) {
+        case NV_FIRMWARE_CHIP_FAMILY_GB10X: return "gb10x";
+        case NV_FIRMWARE_CHIP_FAMILY_GH100: return "gh100";
+        case NV_FIRMWARE_CHIP_FAMILY_AD10X: return "ad10x";
+        case NV_FIRMWARE_CHIP_FAMILY_GA10X: return "ga10x";
+        case NV_FIRMWARE_CHIP_FAMILY_GA100: return "ga100";
+        case NV_FIRMWARE_CHIP_FAMILY_TU11X: return "tu11x";
+        case NV_FIRMWARE_CHIP_FAMILY_TU10X: return "tu10x";
+
+        case NV_FIRMWARE_CHIP_FAMILY_END:  // fall through
+        case NV_FIRMWARE_CHIP_FAMILY_NULL:
+            return NULL;
+    }
+    return NULL;
+}
+
+// The includer may optionally define
+// NV_FIRMWARE_FOR_NAME(name)
+// to return a platform-defined string for a given a gsp_* or gsp_log_* name.
+//
+// The function nv_firmware_for_chip_family will then be available.
+#if defined(NV_FIRMWARE_FOR_NAME)
+static inline const char *nv_firmware_for_chip_family(
+    nv_firmware_type_t fw_type,
+    nv_firmware_chip_family_t fw_chip_family
+)
+{
+    if (fw_type == NV_FIRMWARE_TYPE_GSP)
+    {
+        switch (fw_chip_family)
+        {
+            case NV_FIRMWARE_CHIP_FAMILY_GB10X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_GH100:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_AD10X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_GA10X:
+                return NV_FIRMWARE_FOR_NAME("gsp_ga10x");
+
+            case NV_FIRMWARE_CHIP_FAMILY_GA100:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_TU11X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_TU10X:
+                return NV_FIRMWARE_FOR_NAME("gsp_tu10x");
+
+            case NV_FIRMWARE_CHIP_FAMILY_END:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_NULL:
+                return "";
+        }
+    }
+    else if (fw_type == NV_FIRMWARE_TYPE_GSP_LOG)
+    {
+        switch (fw_chip_family)
+        {
+            case NV_FIRMWARE_CHIP_FAMILY_GB10X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_GH100:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_AD10X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_GA10X:
+                return NV_FIRMWARE_FOR_NAME("gsp_log_ga10x");
+
+            case NV_FIRMWARE_CHIP_FAMILY_GA100:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_TU11X:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_TU10X:
+                return NV_FIRMWARE_FOR_NAME("gsp_log_tu10x");
+
+            case NV_FIRMWARE_CHIP_FAMILY_END:  // fall through
+            case NV_FIRMWARE_CHIP_FAMILY_NULL:
+                return "";
+        }
+    }
+
+    return "";
+}
+#endif  // defined(NV_FIRMWARE_FOR_NAME)
+
+// The includer may optionally define
+// NV_FIRMWARE_DECLARE_GSP(name)
+// which will then be invoked (at the top-level) for each
+// gsp_* (but not gsp_log_*)
+#if defined(NV_FIRMWARE_DECLARE_GSP)
+NV_FIRMWARE_DECLARE_GSP("gsp_ga10x")
+NV_FIRMWARE_DECLARE_GSP("gsp_tu10x")
+#endif  // defined(NV_FIRMWARE_DECLARE_GSP)
+
+#endif  // NV_FIRMWARE_DECLARE_GSP
--- a/kernel-open/common/inc/nv-hash.h
+++ b/kernel-open/common/inc/nv-hash.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2020 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2020-22 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -91,6 +91,6 @@ static inline void _nv_hash_init(struct hlist_head *ht, unsigned int sz)
 * @key: the key of the objects to iterate over
 */
 #define nv_hash_for_each_possible(name, obj, member, key) \
-    nv_hlist_for_each_entry(obj, &name[NV_HASH_MIN(key, NV_HASH_BITS(name))], member)
+    hlist_for_each_entry(obj, &name[NV_HASH_MIN(key, NV_HASH_BITS(name))], member)

 #endif // __NV_HASH_H__
--- a/kernel-open/common/inc/nv-hypervisor.h
+++ b/kernel-open/common/inc/nv-hypervisor.h
@@ -27,24 +27,21 @@
 #include <nv-kernel-interface-api.h>

 // Enums for supported hypervisor types.
-// New hypervisor type should be added before OS_HYPERVISOR_CUSTOM_FORCED
+// New hypervisor type should be added before OS_HYPERVISOR_UNKNOWN
 typedef enum _HYPERVISOR_TYPE
 {
    OS_HYPERVISOR_XEN = 0,
    OS_HYPERVISOR_VMWARE,
    OS_HYPERVISOR_HYPERV,
    OS_HYPERVISOR_KVM,
-    OS_HYPERVISOR_PARALLELS,
-    OS_HYPERVISOR_CUSTOM_FORCED,
    OS_HYPERVISOR_UNKNOWN
 } HYPERVISOR_TYPE;

-#define CMD_VGPU_VFIO_WAKE_WAIT_QUEUE         0
-#define CMD_VGPU_VFIO_INJECT_INTERRUPT        1
-#define CMD_VGPU_VFIO_REGISTER_MDEV           2
-#define CMD_VGPU_VFIO_PRESENT                 3
+#define CMD_VFIO_WAKE_REMOVE_GPU              1
+#define CMD_VGPU_VFIO_PRESENT                 2
+#define CMD_VFIO_PCI_CORE_PRESENT             3

-#define MAX_VF_COUNT_PER_GPU 64
+#define MAX_VF_COUNT_PER_GPU                  64

 typedef enum _VGPU_TYPE_INFO
 {
@@ -55,16 +52,11 @@ typedef enum _VGPU_TYPE_INFO

 typedef struct
 {
-    void  *vgpuVfioRef;
-    void  *waitQueue;
    void  *nv;
-    NvU32 *vgpuTypeIds;
-    NvU32  numVgpuTypes;
-    NvU32  domain;
-    NvU8   bus;
-    NvU8   slot;
-    NvU8   function;
-    NvBool is_virtfn;
+    NvU32 domain;
+    NvU32 bus;
+    NvU32 device;
+    NvU32 return_status;
 } vgpu_vfio_info;

 typedef struct
@@ -92,30 +84,6 @@ typedef enum VGPU_DEVICE_STATE_E
    NV_VGPU_DEV_IN_USE = 2
 } VGPU_DEVICE_STATE;

-typedef enum _VMBUS_CMD_TYPE
-{
-    VMBUS_CMD_TYPE_INVALID    = 0,
-    VMBUS_CMD_TYPE_SETUP      = 1,
-    VMBUS_CMD_TYPE_SENDPACKET = 2,
-    VMBUS_CMD_TYPE_CLEANUP    = 3,
-} VMBUS_CMD_TYPE;
-
-typedef struct
-{
-    NvU32 request_id;
-    NvU32 page_count;
-    NvU64 *pPfns;
-    void *buffer;
-    NvU32 bufferlen;
-} vmbus_send_packet_cmd_params;
-
-
-typedef struct
-{
-    NvU32 override_sint;
-    NvU8 *nv_guid;
-} vmbus_setup_cmd_params;
-
 /*
 * Function prototypes
 */
--- a/kernel-open/common/inc/nv-ioctl-numa.h
+++ b/kernel-open/common/inc/nv-ioctl-numa.h
@@ -25,14 +25,12 @@
 #ifndef NV_IOCTL_NUMA_H
 #define NV_IOCTL_NUMA_H

-#if defined(NV_LINUX)
-
 #include <nv-ioctl-numbers.h>

-#if defined(NV_KERNEL_INTERFACE_LAYER)
-
+#if defined(NV_KERNEL_INTERFACE_LAYER) && defined(NV_LINUX)
 #include <linux/types.h>
-
+#elif defined (NV_KERNEL_INTERFACE_LAYER) && defined(NV_BSD)
+#include <sys/stdint.h>
 #else

 #include <stdint.h>
@@ -62,6 +60,7 @@ typedef struct nv_ioctl_numa_info
    uint64_t memblock_size __aligned(8);
    uint64_t numa_mem_addr __aligned(8);
    uint64_t numa_mem_size __aligned(8);
+    uint8_t  use_auto_online;
    nv_offline_addresses_t offline_addresses __aligned(8);
 } nv_ioctl_numa_info_t;

@@ -80,5 +79,3 @@ typedef struct nv_ioctl_set_numa_status
 #define NV_IOCTL_NUMA_STATUS_OFFLINE_FAILED         6

 #endif
-
-#endif
--- a/kernel-open/common/inc/nv-ioctl-numbers.h
+++ b/kernel-open/common/inc/nv-ioctl-numbers.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -39,5 +39,6 @@
 #define NV_ESC_QUERY_DEVICE_INTR     (NV_IOCTL_BASE + 13)
 #define NV_ESC_SYS_PARAMS            (NV_IOCTL_BASE + 14)
 #define NV_ESC_EXPORT_TO_DMABUF_FD   (NV_IOCTL_BASE + 17)
+#define NV_ESC_WAIT_OPEN_COMPLETE    (NV_IOCTL_BASE + 18)

 #endif
--- a/kernel-open/common/inc/nv-ioctl.h
+++ b/kernel-open/common/inc/nv-ioctl.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -104,7 +104,7 @@ typedef struct nv_ioctl_rm_api_version

 #define NV_RM_API_VERSION_CMD_STRICT         0
 #define NV_RM_API_VERSION_CMD_RELAXED       '1'
-#define NV_RM_API_VERSION_CMD_OVERRIDE      '2'
+#define NV_RM_API_VERSION_CMD_QUERY         '2'

 #define NV_RM_API_VERSION_REPLY_UNRECOGNIZED 0
 #define NV_RM_API_VERSION_REPLY_RECOGNIZED   1
@@ -142,4 +142,10 @@ typedef struct nv_ioctl_export_to_dma_buf_fd
    NvU32       status;
 } nv_ioctl_export_to_dma_buf_fd_t;

+typedef struct nv_ioctl_wait_open_complete
+{
+    int rc;
+    NvU32 adapterStatus;
+} nv_ioctl_wait_open_complete_t;
+
 #endif
--- a/kernel-open/common/inc/nv-kthread-q-os.h
+++ b/kernel-open/common/inc/nv-kthread-q-os.h
@@ -0,0 +1,64 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2016-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __NV_KTHREAD_QUEUE_OS_H__
+#define __NV_KTHREAD_QUEUE_OS_H__
+
+#include <linux/types.h>            // atomic_t
+#include <linux/list.h>             // list
+#include <linux/sched.h>            // task_struct
+#include <linux/numa.h>             // NUMA_NO_NODE
+#include <linux/semaphore.h>
+
+#include "conftest.h"
+
+struct nv_kthread_q
+{
+    struct list_head q_list_head;
+    spinlock_t q_lock;
+
+    // This is a counting semaphore. It gets incremented and decremented
+    // exactly once for each item that is added to the queue.
+    struct semaphore q_sem;
+    atomic_t main_loop_should_exit;
+
+    struct task_struct *q_kthread;
+
+    bool is_unload_flush_ongoing;
+};
+
+struct nv_kthread_q_item
+{
+    struct list_head q_list_node;
+    nv_q_func_t function_to_run;
+    void *function_args;
+};
+
+
+#ifndef NUMA_NO_NODE
+#define NUMA_NO_NODE (-1)
+#endif
+
+#define NV_KTHREAD_NO_NODE NUMA_NO_NODE
+
+#endif
--- a/kernel-open/common/inc/nv-kthread-q.h
+++ b/kernel-open/common/inc/nv-kthread-q.h
@@ -24,18 +24,14 @@
 #ifndef __NV_KTHREAD_QUEUE_H__
 #define __NV_KTHREAD_QUEUE_H__

-#include <linux/types.h>            // atomic_t
-#include <linux/list.h>             // list
-#include <linux/sched.h>            // task_struct
-#include <linux/numa.h>             // NUMA_NO_NODE
+struct nv_kthread_q;
+struct nv_kthread_q_item;
+typedef struct nv_kthread_q nv_kthread_q_t;
+typedef struct nv_kthread_q_item nv_kthread_q_item_t;

-#include "conftest.h"
+typedef void (*nv_q_func_t)(void *args);

-#if defined(NV_LINUX_SEMAPHORE_H_PRESENT)
-    #include <linux/semaphore.h>
-#else
-    #include <asm/semaphore.h>
-#endif
+#include "nv-kthread-q-os.h"

 ////////////////////////////////////////////////////////////////////////////////
 // nv_kthread_q:
@@ -90,43 +86,6 @@
 //
 ////////////////////////////////////////////////////////////////////////////////

-typedef struct nv_kthread_q nv_kthread_q_t;
-typedef struct nv_kthread_q_item nv_kthread_q_item_t;
-
-typedef void (*nv_q_func_t)(void *args);
-
-struct nv_kthread_q
-{
-    struct list_head q_list_head;
-    spinlock_t q_lock;
-
-    // This is a counting semaphore. It gets incremented and decremented
-    // exactly once for each item that is added to the queue.
-    struct semaphore q_sem;
-    atomic_t main_loop_should_exit;
-
-    struct task_struct *q_kthread;
-};
-
-struct nv_kthread_q_item
-{
-    struct list_head q_list_node;
-    nv_q_func_t function_to_run;
-    void *function_args;
-};
-
-#if defined(NV_KTHREAD_CREATE_ON_NODE_PRESENT)
-    #define NV_KTHREAD_Q_SUPPORTS_AFFINITY() 1
-#else
-    #define NV_KTHREAD_Q_SUPPORTS_AFFINITY() 0
-#endif
-
-#ifndef NUMA_NO_NODE
-#define NUMA_NO_NODE (-1)
-#endif
-
-#define NV_KTHREAD_NO_NODE NUMA_NO_NODE
-
 //
 // The queue must not be used before calling this routine.
 //
@@ -142,18 +101,12 @@ struct nv_kthread_q_item
 //
 // A short prefix of the qname arg will show up in []'s, via the ps(1) utility.
 //
-// The kernel thread stack is preferably allocated on the specified NUMA node if
-// NUMA-affinity (NV_KTHREAD_Q_SUPPORTS_AFFINITY() == 1) is supported, but
-// fallback to another node is possible because kernel allocators do not
+// The kernel thread stack is preferably allocated on the specified NUMA node, 
+// but fallback to another node is possible because kernel allocators do not
 // guarantee affinity. Note that NUMA-affinity applies only to
 // the kthread stack. This API does not do anything about limiting the CPU
 // affinity of the kthread. That is left to the caller.
 //
-// On kernels, which do not support NUMA-aware kthread stack allocations
-// (NV_KTHTREAD_Q_SUPPORTS_AFFINITY() == 0), the API will return -ENOTSUPP
-// if the value supplied for 'preferred_node' is anything other than
-// NV_KTHREAD_NO_NODE.
-//
 // Reusing a queue: once a queue is initialized, it must be safely shut down
 // (see "Stopping the queue(s)", below), before it can be reused. So, for
 // a simple queue use case, the following will work:
@@ -171,10 +124,7 @@ int nv_kthread_q_init_on_node(nv_kthread_q_t *q,
 // This routine is the same as nv_kthread_q_init_on_node() with the exception
 // that the queue stack will be allocated on the NUMA node of the caller.
 //
-static inline int nv_kthread_q_init(nv_kthread_q_t *q, const char *qname)
-{
-    return nv_kthread_q_init_on_node(q, qname, NV_KTHREAD_NO_NODE);
-}
+int nv_kthread_q_init(nv_kthread_q_t *q, const char *qname);

 //
 // The caller is responsible for stopping all queues, by calling this routine
--- a/kernel-open/common/inc/nv-linux.h
+++ b/kernel-open/common/inc/nv-linux.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2001-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2001-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -35,6 +35,7 @@
 #include "os-interface.h"
 #include "nv-timer.h"
 #include "nv-time.h"
+#include "nv-chardev-numbers.h"

 #define NV_KERNEL_NAME "Linux"

@@ -57,14 +58,10 @@
 #include <linux/version.h>
 #include <linux/utsname.h>

-#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 32)
-#error "This driver does not support kernels older than 2.6.32!"
-#elif LINUX_VERSION_CODE < KERNEL_VERSION(2, 7, 0)
-#  define KERNEL_2_6
-#elif LINUX_VERSION_CODE >= KERNEL_VERSION(3, 0, 0)
-#  define KERNEL_3
-#else
-#error "This driver does not support development kernels!"
+#if LINUX_VERSION_CODE == KERNEL_VERSION(4, 4, 0)
+// Version 4.4 is allowed, temporarily, although not officially supported.
+#elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 15, 0)
+#error "This driver does not support kernels older than Linux 4.15!"
 #endif

 #if defined (CONFIG_SMP) && !defined (__SMP__)
@@ -191,13 +188,6 @@
 */
 #define NV_CURRENT_EUID() (__kuid_val(current->cred->euid))

-#if !defined(NV_KUID_T_PRESENT)
-static inline uid_t __kuid_val(uid_t uid)
-{
-    return uid;
-}
-#endif
-
 #if defined(CONFIG_VGA_ARB)
 #include <linux/vgaarb.h>
 #endif
@@ -218,6 +208,7 @@ static inline uid_t __kuid_val(uid_t uid)
 #include <linux/highmem.h>

 #include <linux/nodemask.h>
+#include <linux/memory.h>

 #include <linux/workqueue.h>        /* workqueue                        */
 #include "nv-kthread-q.h"           /* kthread based queue              */
@@ -227,24 +218,13 @@ static inline uid_t __kuid_val(uid_t uid)
 #endif

 #include <linux/fb.h>               /* fb_info struct                   */
+#include <linux/screen_info.h>      /* screen_info                      */

 #if !defined(CONFIG_PCI)
 #warning "Attempting to build driver for a platform with no PCI support!"
 #include <asm-generic/pci-dma-compat.h>
 #endif

-#if defined(NV_EFI_ENABLED_PRESENT) && defined(NV_EFI_ENABLED_ARGUMENT_COUNT)
-#if (NV_EFI_ENABLED_ARGUMENT_COUNT == 1)
-#define NV_EFI_ENABLED() efi_enabled(EFI_BOOT)
-#else
-#error "NV_EFI_ENABLED_ARGUMENT_COUNT value unrecognized!"
-#endif
-#elif (defined(NV_EFI_ENABLED_PRESENT) || defined(efi_enabled))
-#define NV_EFI_ENABLED() efi_enabled
-#else
-#define NV_EFI_ENABLED() 0
-#endif
-
 #if defined(CONFIG_CRAY_XT)
 #include <cray/cray_nvidia.h>
 NV_STATUS nvos_forward_error_to_cray(struct pci_dev *, NvU32,
@@ -265,7 +245,7 @@ NV_STATUS nvos_forward_error_to_cray(struct pci_dev *, NvU32,
 #undef NV_SET_PAGES_UC_PRESENT
 #endif

-#if !defined(NVCPU_AARCH64) && !defined(NVCPU_PPC64LE)
+#if !defined(NVCPU_AARCH64) && !defined(NVCPU_PPC64LE) && !defined(NVCPU_RISCV64)
 #if !defined(NV_SET_MEMORY_UC_PRESENT) && !defined(NV_SET_PAGES_UC_PRESENT)
 #error "This driver requires the ability to change memory types!"
 #endif
@@ -423,32 +403,6 @@ extern int nv_pat_mode;
 #define NV_GFP_DMA32 (NV_GFP_KERNEL)
 #endif

-extern NvBool nvos_is_chipset_io_coherent(void);
-
-#if defined(NVCPU_X86_64)
-#define CACHE_FLUSH()  asm volatile("wbinvd":::"memory")
-#define WRITE_COMBINE_FLUSH() asm volatile("sfence":::"memory")
-#elif defined(NVCPU_AARCH64)
-    static inline void nv_flush_cache_cpu(void *info)
-    {
-        if (!nvos_is_chipset_io_coherent())
-        {
-#if defined(NV_FLUSH_CACHE_ALL_PRESENT)
-            flush_cache_all();
-#else
-            WARN_ONCE(0, "NVRM: kernel does not support flush_cache_all()\n");
-#endif
-        }
-    }
-#define CACHE_FLUSH()            nv_flush_cache_cpu(NULL)
-#define CACHE_FLUSH_ALL()        on_each_cpu(nv_flush_cache_cpu, NULL, 1)
-#define WRITE_COMBINE_FLUSH()    mb()
-#elif defined(NVCPU_PPC64LE)
-#define CACHE_FLUSH()            asm volatile("sync;  \n" \
-                                              "isync; \n" ::: "memory")
-#define WRITE_COMBINE_FLUSH()    CACHE_FLUSH()
-#endif
-
 typedef enum
 {
    NV_MEMORY_TYPE_SYSTEM,      /* Memory mapped for ROM, SBIOS and physical RAM. */
@@ -457,7 +411,7 @@ typedef enum
    NV_MEMORY_TYPE_DEVICE_MMIO, /* All kinds of MMIO referred by NVRM e.g. BARs and MCFG of device */
 } nv_memory_type_t;

-#if defined(NVCPU_AARCH64) || defined(NVCPU_PPC64LE)
+#if defined(NVCPU_AARCH64) || defined(NVCPU_PPC64LE) || defined(NVCPU_RISCV64)
 #define NV_ALLOW_WRITE_COMBINING(mt)    1
 #elif defined(NVCPU_X86_64)
 #if defined(NV_ENABLE_PAT_SUPPORT)
@@ -516,11 +470,13 @@ static inline void *nv_vmalloc(unsigned long size)
    void *ptr = __vmalloc(size, GFP_KERNEL);
 #endif
    if (ptr)
+    {
        NV_MEMDBG_ADD(ptr, size);
+    }
    return ptr;
 }

-static inline void nv_vfree(void *ptr, NvU32 size)
+static inline void nv_vfree(void *ptr, NvU64 size)
 {
    NV_MEMDBG_REMOVE(ptr, size);
    vfree(ptr);
@@ -528,9 +484,15 @@ static inline void nv_vfree(void *ptr, NvU32 size)

 static inline void *nv_ioremap(NvU64 phys, NvU64 size)
 {
+#if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_PRESENT)
+    void *ptr = ioremap_driver_hardened(phys, size);
+#else
    void *ptr = ioremap(phys, size);
+#endif
    if (ptr)
+    {
        NV_MEMDBG_ADD(ptr, size);
+    }
    return ptr;
 }

@@ -541,11 +503,11 @@ static inline void *nv_ioremap_nocache(NvU64 phys, NvU64 size)

 static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
 {
-#if defined(NV_IOREMAP_CACHE_PRESENT)
-    void *ptr = ioremap_cache(phys, size);
-    if (ptr)
-        NV_MEMDBG_ADD(ptr, size);
-    return ptr;
+    void *ptr = NULL;
+#if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_CACHE_SHARED_PRESENT)
+    ptr = ioremap_cache_shared(phys, size);
+#elif defined(NV_IOREMAP_CACHE_PRESENT)
+    ptr = ioremap_cache(phys, size);
 #elif defined(NVCPU_PPC64LE)
    //
    // ioremap_cache() has been only implemented correctly for ppc64le with
@@ -560,25 +522,34 @@ static inline void *nv_ioremap_cache(NvU64 phys, NvU64 size)
    // (commit 40f1ce7fb7e8, kernel 3.0+) and that covers all kernels we
    // support on power.
    //
-    void *ptr = ioremap_prot(phys, size, pgprot_val(PAGE_KERNEL));
-    if (ptr)
-        NV_MEMDBG_ADD(ptr, size);
-    return ptr;
+    ptr = ioremap_prot(phys, size, pgprot_val(PAGE_KERNEL));
 #else
    return nv_ioremap(phys, size);
 #endif
+
+    if (ptr)
+    {
+        NV_MEMDBG_ADD(ptr, size);
+    }
+    return ptr;
 }

 static inline void *nv_ioremap_wc(NvU64 phys, NvU64 size)
 {
-#if defined(NV_IOREMAP_WC_PRESENT)
-    void *ptr = ioremap_wc(phys, size);
-    if (ptr)
-        NV_MEMDBG_ADD(ptr, size);
-    return ptr;
+    void *ptr = NULL;
+#if IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && defined(NV_IOREMAP_DRIVER_HARDENED_WC_PRESENT)
+    ptr = ioremap_driver_hardened_wc(phys, size);
+#elif defined(NV_IOREMAP_WC_PRESENT)
+    ptr = ioremap_wc(phys, size);
 #else
    return nv_ioremap_nocache(phys, size);
 #endif
+
+    if (ptr)
+    {
+        NV_MEMDBG_ADD(ptr, size);
+    }
+    return ptr;
 }

 static inline void nv_iounmap(void *ptr, NvU64 size)
@@ -591,11 +562,7 @@ static NvBool nv_numa_node_has_memory(int node_id)
 {
    if (node_id < 0 || node_id >= MAX_NUMNODES)
        return NV_FALSE;
-#if defined(NV_NODE_STATES_N_MEMORY_PRESENT)
    return node_state(node_id, N_MEMORY) ? NV_TRUE : NV_FALSE;
-#else
-    return node_state(node_id, N_HIGH_MEMORY) ? NV_TRUE : NV_FALSE;
-#endif
 }

 #define NV_KMALLOC(ptr, size) \
@@ -605,6 +572,13 @@ static NvBool nv_numa_node_has_memory(int node_id)
            NV_MEMDBG_ADD(ptr, size); \
    }

+#define NV_KZALLOC(ptr, size) \
+    { \
+        (ptr) = kzalloc(size, NV_GFP_KERNEL); \
+        if (ptr) \
+            NV_MEMDBG_ADD(ptr, size); \
+    }
+
 #define NV_KMALLOC_ATOMIC(ptr, size) \
    { \
        (ptr) = kmalloc(size, NV_GFP_ATOMIC); \
@@ -648,6 +622,26 @@ static NvBool nv_numa_node_has_memory(int node_id)
        free_pages(ptr, order);                      \
    }

+static inline pgprot_t nv_sme_clr(pgprot_t prot)
+{
+#if defined(__sme_clr)
+    return __pgprot(__sme_clr(pgprot_val(prot)));
+#else
+    return prot;
+#endif // __sme_clr
+}
+
+static inline pgprot_t nv_adjust_pgprot(pgprot_t vm_prot, NvU32 extra)
+{
+    pgprot_t prot = __pgprot(pgprot_val(vm_prot) | extra);
+
+#if defined(pgprot_decrypted)
+    return pgprot_decrypted(prot);
+#else
+    return nv_sme_clr(prot);
+#endif // pgprot_decrypted
+}
+
 #if defined(PAGE_KERNEL_NOENC)
 #if defined(__pgprot_mask)
 #define NV_PAGE_KERNEL_NOCACHE_NOENC __pgprot_mask(__PAGE_KERNEL_NOCACHE)
@@ -669,7 +663,8 @@ static inline NvUPtr nv_vmap(struct page **pages, NvU32 page_count,
 #if defined(PAGE_KERNEL_NOENC)
    if (unencrypted)
    {
-        prot = cached ? PAGE_KERNEL_NOENC : NV_PAGE_KERNEL_NOCACHE_NOENC;
+        prot = cached ? nv_adjust_pgprot(PAGE_KERNEL_NOENC, 0) :
+                        nv_adjust_pgprot(NV_PAGE_KERNEL_NOCACHE_NOENC, 0);
    }
    else
 #endif
@@ -682,7 +677,9 @@ static inline NvUPtr nv_vmap(struct page **pages, NvU32 page_count,
    /* All memory cached in PPC64LE; can't honor 'cached' input. */
    ptr = vmap(pages, page_count, VM_MAP, prot);
    if (ptr)
+    {
        NV_MEMDBG_ADD(ptr, page_count * PAGE_SIZE);
+    }
    return (NvUPtr)ptr;
 }

@@ -727,6 +724,7 @@ static inline dma_addr_t nv_phys_to_dma(struct device *dev, NvU64 pa)
 #endif
 }

+#define NV_GET_OFFSET_IN_PAGE(phys_page) offset_in_page(phys_page)
 #define NV_GET_PAGE_STRUCT(phys_page) virt_to_page(__va(phys_page))
 #define NV_VMA_PGOFF(vma)             ((vma)->vm_pgoff)
 #define NV_VMA_SIZE(vma)              ((vma)->vm_end - (vma)->vm_start)
@@ -735,7 +733,6 @@ static inline dma_addr_t nv_phys_to_dma(struct device *dev, NvU64 pa)
 #define NV_VMA_FILE(vma)              ((vma)->vm_file)

 #define NV_DEVICE_MINOR_NUMBER(x)     minor((x)->i_rdev)
-#define NV_CONTROL_DEVICE_MINOR       255

 #define NV_PCI_DISABLE_DEVICE(pci_dev)                           \
    {                                                            \
@@ -837,25 +834,23 @@ static inline dma_addr_t nv_phys_to_dma(struct device *dev, NvU64 pa)
    })
 #endif

-#if defined(NV_PCI_STOP_AND_REMOVE_BUS_DEVICE_PRESENT)  // introduced in 3.4.9
+#if defined(NV_PCI_STOP_AND_REMOVE_BUS_DEVICE_PRESENT)  // introduced in 3.18-rc1 for aarch64
 #define NV_PCI_STOP_AND_REMOVE_BUS_DEVICE(pci_dev) pci_stop_and_remove_bus_device(pci_dev)
-#elif defined(NV_PCI_REMOVE_BUS_DEVICE_PRESENT) // introduced in 2.6
-#define NV_PCI_STOP_AND_REMOVE_BUS_DEVICE(pci_dev) pci_remove_bus_device(pci_dev)
 #endif

 #define NV_PRINT_AT(nv_debug_level,at)                                           \
    {                                                                            \
        nv_printf(nv_debug_level,                                                \
-            "NVRM: VM: %s:%d: 0x%p, %d page(s), count = %d, flags = 0x%08x, "    \
+            "NVRM: VM: %s:%d: 0x%p, %d page(s), count = %d, "                    \
            "page_table = 0x%p\n",  __FUNCTION__, __LINE__, at,                  \
            at->num_pages, NV_ATOMIC_READ(at->usage_count),                      \
-            at->flags, at->page_table);                                          \
+            at->page_table);                                                     \
    }

 #define NV_PRINT_VMA(nv_debug_level,vma)                                                 \
    {                                                                                    \
        nv_printf(nv_debug_level,                                                        \
-            "NVRM: VM: %s:%d: 0x%lx - 0x%lx, 0x%08x bytes @ 0x%016llx, 0x%p, 0x%p\n",    \
+            "NVRM: VM: %s:%d: 0x%lx - 0x%lx, 0x%08lx bytes @ 0x%016llx, 0x%p, 0x%p\n",    \
            __FUNCTION__, __LINE__, vma->vm_start, vma->vm_end, NV_VMA_SIZE(vma),        \
            NV_VMA_OFFSET(vma), NV_VMA_PRIVATE(vma), NV_VMA_FILE(vma));                  \
    }
@@ -956,35 +951,15 @@ static inline int nv_remap_page_range(struct vm_area_struct *vma,
    return ret;
 }

-static inline pgprot_t nv_adjust_pgprot(pgprot_t vm_prot, NvU32 extra)
-{
-    pgprot_t prot = __pgprot(pgprot_val(vm_prot) | extra);
-#if defined(CONFIG_AMD_MEM_ENCRYPT) && defined(NV_PGPROT_DECRYPTED_PRESENT)
-    /*
-     * When AMD memory encryption is enabled, device memory mappings with the
-     * C-bit set read as 0xFF, so ensure the bit is cleared for user mappings.
-     *
-     * If cc_mkdec() is present, then pgprot_decrypted() can't be used.
-     */
-#if defined(NV_CC_MKDEC_PRESENT)
-    prot =  __pgprot(__sme_clr(pgprot_val(vm_prot)));
-#else
-    prot = pgprot_decrypted(prot);
-#endif
-#endif
-
-    return prot;
-}
-
 static inline int nv_io_remap_page_range(struct vm_area_struct *vma,
-    NvU64 phys_addr, NvU64 size, NvU32 extra_prot)
+    NvU64 phys_addr, NvU64 size, NvU32 extra_prot, NvU64 start)
 {
    int ret = -1;
 #if !defined(NV_XEN_SUPPORT_FULLY_VIRTUALIZED_KERNEL)
-    ret = nv_remap_page_range(vma, vma->vm_start, phys_addr, size,
+    ret = nv_remap_page_range(vma, start, phys_addr, size,
        nv_adjust_pgprot(vma->vm_page_prot, extra_prot));
 #else
-    ret = io_remap_pfn_range(vma, vma->vm_start, (phys_addr >> PAGE_SHIFT),
+    ret = io_remap_pfn_range(vma, start, (phys_addr >> PAGE_SHIFT),
        size, nv_adjust_pgprot(vma->vm_page_prot, extra_prot));
 #endif
    return ret;
@@ -1036,6 +1011,32 @@ static inline vm_fault_t nv_insert_pfn(struct vm_area_struct *vma,
    return VM_FAULT_SIGBUS;
 }

+/* Converts BAR index to Linux specific PCI BAR index */
+static inline NvU8 nv_bar_index_to_os_bar_index
+(
+    struct pci_dev *dev,
+    NvU8 nv_bar_index
+)
+{
+    NvU8 bar_index = 0;
+    NvU8 i;
+
+    BUG_ON(nv_bar_index >= NV_GPU_NUM_BARS);
+
+    for (i = 0; i < nv_bar_index; i++)
+    {
+        if (NV_PCI_RESOURCE_FLAGS(dev, bar_index) & PCI_BASE_ADDRESS_MEM_TYPE_64)
+        {
+            bar_index += 2;
+        }
+        else
+        {
+            bar_index++;
+        }
+    }
+
+    return bar_index;
+}

 #define NV_PAGE_MASK    (NvU64)(long)PAGE_MASK

@@ -1082,6 +1083,8 @@ static inline void nv_kmem_ctor_dummy(void *arg)
        kmem_cache_destroy(kmem_cache);     \
    }

+#define NV_KMEM_CACHE_ALLOC_ATOMIC(kmem_cache)     \
+    kmem_cache_alloc(kmem_cache, GFP_ATOMIC)
 #define NV_KMEM_CACHE_ALLOC(kmem_cache)     \
    kmem_cache_alloc(kmem_cache, GFP_KERNEL)
 #define NV_KMEM_CACHE_FREE(ptr, kmem_cache) \
@@ -1108,15 +1111,35 @@ static inline void *nv_kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags)
 #endif
 }

+static inline int nv_kmem_cache_alloc_stack_atomic(nvidia_stack_t **stack)
+{
+    nvidia_stack_t *sp = NULL;
+#if defined(NVCPU_X86_64)
+    if (rm_is_altstack_in_use())
+    {
+        sp = NV_KMEM_CACHE_ALLOC_ATOMIC(nvidia_stack_t_cache);
+        if (sp == NULL)
+            return -ENOMEM;
+        sp->size = sizeof(sp->stack);
+        sp->top = sp->stack + sp->size;
+    }
+#endif
+    *stack = sp;
+    return 0;
+}
+
 static inline int nv_kmem_cache_alloc_stack(nvidia_stack_t **stack)
 {
    nvidia_stack_t *sp = NULL;
 #if defined(NVCPU_X86_64)
-    sp = NV_KMEM_CACHE_ALLOC(nvidia_stack_t_cache);
-    if (sp == NULL)
-        return -ENOMEM;
-    sp->size = sizeof(sp->stack);
-    sp->top = sp->stack + sp->size;
+    if (rm_is_altstack_in_use())
+    {
+        sp = NV_KMEM_CACHE_ALLOC(nvidia_stack_t_cache);
+        if (sp == NULL)
+            return -ENOMEM;
+        sp->size = sizeof(sp->stack);
+        sp->top = sp->stack + sp->size;
+    }
 #endif
    *stack = sp;
    return 0;
@@ -1125,7 +1148,7 @@ static inline int nv_kmem_cache_alloc_stack(nvidia_stack_t **stack)
 static inline void nv_kmem_cache_free_stack(nvidia_stack_t *stack)
 {
 #if defined(NVCPU_X86_64)
-    if (stack != NULL)
+    if (stack != NULL && rm_is_altstack_in_use())
    {
        NV_KMEM_CACHE_FREE(stack, nvidia_stack_t_cache);
    }
@@ -1160,15 +1183,15 @@ typedef struct nvidia_pte_s {
    unsigned int    page_count;
 } nvidia_pte_t;

-
-
-
-
-
-
-
-
-
+#if defined(CONFIG_DMA_SHARED_BUFFER)
+/* Standard dma_buf-related information. */
+struct nv_dma_buf
+{
+    struct dma_buf *dma_buf;
+    struct dma_buf_attachment *dma_attach;
+    struct sg_table *sgt;
+};
+#endif // CONFIG_DMA_SHARED_BUFFER

 typedef struct nv_alloc_s {
    struct nv_alloc_s *next;
@@ -1180,11 +1203,12 @@ typedef struct nv_alloc_s {
        NvBool zeroed      : 1;
        NvBool aliased     : 1;
        NvBool user        : 1;
-        NvBool node0       : 1;
+        NvBool node        : 1;
        NvBool peer_io     : 1;
        NvBool physical    : 1;
        NvBool unencrypted : 1;
        NvBool coherent    : 1;
+        NvBool carveout    : 1;
    } flags;
    unsigned int   cache_type;
    unsigned int   num_pages;
@@ -1194,6 +1218,7 @@ typedef struct nv_alloc_s {
    unsigned int   pid;
    struct page  **user_pages;
    NvU64         guest_id;             /* id of guest VM */
+    NvS32         node_id;              /* Node id for memory allocation when node is set in flags */
    void          *import_priv;
    struct sg_table *import_sgt;
 } nv_alloc_t;
@@ -1306,7 +1331,7 @@ nv_dma_maps_swiotlb(struct device *dev)
     * SEV memory encryption") forces SWIOTLB to be enabled when AMD SEV 
     * is active in all cases.
     */
-    if (os_sev_enabled)
+    if (os_cc_enabled)
        swiotlb_in_use = NV_TRUE;
 #endif

@@ -1360,7 +1385,19 @@ typedef struct nv_dma_map_s {
         i < dm->mapping.discontig.submap_count;                              \
         i++, sm = &dm->mapping.discontig.submaps[i])

+/*
+ * On 4K ARM kernels, use max submap size a multiple of 64K to keep nv-p2p happy.
+ * Despite 4K OS pages, we still use 64K P2P pages due to dependent modules still using 64K.
+ * Instead of using (4G-4K), use max submap size as (4G-64K) since the mapped IOVA range
+ * must be aligned at 64K boundary.
+ */
+#if defined(CONFIG_ARM64_4K_PAGES)
+#define NV_DMA_U32_MAX_4K_PAGES           ((NvU32)((NV_U32_MAX >> PAGE_SHIFT) + 1))
+#define NV_DMA_SUBMAP_MAX_PAGES           ((NvU32)(NV_DMA_U32_MAX_4K_PAGES - 16))
+#else
 #define NV_DMA_SUBMAP_MAX_PAGES           ((NvU32)(NV_U32_MAX >> PAGE_SHIFT))
+#endif
+
 #define NV_DMA_SUBMAP_IDX_TO_PAGE_IDX(s)  (s * NV_DMA_SUBMAP_MAX_PAGES)

 /*
@@ -1369,8 +1406,7 @@ typedef struct nv_dma_map_s {
 * xen_swiotlb_map_sg_attrs may try to route to the SWIOTLB. We must only use
 * single-page sg elements on Xen Server.
 */
-#if defined(NV_SG_ALLOC_TABLE_FROM_PAGES_PRESENT) && \
-    !defined(NV_DOM0_KERNEL_PRESENT)
+#if !defined(NV_DOM0_KERNEL_PRESENT)
    #define NV_ALLOC_DMA_SUBMAP_SCATTERLIST(dm, sm, i)                        \
        ((sg_alloc_table_from_pages(&sm->sgt,                                 \
            &dm->pages[NV_DMA_SUBMAP_IDX_TO_PAGE_IDX(i)],                     \
@@ -1412,34 +1448,6 @@ struct os_wait_queue {
    struct completion q;
 };

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
 /*
 * To report error in msi/msix when unhandled count reaches a threshold
 */
@@ -1463,18 +1471,39 @@ struct nv_dma_device {
    NvBool nvlink;
 };

+/* Properties of the coherent link */
+typedef struct coherent_link_info_s {
+    /* Physical Address of the GPU memory in SOC AMAP. In the case of
+     * baremetal OS environment it is System Physical Address(SPA) and in the case
+     * of virutalized OS environment it is Intermediate Physical Address(IPA) */
+    NvU64 gpu_mem_pa;

+    /* Physical address of the reserved portion of the GPU memory, applicable
+     * only in Grace Hopper self hosted passthrough virtualizatioan platform. */
+    NvU64 rsvd_mem_pa;

+    /* Bitmap of NUMA node ids, corresponding to the reserved PXMs,
+     * available for adding GPU memory to the kernel as system RAM */
+    DECLARE_BITMAP(free_node_bitmap, MAX_NUMNODES);
+} coherent_link_info_t;

-
-
-
-
-
-
-
-
-
+#if defined(NV_LINUX_ACPI_EVENTS_SUPPORTED)
+/*
+ * acpi data storage structure
+ *
+ * This structure retains the pointer to the device,
+ * and any other baggage we want to carry along
+ *
+ */
+typedef struct
+{
+    nvidia_stack_t *sp;
+    struct acpi_device *device;
+    struct acpi_handle *handle;
+    void *notifier_data;
+    int notify_handler_installed;
+} nv_acpi_t;
+#endif

 /* linux-specific version of old nv_state_t */
 /* this is a general os-specific state structure. the first element *must* be
@@ -1491,10 +1520,12 @@ typedef struct nv_linux_state_s {
    /* IBM-NPU info associated with this GPU */
    nv_ibmnpu_info_t *npu;

+    /* coherent link information */
+     coherent_link_info_t coherent_link_info;

-
-
-
+    /* Dedicated queue to be used for removing FB memory which is onlined
+     * to kernel as a NUMA node. Refer Bug : 3879845*/
+    nv_kthread_q_t remove_numa_memory_q;

    /* NUMA node information for the platforms where GPU memory is presented
     * as a NUMA node to the kernel */
@@ -1506,6 +1537,7 @@ typedef struct nv_linux_state_s {
        /* NUMA online/offline status for platforms that support GPU memory as
         * NUMA node */
        atomic_t status;
+        NvBool use_auto_online;
    } numa_info;

    nvidia_stack_t *sp[NV_DEV_STACK_COUNT];
@@ -1575,25 +1607,13 @@ typedef struct nv_linux_state_s {
    /* Per-device notifier block for ACPI events */
    struct notifier_block acpi_nb;

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+#if defined(NV_LINUX_ACPI_EVENTS_SUPPORTED)
+    nv_acpi_t* nv_acpi_object;
+#endif

    /* Lock serializing ISRs for different SOC vectors */
    nv_spinlock_t soc_isr_lock;
+    void *soc_bh_mutex;

    struct nv_timer snapshot_timer;
    nv_spinlock_t snapshot_timer_lock;
@@ -1609,6 +1629,30 @@ typedef struct nv_linux_state_s {

    struct nv_dma_device dma_dev;
    struct nv_dma_device niso_dma_dev;
+
+    /*
+     * Background kthread for handling deferred open operations
+     * (e.g. from O_NONBLOCK).
+     *
+     * Adding to open_q and reading/writing is_accepting_opens
+     * are protected by nvl->open_q_lock (not nvl->ldata_lock).
+     * This allows new deferred open operations to be enqueued without
+     * blocking behind previous ones (which hold nvl->ldata_lock).
+     *
+     * Adding to open_q is only safe if is_accepting_opens is true.
+     * This prevents open operations from racing with device removal.
+     *
+     * Stopping open_q is only safe after setting is_accepting_opens to false.
+     * This ensures that the open_q (and the larger nvl structure) will
+     * outlive any of the open operations enqueued.
+     */
+    nv_kthread_q_t open_q;
+    NvBool is_accepting_opens;
+    struct semaphore open_q_lock;
+#if defined(NV_VGPU_KVM_BUILD)
+    wait_queue_head_t wait;
+    NvS32 return_status;
+#endif
 } nv_linux_state_t;

 extern nv_linux_state_t *nv_linux_devices;
@@ -1639,24 +1683,6 @@ extern struct rw_semaphore nv_system_pm_lock;

 extern NvBool nv_ats_supported;

-#if defined(NV_LINUX_ACPI_EVENTS_SUPPORTED)
-/*
- * acpi data storage structure
- *
- * This structure retains the pointer to the device,
- * and any other baggage we want to carry along
- *
- */
-typedef struct
-{
-    nvidia_stack_t *sp;
-    struct acpi_device *device;
-    struct acpi_handle *handle;
-    int notify_handler_installed;
-} nv_acpi_t;
-
-#endif
-
 /*
 * file-private data
 * hide a pointer to our data structures in a file-private ptr
@@ -1670,22 +1696,13 @@ typedef struct nvidia_event
    nv_event_t event;
 } nvidia_event_t;

-typedef enum
-{
-    NV_FOPS_STACK_INDEX_MMAP,
-    NV_FOPS_STACK_INDEX_IOCTL,
-    NV_FOPS_STACK_INDEX_COUNT
-} nvidia_entry_point_index_t;
-
 typedef struct
 {
    nv_file_private_t nvfp;

    nvidia_stack_t *sp;
-    nvidia_stack_t *fops_sp[NV_FOPS_STACK_INDEX_COUNT];
-    struct semaphore fops_sp_lock[NV_FOPS_STACK_INDEX_COUNT];
    nv_alloc_t *free_list;
-    void *nvptr;
+    nv_linux_state_t *nvptr;
    nvidia_event_t *event_data_head, *event_data_tail;
    NvBool dataless_event_pending;
    nv_spinlock_t fp_lock;
@@ -1696,6 +1713,12 @@ typedef struct
    nv_alloc_mapping_context_t mmap_context;
    struct address_space mapping;

+    nv_kthread_q_item_t open_q_item;
+    struct completion open_complete;
+    nv_linux_state_t *deferred_open_nvl;
+    int open_rc;
+    NV_STATUS adapter_status;
+
    struct list_head entry;
 } nv_linux_file_private_t;

@@ -1704,6 +1727,21 @@ static inline nv_linux_file_private_t *nv_get_nvlfp_from_nvfp(nv_file_private_t
    return container_of(nvfp, nv_linux_file_private_t, nvfp);
 }

+static inline int nv_wait_open_complete_interruptible(nv_linux_file_private_t *nvlfp)
+{
+    return wait_for_completion_interruptible(&nvlfp->open_complete);
+}
+
+static inline void nv_wait_open_complete(nv_linux_file_private_t *nvlfp)
+{
+    wait_for_completion(&nvlfp->open_complete);
+}
+
+static inline NvBool nv_is_open_complete(nv_linux_file_private_t *nvlfp)
+{
+    return completion_done(&nvlfp->open_complete);
+}
+
 #define NV_SET_FILE_PRIVATE(filep,data) ((filep)->private_data = (data))
 #define NV_GET_LINUX_FILE_PRIVATE(filep) ((nv_linux_file_private_t *)(filep)->private_data)

@@ -1713,7 +1751,6 @@ static inline nv_linux_file_private_t *nv_get_nvlfp_from_nvfp(nv_file_private_t

 #define NV_STATE_PTR(nvl)   &(((nv_linux_state_t *)(nvl))->nv_state)

-
 #define NV_ATOMIC_READ(data)            atomic_read(&(data))
 #define NV_ATOMIC_SET(data,val)         atomic_set(&(data), (val))
 #define NV_ATOMIC_INC(data)             atomic_inc(&(data))
@@ -1759,12 +1796,10 @@ static inline struct kmem_cache *nv_kmem_cache_create(const char *name, unsigned
    return cache;
 }

-
 #if defined(CONFIG_PCI_IOV)
 #define NV_PCI_SRIOV_SUPPORT
 #endif /* CONFIG_PCI_IOV */

-
 #define NV_PCIE_CFG_MAX_OFFSET 0x1000

 #include "nv-proto.h"
@@ -1787,12 +1822,19 @@ static inline NV_STATUS nv_check_gpu_state(nv_state_t *nv)

 extern NvU32 NVreg_EnableUserNUMAManagement;
 extern NvU32 NVreg_RegisterPCIDriver;
+extern NvU32 NVreg_EnableResizableBar;
+extern NvU32 NVreg_EnableNonblockingOpen;

 extern NvU32 num_probed_nv_devices;
 extern NvU32 num_nv_devices;

 #define NV_FILE_INODE(file) (file)->f_inode

+static inline int nv_is_control_device(struct inode *inode)
+{
+    return (minor((inode)->i_rdev) == NV_MINOR_DEVICE_NUMBER_CONTROL_DEVICE);
+}
+
 #if defined(NV_DOM0_KERNEL_PRESENT) || defined(NV_VGPU_KVM_BUILD)
 #define NV_VGX_HYPER
 #if defined(NV_XEN_IOEMU_INJECT_MSI)
@@ -1800,20 +1842,6 @@ extern NvU32 num_nv_devices;
 #endif
 #endif

-static inline NvU64 nv_pci_bus_address(struct pci_dev *dev, NvU8 bar_index)
-{
-    NvU64 bus_addr = 0;
-#if defined(NV_PCI_BUS_ADDRESS_PRESENT)
-    bus_addr = pci_bus_address(dev, bar_index);
-#elif defined(CONFIG_PCI)
-    struct pci_bus_region region;
-
-    pcibios_resource_to_bus(dev, &region, &dev->resource[bar_index]);
-    bus_addr = region.start;
-#endif
-    return bus_addr;
-}
-
 /*
 * Decrements the usage count of the allocation, and moves the allocation to
 * the given nvlfp's free list if the usage count drops to zero.
@@ -1943,25 +1971,12 @@ static inline NvU32 nv_default_irq_flags(nv_state_t *nv)
    #define NV_GET_UNUSED_FD_FLAGS(flags)  (-1)
 #endif

-#if defined(NV_SET_CLOSE_ON_EXEC_PRESENT)
-    #define NV_SET_CLOSE_ON_EXEC(fd, fdt) __set_close_on_exec(fd, fdt)
-#elif defined(NV_LINUX_TIME_H_PRESENT) && defined(FD_SET)
-    #define NV_SET_CLOSE_ON_EXEC(fd, fdt) FD_SET(fd, fdt->close_on_exec)
-#else
-    #define NV_SET_CLOSE_ON_EXEC(fd, fdt) __set_bit(fd, fdt->close_on_exec)
-#endif
-
 #define MODULE_BASE_NAME "nvidia"
 #define MODULE_INSTANCE_NUMBER 0
 #define MODULE_INSTANCE_STRING ""
 #define MODULE_NAME MODULE_BASE_NAME MODULE_INSTANCE_STRING

-NvS32 nv_request_soc_irq(nv_linux_state_t *, NvU32, nv_soc_irq_type_t, NvU32, NvU32);
-
-
-
-
-
+NvS32 nv_request_soc_irq(nv_linux_state_t *, NvU32, nv_soc_irq_type_t, NvU32, NvU32, const char*);

 static inline void nv_mutex_destroy(struct mutex *lock)
 {
@@ -1994,6 +2009,11 @@ static inline int nv_set_numa_status(nv_linux_state_t *nvl, int status)
    return 0;
 }

+static inline NvBool nv_platform_use_auto_online(nv_linux_state_t *nvl)
+{
+    return nvl->numa_info.use_auto_online;
+}
+
 typedef enum
 {
    NV_NUMA_STATUS_DISABLED             = 0,
@@ -2054,4 +2074,7 @@ typedef enum
 #include <linux/clk-provider.h>
 #endif

+#define NV_EXPORT_SYMBOL(symbol)        EXPORT_SYMBOL_GPL(symbol)
+#define NV_CHECK_EXPORT_SYMBOL(symbol)  NV_IS_EXPORT_SYMBOL_PRESENT_##symbol
+
 #endif  /* _NV_LINUX_H_ */
--- a/kernel-open/common/inc/nv-list-helpers.h
+++ b/kernel-open/common/inc/nv-list-helpers.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2013-2020 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2013-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -73,21 +73,4 @@
    }
 #endif

-#if defined(NV_HLIST_FOR_EACH_ENTRY_ARGUMENT_COUNT)
-#if NV_HLIST_FOR_EACH_ENTRY_ARGUMENT_COUNT == 3
-#define nv_hlist_for_each_entry(pos, head, member) \
-    hlist_for_each_entry(pos, head, member)
-#else
-#if !defined(hlist_entry_safe)
-#define hlist_entry_safe(ptr, type, member) \
-    (ptr) ? hlist_entry(ptr, type, member) : NULL
-#endif
-
-#define nv_hlist_for_each_entry(pos, head, member)                          \
-    for (pos = hlist_entry_safe((head)->first, typeof(*(pos)), member);     \
-         pos;                                                               \
-         pos = hlist_entry_safe((pos)->member.next, typeof(*(pos)), member))
-#endif
-#endif // NV_HLIST_FOR_EACH_ENTRY_ARGUMENT_COUNT
-
 #endif // __NV_LIST_HELPERS_H__
--- a/kernel-open/common/inc/nv-lock.h
+++ b/kernel-open/common/inc/nv-lock.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2017 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2017-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -29,19 +29,15 @@
 #include <linux/spinlock.h>
 #include <linux/rwsem.h>
 #include <linux/sched.h> /* signal_pending, cond_resched */
+#include <linux/semaphore.h>

 #if defined(NV_LINUX_SCHED_SIGNAL_H_PRESENT)
 #include <linux/sched/signal.h>     /* signal_pending for kernels >= 4.11 */
 #endif

-#if defined(NV_LINUX_SEMAPHORE_H_PRESENT)
-#include <linux/semaphore.h>
-#else
-#include <asm/semaphore.h>
-#endif
-
 #if defined(CONFIG_PREEMPT_RT) || defined(CONFIG_PREEMPT_RT_FULL)
 typedef raw_spinlock_t            nv_spinlock_t;
+#define NV_DEFINE_SPINLOCK(lock)  DEFINE_RAW_SPINLOCK(lock)
 #define NV_SPIN_LOCK_INIT(lock)   raw_spin_lock_init(lock)
 #define NV_SPIN_LOCK_IRQ(lock)    raw_spin_lock_irq(lock)
 #define NV_SPIN_UNLOCK_IRQ(lock)  raw_spin_unlock_irq(lock)
@@ -52,6 +48,7 @@ typedef raw_spinlock_t            nv_spinlock_t;
 #define NV_SPIN_UNLOCK_WAIT(lock) raw_spin_unlock_wait(lock)
 #else
 typedef spinlock_t                nv_spinlock_t;
+#define NV_DEFINE_SPINLOCK(lock)  DEFINE_SPINLOCK(lock)
 #define NV_SPIN_LOCK_INIT(lock)   spin_lock_init(lock)
 #define NV_SPIN_LOCK_IRQ(lock)    spin_lock_irq(lock)
 #define NV_SPIN_UNLOCK_IRQ(lock)  spin_unlock_irq(lock)
@@ -62,20 +59,7 @@ typedef spinlock_t                nv_spinlock_t;
 #define NV_SPIN_UNLOCK_WAIT(lock) spin_unlock_wait(lock)
 #endif

-#if defined(NV_CONFIG_PREEMPT_RT)
-#define NV_INIT_SEMA(sema, val) sema_init(sema,val)
-#else
-#if !defined(__SEMAPHORE_INITIALIZER) && defined(__COMPAT_SEMAPHORE_INITIALIZER)
-#define __SEMAPHORE_INITIALIZER __COMPAT_SEMAPHORE_INITIALIZER
-#endif
-#define NV_INIT_SEMA(sema, val)                    \
-    {                                              \
-        struct semaphore __sema =                  \
-            __SEMAPHORE_INITIALIZER(*(sema), val); \
-        *(sema) = __sema;                          \
-    }
-#endif
-#define NV_INIT_MUTEX(mutex) NV_INIT_SEMA(mutex, 1)
+#define NV_INIT_MUTEX(mutex) sema_init(mutex, 1)

 static inline int nv_down_read_interruptible(struct rw_semaphore *lock)
 {
--- a/kernel-open/common/inc/nv-mm.h
+++ b/kernel-open/common/inc/nv-mm.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2016-2017 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2016-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -29,86 +29,125 @@
 typedef int vm_fault_t;
 #endif

-/* get_user_pages
+/*
+ * pin_user_pages()
 *
- * The 8-argument version of get_user_pages was deprecated by commit
- * (2016 Feb 12: cde70140fed8429acf7a14e2e2cbd3e329036653)for the non-remote case
+ * Presence of pin_user_pages() also implies the presence of unpin-user_page().
+ * Both were added in the v5.6.
+ *
+ * pin_user_pages() was added by commit eddb1c228f79
+ * ("mm/gup: introduce pin_user_pages*() and FOLL_PIN") in v5.6.
+ *
+ * Removed vmas parameter from pin_user_pages() by commit 4c630f307455
+ * ("mm/gup: remove vmas parameter from pin_user_pages()") in v6.5.
+ */
+
+#include <linux/mm.h>
+#include <linux/sched.h>
+
+/*
+ * FreeBSD's pin_user_pages's conftest breaks since pin_user_pages is an inline
+ * function. Because it simply maps to get_user_pages, we can just replace
+ * NV_PIN_USER_PAGES with NV_GET_USER_PAGES on FreeBSD
+ */
+#if defined(NV_PIN_USER_PAGES_PRESENT) && !defined(NV_BSD)
+    #if defined(NV_PIN_USER_PAGES_HAS_ARGS_VMAS)
+        #define NV_PIN_USER_PAGES(start, nr_pages, gup_flags, pages) \
+            pin_user_pages(start, nr_pages, gup_flags, pages, NULL)
+    #else
+        #define NV_PIN_USER_PAGES pin_user_pages
+    #endif // NV_PIN_USER_PAGES_HAS_ARGS_VMAS
+    #define NV_UNPIN_USER_PAGE unpin_user_page
+#else
+    #define NV_PIN_USER_PAGES NV_GET_USER_PAGES
+    #define NV_UNPIN_USER_PAGE put_page
+#endif // NV_PIN_USER_PAGES_PRESENT
+
+/*
+ * get_user_pages()
+ *
+ * The 8-argument version of get_user_pages() was deprecated by commit
+ * cde70140fed8 ("mm/gup: Overload get_user_pages() functions") in v4.6-rc1.
 * (calling get_user_pages with current and current->mm).
 *
- * Completely moved to the 6 argument version of get_user_pages -
- * 2016 Apr 4: c12d2da56d0e07d230968ee2305aaa86b93a6832
+ * Completely moved to the 6 argument version of get_user_pages() by
+ * commit c12d2da56d0e ("mm/gup: Remove the macro overload API migration
+ * helpers from the get_user*() APIs") in v4.6-rc4.
 *
- * write and force parameters were replaced with gup_flags by -
- * 2016 Oct 12: 768ae309a96103ed02eb1e111e838c87854d8b51
+ * write and force parameters were replaced with gup_flags by
+ * commit 768ae309a961 ("mm: replace get_user_pages() write/force parameters
+ * with gup_flags") in v4.9.
 *
 * A 7-argument version of get_user_pages was introduced into linux-4.4.y by
- * commit 8e50b8b07f462ab4b91bc1491b1c91bd75e4ad40 which cherry-picked the
- * replacement of the write and force parameters with gup_flags
+ * commit 8e50b8b07f462 ("mm: replace get_user_pages() write/force parameters
+ * with gup_flags") which cherry-picked the replacement of the write and
+ * force parameters with gup_flags.
+ *
+ * Removed vmas parameter from get_user_pages() by commit 54d020692b34
+ * ("mm/gup: remove unused vmas parameter from get_user_pages()") in v6.5.
 *
 */

-#if defined(NV_GET_USER_PAGES_HAS_TASK_STRUCT)
-    #if defined(NV_GET_USER_PAGES_HAS_WRITE_AND_FORCE_ARGS)
-        #define NV_GET_USER_PAGES(start, nr_pages, write, force, pages, vmas) \
-            get_user_pages(current, current->mm, start, nr_pages, write, force, pages, vmas)
-    #else
-        #include <linux/mm.h>
-        #include <linux/sched.h>
-
-        static inline long NV_GET_USER_PAGES(unsigned long start,
-                                             unsigned long nr_pages,
-                                             int write,
-                                             int force,
-                                             struct page **pages,
-                                             struct vm_area_struct **vmas)
-        {
-            unsigned int flags = 0;
-
-            if (write)
-                flags |= FOLL_WRITE;
-            if (force)
-                flags |= FOLL_FORCE;
-
-            return get_user_pages(current, current->mm, start, nr_pages, flags,
-                                  pages, vmas);
-        }
-    #endif
+#if defined(NV_GET_USER_PAGES_HAS_ARGS_FLAGS)
+    #define NV_GET_USER_PAGES get_user_pages
+#elif defined(NV_GET_USER_PAGES_HAS_ARGS_FLAGS_VMAS)
+    #define NV_GET_USER_PAGES(start, nr_pages, flags, pages) \
+        get_user_pages(start, nr_pages, flags, pages, NULL)
+#elif defined(NV_GET_USER_PAGES_HAS_ARGS_TSK_FLAGS_VMAS)
+    #define NV_GET_USER_PAGES(start, nr_pages, flags, pages) \
+        get_user_pages(current, current->mm, start, nr_pages, flags, pages, NULL)
 #else
-    #if defined(NV_GET_USER_PAGES_HAS_WRITE_AND_FORCE_ARGS)
-        #define NV_GET_USER_PAGES get_user_pages
+    static inline long NV_GET_USER_PAGES(unsigned long start,
+                                         unsigned long nr_pages,
+                                         unsigned int flags,
+                                         struct page **pages)
+    {
+        int write = flags & FOLL_WRITE;
+        int force = flags & FOLL_FORCE;
+
+    #if defined(NV_GET_USER_PAGES_HAS_ARGS_WRITE_FORCE_VMAS)
+        return get_user_pages(start, nr_pages, write, force, pages, NULL);
    #else
-        #include <linux/mm.h>
+        // NV_GET_USER_PAGES_HAS_ARGS_TSK_WRITE_FORCE_VMAS
+        return get_user_pages(current, current->mm, start, nr_pages, write,
+                              force, pages, NULL);
+    #endif // NV_GET_USER_PAGES_HAS_ARGS_WRITE_FORCE_VMAS
+    }
+#endif // NV_GET_USER_PAGES_HAS_ARGS_FLAGS

-        static inline long NV_GET_USER_PAGES(unsigned long start,
-                                             unsigned long nr_pages,
-                                             int write,
-                                             int force,
-                                             struct page **pages,
-                                             struct vm_area_struct **vmas)
-        {
-            unsigned int flags = 0;
+/*
+ * pin_user_pages_remote()
+ *
+ * pin_user_pages_remote() was added by commit eddb1c228f79
+ * ("mm/gup: introduce pin_user_pages*() and FOLL_PIN") in v5.6.
+ *
+ * pin_user_pages_remote() removed 'tsk' parameter by commit
+ * 64019a2e467a ("mm/gup: remove task_struct pointer for all gup code")
+ * in v5.9.
+ *
+ * Removed unused vmas parameter from pin_user_pages_remote() by commit
+ * 0b295316b3a9 ("mm/gup: remove unused vmas parameter from
+ * pin_user_pages_remote()") in v6.5.
+ *
+ */

-            if (write)
-                flags |= FOLL_WRITE;
-            if (force)
-                flags |= FOLL_FORCE;
-
-            return get_user_pages(start, nr_pages, flags, pages, vmas);
-        }
-    #endif
-#endif
+#if defined(NV_PIN_USER_PAGES_REMOTE_PRESENT)
+    #if defined(NV_PIN_USER_PAGES_REMOTE_HAS_ARGS_TSK_VMAS)
+        #define NV_PIN_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            pin_user_pages_remote(NULL, mm, start, nr_pages, flags, pages, NULL, locked)
+    #elif defined(NV_PIN_USER_PAGES_REMOTE_HAS_ARGS_VMAS)
+        #define NV_PIN_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            pin_user_pages_remote(mm, start, nr_pages, flags, pages, NULL, locked)
+    #else
+        #define NV_PIN_USER_PAGES_REMOTE pin_user_pages_remote
+    #endif // NV_PIN_USER_PAGES_REMOTE_HAS_ARGS_TSK_VMAS
+#else
+    #define NV_PIN_USER_PAGES_REMOTE NV_GET_USER_PAGES_REMOTE
+#endif // NV_PIN_USER_PAGES_REMOTE_PRESENT

 /*
 * get_user_pages_remote() was added by commit 1e9877902dc7
- * ("mm/gup: Introduce get_user_pages_remote()") in v4.6 (2016-02-12).
- *
- * The very next commit cde70140fed8 ("mm/gup: Overload get_user_pages()
- * functions") deprecated the 8-argument version of get_user_pages for the
- * non-remote case (calling get_user_pages with current and current->mm).
- *
- * The guidelines are: call NV_GET_USER_PAGES_REMOTE if you need the 8-argument
- * version that uses something other than current and current->mm. Use
- * NV_GET_USER_PAGES if you are refering to current and current->mm.
+ * ("mm/gup: Introduce get_user_pages_remote()") in v4.6.
 *
 * Note that get_user_pages_remote() requires the caller to hold a reference on
 * the task_struct (if non-NULL and if this API has tsk argument) and the mm_struct.
@@ -118,84 +157,72 @@ typedef int vm_fault_t;
 *
 * get_user_pages_remote() write/force parameters were replaced
 * with gup_flags by commit 9beae1ea8930 ("mm: replace get_user_pages_remote()
- * write/force parameters with gup_flags") in v4.9 (2016-10-13).
+ * write/force parameters with gup_flags") in v4.9.
 *
 * get_user_pages_remote() added 'locked' parameter by commit 5b56d49fc31d
- * ("mm: add locked parameter to get_user_pages_remote()") in
- * v4.10 (2016-12-14).
+ * ("mm: add locked parameter to get_user_pages_remote()") in v4.10.
 *
 * get_user_pages_remote() removed 'tsk' parameter by
 * commit 64019a2e467a ("mm/gup: remove task_struct pointer for
- * all gup code") in v5.9-rc1 (2020-08-11).
+ * all gup code") in v5.9.
+ *
+ * Removed vmas parameter from get_user_pages_remote() by commit ca5e863233e8
+ * ("mm/gup: remove vmas parameter from get_user_pages_remote()") in v6.5.
 *
 */

 #if defined(NV_GET_USER_PAGES_REMOTE_PRESENT)
-    #if defined(NV_GET_USER_PAGES_REMOTE_HAS_WRITE_AND_FORCE_ARGS)
-        #define NV_GET_USER_PAGES_REMOTE    get_user_pages_remote
+    #if defined(NV_GET_USER_PAGES_REMOTE_HAS_ARGS_FLAGS_LOCKED)
+        #define NV_GET_USER_PAGES_REMOTE get_user_pages_remote
+
+    #elif defined(NV_GET_USER_PAGES_REMOTE_HAS_ARGS_FLAGS_LOCKED_VMAS)
+        #define NV_GET_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            get_user_pages_remote(mm, start, nr_pages, flags, pages, NULL, locked)
+
+    #elif defined(NV_GET_USER_PAGES_REMOTE_HAS_ARGS_TSK_FLAGS_LOCKED_VMAS)
+        #define NV_GET_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            get_user_pages_remote(NULL, mm, start, nr_pages, flags, pages, NULL, locked)
+
+    #elif defined(NV_GET_USER_PAGES_REMOTE_HAS_ARGS_TSK_FLAGS_VMAS)
+        #define NV_GET_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            get_user_pages_remote(NULL, mm, start, nr_pages, flags, pages, NULL)
+
    #else
-        static inline long NV_GET_USER_PAGES_REMOTE(struct task_struct *tsk,
-                                                    struct mm_struct *mm,
+        // NV_GET_USER_PAGES_REMOTE_HAS_ARGS_TSK_WRITE_FORCE_VMAS
+        static inline long NV_GET_USER_PAGES_REMOTE(struct mm_struct *mm,
                                                    unsigned long start,
                                                    unsigned long nr_pages,
-                                                    int write,
-                                                    int force,
+                                                    unsigned int flags,
                                                    struct page **pages,
-                                                    struct vm_area_struct **vmas)
+                                                    int *locked)
        {
-            unsigned int flags = 0;
-
-            if (write)
-                flags |= FOLL_WRITE;
-            if (force)
-                flags |= FOLL_FORCE;
-
-        #if defined(NV_GET_USER_PAGES_REMOTE_HAS_LOCKED_ARG)
-            #if defined (NV_GET_USER_PAGES_REMOTE_HAS_TSK_ARG)
-               return get_user_pages_remote(tsk, mm, start, nr_pages, flags,
-                                            pages, vmas, NULL);
-            #else
-               return get_user_pages_remote(mm, start, nr_pages, flags,
-                                            pages, vmas, NULL);
-            #endif
-
-        #else
-
-               return get_user_pages_remote(tsk, mm, start, nr_pages, flags,
-                                            pages, vmas);
-
-        #endif
+            int write = flags & FOLL_WRITE;
+            int force = flags & FOLL_FORCE;

+            return get_user_pages_remote(NULL, mm, start, nr_pages, write, force,
+                                         pages, NULL);
        }
-    #endif
+    #endif // NV_GET_USER_PAGES_REMOTE_HAS_ARGS_FLAGS_LOCKED
 #else
-    #if defined(NV_GET_USER_PAGES_HAS_WRITE_AND_FORCE_ARGS)
-        #define NV_GET_USER_PAGES_REMOTE    get_user_pages
-    #else
-        #include <linux/mm.h>
-        #include <linux/sched.h>
-
-        static inline long NV_GET_USER_PAGES_REMOTE(struct task_struct *tsk,
-                                                    struct mm_struct *mm,
+    #if defined(NV_GET_USER_PAGES_HAS_ARGS_TSK_WRITE_FORCE_VMAS)
+        static inline long NV_GET_USER_PAGES_REMOTE(struct mm_struct *mm,
                                                    unsigned long start,
                                                    unsigned long nr_pages,
-                                                    int write,
-                                                    int force,
+                                                    unsigned int flags,
                                                    struct page **pages,
-                                                    struct vm_area_struct **vmas)
+                                                    int *locked)
        {
-            unsigned int flags = 0;
+            int write = flags & FOLL_WRITE;
+            int force = flags & FOLL_FORCE;

-            if (write)
-                flags |= FOLL_WRITE;
-            if (force)
-                flags |= FOLL_FORCE;
-
-            return get_user_pages(tsk, mm, start, nr_pages, flags, pages, vmas);
+            return get_user_pages(NULL, mm, start, nr_pages, write, force, pages, NULL);
        }
-    #endif
-#endif

+    #else
+        #define NV_GET_USER_PAGES_REMOTE(mm, start, nr_pages, flags, pages, locked) \
+            get_user_pages(NULL, mm, start, nr_pages, flags, pages, NULL)
+    #endif // NV_GET_USER_PAGES_HAS_ARGS_TSK_WRITE_FORCE_VMAS
+#endif // NV_GET_USER_PAGES_REMOTE_PRESENT

 /*
 * The .virtual_address field was effectively renamed to .address, by these
@@ -270,4 +297,22 @@ static inline struct rw_semaphore *nv_mmap_get_lock(struct mm_struct *mm)
 #endif
 }

+static inline void nv_vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags)
+{
+#if defined(NV_VM_AREA_STRUCT_HAS_CONST_VM_FLAGS)
+    vm_flags_set(vma, flags);
+#else
+    vma->vm_flags |= flags;
+#endif
+}
+
+static inline void nv_vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags)
+{
+#if defined(NV_VM_AREA_STRUCT_HAS_CONST_VM_FLAGS)
+    vm_flags_clear(vma, flags);
+#else
+    vma->vm_flags &= ~flags;
+#endif
+}
+
 #endif // __NV_MM_H__
--- a/kernel-open/common/inc/nv-pci.h
+++ b/kernel-open/common/inc/nv-pci.h
@@ -27,16 +27,8 @@
 #include <linux/pci.h>
 #include "nv-linux.h"

-#if defined(NV_DEV_IS_PCI_PRESENT)
-#define nv_dev_is_pci(dev) dev_is_pci(dev)
-#else
-/*
- * Non-PCI devices are only supported on kernels which expose the
- * dev_is_pci() function. For older kernels, we only support PCI
- * devices, hence returning true to take all the PCI code paths.
- */
-#define nv_dev_is_pci(dev) (true)
-#endif
+#define NV_GPU_BAR1 1
+#define NV_GPU_BAR3 3

 int nv_pci_register_driver(void);
 void nv_pci_unregister_driver(void);
--- a/kernel-open/common/inc/nv-pgprot.h
+++ b/kernel-open/common/inc/nv-pgprot.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2015 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2015-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -60,6 +60,7 @@ static inline pgprot_t pgprot_modify_writecombine(pgprot_t old_prot)
 #endif /* !defined(NV_VMWARE) */

 #if defined(NVCPU_AARCH64)
+extern NvBool nvos_is_chipset_io_coherent(void);
 /*
 * Don't rely on the kernel's definition of pgprot_noncached(), as on 64-bit
 * ARM that's not for system memory, but device memory instead. For I/O cache
@@ -78,13 +79,8 @@ static inline pgprot_t pgprot_modify_writecombine(pgprot_t old_prot)

 #define NV_PGPROT_UNCACHED_DEVICE(old_prot)     pgprot_noncached(old_prot)
 #if defined(NVCPU_AARCH64)
-#if defined(NV_MT_DEVICE_GRE_PRESENT)
-#define NV_PROT_WRITE_COMBINED_DEVICE   (PROT_DEFAULT | PTE_PXN | PTE_UXN |   \
-                                         PTE_ATTRINDX(MT_DEVICE_GRE))
-#else
 #define NV_PROT_WRITE_COMBINED_DEVICE   (PROT_DEFAULT | PTE_PXN | PTE_UXN |   \
                                         PTE_ATTRINDX(MT_DEVICE_nGnRE))
-#endif
 #define NV_PGPROT_WRITE_COMBINED_DEVICE(old_prot)                             \
    __pgprot_modify(old_prot, PTE_ATTRINDX_MASK, NV_PROT_WRITE_COMBINED_DEVICE)
 #define NV_PGPROT_WRITE_COMBINED(old_prot)      NV_PGPROT_UNCACHED(old_prot)
@@ -124,6 +120,13 @@ static inline pgprot_t pgprot_modify_writecombine(pgprot_t old_prot)
 #define NV_PGPROT_WRITE_COMBINED(old_prot)    old_prot
 #define NV_PGPROT_READ_ONLY(old_prot)                                         \
    __pgprot(pgprot_val((old_prot)) & ~NV_PAGE_RW)
+#elif defined(NVCPU_RISCV64)
+#define NV_PGPROT_WRITE_COMBINED_DEVICE(old_prot)                             \
+    pgprot_writecombine(old_prot)
+/* Don't attempt to mark sysmem pages as write combined on riscv */
+#define NV_PGPROT_WRITE_COMBINED(old_prot)     old_prot
+#define NV_PGPROT_READ_ONLY(old_prot)                                         \
+            __pgprot(pgprot_val((old_prot)) & ~_PAGE_WRITE)
 #else
 /* Writecombine is not supported */
 #undef NV_PGPROT_WRITE_COMBINED_DEVICE(old_prot)
--- a/kernel-open/common/inc/nv-procfs-utils.h
+++ b/kernel-open/common/inc/nv-procfs-utils.h
@@ -74,21 +74,8 @@ typedef struct file_operations nv_proc_ops_t;
        __entry;                                                         \
    })

-/*
- * proc_mkdir_mode exists in Linux 2.6.9, but isn't exported until Linux 3.0.
- * Use the older interface instead unless the newer interface is necessary.
- */
-#if defined(NV_PROC_REMOVE_PRESENT)
 # define NV_PROC_MKDIR_MODE(name, mode, parent)                \
    proc_mkdir_mode(name, mode, parent)
-#else
-# define NV_PROC_MKDIR_MODE(name, mode, parent)                \
-   ({                                                          \
-        struct proc_dir_entry *__entry;                        \
-        __entry = create_proc_entry(name, mode, parent);       \
-        __entry;                                               \
-    })
-#endif

 #define NV_CREATE_PROC_DIR(name,parent)                        \
   ({                                                          \
@@ -104,17 +91,25 @@ typedef struct file_operations nv_proc_ops_t;
 #define NV_PDE_DATA(inode) PDE_DATA(inode)
 #endif

-#if defined(NV_PROC_REMOVE_PRESENT)
-# define NV_REMOVE_PROC_ENTRY(entry)                           \
-    proc_remove(entry);
-#else
-# define NV_REMOVE_PROC_ENTRY(entry)                           \
-    remove_proc_entry(entry->name, entry->parent);
-#endif
-
-void nv_procfs_unregister_all(struct proc_dir_entry *entry,
-                              struct proc_dir_entry *delimiter);
 #define NV_DEFINE_SINGLE_PROCFS_FILE_HELPER(name, lock)                     \
+    static ssize_t nv_procfs_read_lock_##name(                              \
+        struct file *file,                                                  \
+        char __user *buf,                                                   \
+        size_t size,                                                        \
+        loff_t *ppos                                                        \
+    )                                                                       \
+    {                                                                       \
+        int ret;                                                            \
+        ret = nv_down_read_interruptible(&lock);                            \
+        if (ret < 0)                                                        \
+        {                                                                   \
+            return ret;                                                     \
+        }                                                                   \
+        size = seq_read(file, buf, size, ppos);                             \
+        up_read(&lock);                                                     \
+        return size;                                                        \
+    }                                                                       \
+                                                                            \
    static int nv_procfs_open_##name(                                       \
        struct inode *inode,                                                \
        struct file *filep                                                  \
@@ -127,11 +122,6 @@ void nv_procfs_unregister_all(struct proc_dir_entry *entry,
        {                                                                   \
            return ret;                                                     \
        }                                                                   \
-        ret = nv_down_read_interruptible(&lock);                            \
-        if (ret < 0)                                                        \
-        {                                                                   \
-            single_release(inode, filep);                                   \
-        }                                                                   \
        return ret;                                                         \
    }                                                                       \
                                                                            \
@@ -140,7 +130,6 @@ void nv_procfs_unregister_all(struct proc_dir_entry *entry,
        struct file *filep                                                  \
    )                                                                       \
    {                                                                       \
-        up_read(&lock);                                                     \
        return single_release(inode, filep);                                \
    }

@@ -150,46 +139,7 @@ void nv_procfs_unregister_all(struct proc_dir_entry *entry,
    static const nv_proc_ops_t nv_procfs_##name##_fops = {                  \
        NV_PROC_OPS_SET_OWNER()                                             \
        .NV_PROC_OPS_OPEN    = nv_procfs_open_##name,                       \
-        .NV_PROC_OPS_READ    = seq_read,                                    \
-        .NV_PROC_OPS_LSEEK   = seq_lseek,                                   \
-        .NV_PROC_OPS_RELEASE = nv_procfs_release_##name,                    \
-    };
-
-
-#define NV_DEFINE_SINGLE_PROCFS_FILE_READ_WRITE(name, lock,                 \
-write_callback)                                                             \
-    NV_DEFINE_SINGLE_PROCFS_FILE_HELPER(name, lock)                         \
-                                                                            \
-    static ssize_t nv_procfs_write_##name(                                  \
-        struct file *file,                                                  \
-        const char __user *buf,                                             \
-        size_t size,                                                        \
-        loff_t *ppos                                                        \
-    )                                                                       \
-    {                                                                       \
-        ssize_t ret;                                                        \
-        struct seq_file *s;                                                 \
-                                                                            \
-        s = file->private_data;                                             \
-        if (s == NULL)                                                      \
-        {                                                                   \
-            return -EIO;                                                    \
-        }                                                                   \
-                                                                            \
-        ret = write_callback(s, buf + *ppos, size - *ppos);                 \
-        if (ret == 0)                                                       \
-        {                                                                   \
-            /* avoid infinite loop */                                       \
-            ret = -EIO;                                                     \
-        }                                                                   \
-        return ret;                                                         \
-    }                                                                       \
-                                                                            \
-    static const nv_proc_ops_t nv_procfs_##name##_fops = {                  \
-        NV_PROC_OPS_SET_OWNER()                                             \
-        .NV_PROC_OPS_OPEN    = nv_procfs_open_##name,                       \
-        .NV_PROC_OPS_READ    = seq_read,                                    \
-        .NV_PROC_OPS_WRITE   = nv_procfs_write_##name,                      \
+        .NV_PROC_OPS_READ    = nv_procfs_read_lock_##name,                  \
        .NV_PROC_OPS_LSEEK   = seq_lseek,                                   \
        .NV_PROC_OPS_RELEASE = nv_procfs_release_##name,                    \
    };
--- a/kernel-open/common/inc/nv-proto.h
+++ b/kernel-open/common/inc/nv-proto.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 1999-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 1999-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -25,13 +25,8 @@
 #define _NV_PROTO_H_

 #include "nv-pci.h"
-#include "nv-register-module.h"
-
-
-

 extern const char *nv_device_name;
-extern nvidia_module_t nv_fops;

 void        nv_acpi_register_notifier   (nv_linux_state_t *);
 void        nv_acpi_unregister_notifier (nv_linux_state_t *);
@@ -57,8 +52,6 @@ void        nv_free_contig_pages        (nv_alloc_t *);
 NV_STATUS   nv_alloc_system_pages       (nv_state_t *, nv_alloc_t *);
 void        nv_free_system_pages        (nv_alloc_t *);

-void        nv_address_space_init_once  (struct address_space *mapping);
-
 int         nv_uvm_init                 (void);
 void        nv_uvm_exit                 (void);
 NV_STATUS   nv_uvm_suspend              (void);
@@ -66,6 +59,8 @@ NV_STATUS   nv_uvm_resume               (void);
 void        nv_uvm_notify_start_device  (const NvU8 *uuid);
 void        nv_uvm_notify_stop_device   (const NvU8 *uuid);
 NV_STATUS   nv_uvm_event_interrupt      (const NvU8 *uuid);
+NV_STATUS   nv_uvm_drain_P2P            (const NvU8 *uuid);
+NV_STATUS   nv_uvm_resume_P2P           (const NvU8 *uuid);

 /* Move these to nv.h once implemented by other UNIX platforms */
 NvBool      nvidia_get_gpuid_list       (NvU32 *gpu_ids, NvU32 *gpu_count);
@@ -91,8 +86,11 @@ void          nv_shutdown_adapter(nvidia_stack_t *, nv_state_t *, nv_linux_state
 void          nv_dev_free_stacks(nv_linux_state_t *);
 NvBool        nv_lock_init_locks(nvidia_stack_t *, nv_state_t *);
 void          nv_lock_destroy_locks(nvidia_stack_t *, nv_state_t *);
-void          nv_linux_add_device_locked(nv_linux_state_t *);
+int           nv_linux_add_device_locked(nv_linux_state_t *);
 void          nv_linux_remove_device_locked(nv_linux_state_t *);
 NvBool        nv_acpi_power_resource_method_present(struct pci_dev *);

+int           nv_linux_init_open_q(nv_linux_state_t *);
+void          nv_linux_stop_open_q(nv_linux_state_t *);
+
 #endif /* _NV_PROTO_H_ */
--- a/kernel-open/common/inc/nv.h
+++ b/kernel-open/common/inc/nv.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 1999-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 1999-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -40,23 +40,18 @@
 #include <nvstatus.h>
 #include "nv_stdarg.h"
 #include <nv-caps.h>
+#include <nv-firmware.h>
 #include <nv-ioctl.h>
+#include <nv-ioctl-numa.h>
 #include <nvmisc.h>
+#include <os/nv_memory_area.h>

 extern nv_cap_t *nvidia_caps_root;

 extern const NvBool nv_is_rm_firmware_supported_os;

-
-
-
-
-
 #include <nv-kernel-interface-api.h>

-/* NVIDIA's reserved major character device number (Linux). */
-#define NV_MAJOR_DEVICE_NUMBER 195
-
 #define GPU_UUID_LEN    (16)

 /*
@@ -116,15 +111,15 @@ typedef enum _TEGRASOC_WHICH_CLK
    TEGRASOC_WHICH_CLK_DSIPLL_CLKOUTPN,
    TEGRASOC_WHICH_CLK_DSIPLL_CLKOUTA,
    TEGRASOC_WHICH_CLK_SPPLL0_VCO,
-    TEGRASOC_WHICH_CLK_SPPLL0_CLKOUTPN,
    TEGRASOC_WHICH_CLK_SPPLL0_CLKOUTA,
    TEGRASOC_WHICH_CLK_SPPLL0_CLKOUTB,
+    TEGRASOC_WHICH_CLK_SPPLL0_CLKOUTPN,
+    TEGRASOC_WHICH_CLK_SPPLL1_CLKOUTPN,
+    TEGRASOC_WHICH_CLK_SPPLL0_DIV27,
+    TEGRASOC_WHICH_CLK_SPPLL1_DIV27,
    TEGRASOC_WHICH_CLK_SPPLL0_DIV10,
    TEGRASOC_WHICH_CLK_SPPLL0_DIV25,
-    TEGRASOC_WHICH_CLK_SPPLL0_DIV27,
    TEGRASOC_WHICH_CLK_SPPLL1_VCO,
-    TEGRASOC_WHICH_CLK_SPPLL1_CLKOUTPN,
-    TEGRASOC_WHICH_CLK_SPPLL1_DIV27,
    TEGRASOC_WHICH_CLK_VPLL0_REF,
    TEGRASOC_WHICH_CLK_VPLL0,
    TEGRASOC_WHICH_CLK_VPLL1,
@@ -138,7 +133,7 @@ typedef enum _TEGRASOC_WHICH_CLK
    TEGRASOC_WHICH_CLK_DSI_PIXEL,
    TEGRASOC_WHICH_CLK_PRE_SOR0,
    TEGRASOC_WHICH_CLK_PRE_SOR1,
-    TEGRASOC_WHICH_CLK_DP_LINK_REF,
+    TEGRASOC_WHICH_CLK_DP_LINKA_REF,
    TEGRASOC_WHICH_CLK_SOR_LINKA_INPUT,
    TEGRASOC_WHICH_CLK_SOR_LINKA_AFIFO,
    TEGRASOC_WHICH_CLK_SOR_LINKA_AFIFO_M,
@@ -149,7 +144,7 @@ typedef enum _TEGRASOC_WHICH_CLK
    TEGRASOC_WHICH_CLK_PLLHUB,
    TEGRASOC_WHICH_CLK_SOR0,
    TEGRASOC_WHICH_CLK_SOR1,
-    TEGRASOC_WHICH_CLK_SOR_PAD_INPUT,
+    TEGRASOC_WHICH_CLK_SOR_PADA_INPUT,
    TEGRASOC_WHICH_CLK_PRE_SF0,
    TEGRASOC_WHICH_CLK_SF0,
    TEGRASOC_WHICH_CLK_SF1,
@@ -165,8 +160,14 @@ typedef enum _TEGRASOC_WHICH_CLK
    TEGRASOC_WHICH_CLK_MAUD,
    TEGRASOC_WHICH_CLK_AZA_2XBIT,
    TEGRASOC_WHICH_CLK_AZA_BIT,
-    TEGRA234_CLK_MIPI_CAL,
-    TEGRA234_CLK_UART_FST_MIPI_CAL,
+    TEGRASOC_WHICH_CLK_MIPI_CAL,
+    TEGRASOC_WHICH_CLK_UART_FST_MIPI_CAL,
+    TEGRASOC_WHICH_CLK_SOR0_DIV,
+    TEGRASOC_WHICH_CLK_DISP_ROOT,
+    TEGRASOC_WHICH_CLK_HUB_ROOT,
+    TEGRASOC_WHICH_CLK_PLLA_DISP,
+    TEGRASOC_WHICH_CLK_PLLA_DISPHUB,
+    TEGRASOC_WHICH_CLK_PLLA,
    TEGRASOC_WHICH_CLK_MAX, // TEGRASOC_WHICH_CLK_MAX is defined for boundary checks only.
 } TEGRASOC_WHICH_CLK;

@@ -221,7 +222,6 @@ typedef struct
 #define NV_RM_PAGE_MASK     (NV_RM_PAGE_SIZE - 1)

 #define NV_RM_TO_OS_PAGE_SHIFT      (os_page_shift - NV_RM_PAGE_SHIFT)
-#define NV_RM_PAGES_PER_OS_PAGE     (1U << NV_RM_TO_OS_PAGE_SHIFT)
 #define NV_RM_PAGES_TO_OS_PAGES(count) \
    ((((NvUPtr)(count)) >> NV_RM_TO_OS_PAGE_SHIFT) + \
     ((((count) & ((1 << NV_RM_TO_OS_PAGE_SHIFT) - 1)) != 0) ? 1 : 0))
@@ -280,12 +280,12 @@ typedef struct nv_usermap_access_params_s
    NvU64    offset;
    NvU64   *page_array;
    NvU64    num_pages;
-    NvU64    mmap_start;
-    NvU64    mmap_size;
+    MemoryArea memArea;
    NvU64    access_start;
    NvU64    access_size;
    NvU64    remap_prot_extra;
    NvBool   contig;
+    NvU32    caching;
 } nv_usermap_access_params_t;

 /*
@@ -296,21 +296,22 @@ typedef struct nv_alloc_mapping_context_s {
    NvU64  page_index;
    NvU64 *page_array;
    NvU64  num_pages;
-    NvU64  mmap_start;
-    NvU64  mmap_size;
+    MemoryArea memArea;
    NvU64  access_start;
    NvU64  access_size;
    NvU64  remap_prot_extra;
    NvU32  prot;
    NvBool valid;
+    NvU32  caching;
 } nv_alloc_mapping_context_t;

 typedef enum
 {
-    NV_SOC_IRQ_DISPLAY_TYPE,
+    NV_SOC_IRQ_DISPLAY_TYPE = 0x1,
    NV_SOC_IRQ_DPAUX_TYPE,
    NV_SOC_IRQ_GPIO_TYPE,
    NV_SOC_IRQ_HDACODEC_TYPE,
+    NV_SOC_IRQ_TCPC2DISP_TYPE,
    NV_SOC_IRQ_INVALID_TYPE
 } nv_soc_irq_type_t;

@@ -325,12 +326,18 @@ typedef struct nv_soc_irq_info_s {
        NvU32 gpio_num;
        NvU32 dpaux_instance;
    } irq_data;
+    NvS32 ref_count;
 } nv_soc_irq_info_t;

-#define NV_MAX_SOC_IRQS              6
+#define NV_MAX_SOC_IRQS              10
 #define NV_MAX_DPAUX_NUM_DEVICES     4
-#define NV_MAX_SOC_DPAUX_NUM_DEVICES 2 // From SOC_DEV_MAPPING

+#define NV_MAX_SOC_DPAUX_NUM_DEVICES 2
+
+
+#define NV_IGPU_LEGACY_STALL_IRQ     70
+#define NV_IGPU_MAX_STALL_IRQS       3
+#define NV_IGPU_MAX_NONSTALL_IRQS    1
 /*
 * per device state
 */
@@ -338,6 +345,12 @@ typedef struct nv_soc_irq_info_s {
 /* DMA-capable device data, defined by kernel interface layer */
 typedef struct nv_dma_device nv_dma_device_t;

+typedef struct nv_phys_addr_range
+{
+    NvU64 addr;
+    NvU64 len;
+} nv_phys_addr_range_t;
+
 typedef struct nv_state_t
 {
    void  *priv;                    /* private data */
@@ -367,6 +380,8 @@ typedef struct nv_state_t
    nv_aperture_t *hdacodec_regs;
    nv_aperture_t *mipical_regs;
    nv_aperture_t *fb, ud;
+    nv_aperture_t *simregs;
+    nv_aperture_t *emc_regs;

    NvU32  num_dpaux_instance;
    NvU32  interrupt_line;
@@ -375,10 +390,17 @@ typedef struct nv_state_t
    NvS32 current_soc_irq;
    NvU32 num_soc_irqs;
    NvU32 hdacodec_irq;
+    NvU32 tcpc2disp_irq;
    NvU8 *soc_dcb_blob;
    NvU32 soc_dcb_size;
    NvU32 disp_sw_soc_chip_id;
+    NvBool soc_is_dpalt_mode_supported;

+    NvU32 igpu_stall_irq[NV_IGPU_MAX_STALL_IRQS];
+    NvU32 igpu_nonstall_irq;
+    NvU32 num_stall_irqs;
+    NvU64 dma_mask;
+    
    NvBool primary_vga;

    NvU32 sim_env;
@@ -424,9 +446,6 @@ typedef struct nv_state_t
    /* Variable to force allocation of 32-bit addressable memory */
    NvBool force_dma32_alloc;

-    /* Variable to track if device has entered dynamic power state */
-    NvBool dynamic_power_entered;
-
    /* PCI power state should be D0 during system suspend */
    NvBool d0_state_in_suspend;

@@ -448,37 +467,64 @@ typedef struct nv_state_t
        NvHandle hDisp;
    } rmapi;

-    /* Bool to check if ISO iommu enabled */
-    NvBool iso_iommu_present;
-
    /* Bool to check if dma-buf is supported */
    NvBool dma_buf_supported;

-    NvBool printed_openrm_enable_unsupported_gpus_error;
+    /* Check if NVPCF DSM function is implemented under NVPCF or GPU device scope */
+    NvBool nvpcf_dsm_in_gpu_scope;

+    /* Bool to check if the device received a shutdown notification */
+    NvBool is_shutdown;
+
+    /* Bool to check if the GPU has a coherent sysmem link */
+    NvBool coherent;
+
+    /*
+     * NUMA node ID of the CPU to which the GPU is attached.
+     * Holds NUMA_NO_NODE on platforms that don't support NUMA configuration.
+     */
+    NvS32 cpu_numa_node_id;
+
+    struct {
+        /* Bool to check if ISO iommu enabled */
+        NvBool iso_iommu_present;
+        /* Bool to check if NISO iommu enabled */
+        NvBool niso_iommu_present;
+        /* Display SMMU Stream IDs */
+        NvU32 dispIsoStreamId;
+        NvU32 dispNisoStreamId;
+    } iommus;
 } nv_state_t;

-// These define need to be in sync with defines in system.h
-#define OS_TYPE_LINUX   0x1
-#define OS_TYPE_FREEBSD 0x2
-#define OS_TYPE_SUNOS   0x3
-#define OS_TYPE_VMWARE  0x4
+#define NVFP_TYPE_NONE       0x0
+#define NVFP_TYPE_REFCOUNTED 0x1
+#define NVFP_TYPE_REGISTERED 0x2

 struct nv_file_private_t
 {
    NvHandle *handles;
    NvU16 maxHandles;
    NvU32 deviceInstance;
+    NvU32 gpuInstanceId;
    NvU8 metadata[64];

    nv_file_private_t *ctl_nvfp;
    void *ctl_nvfp_priv;
+    NvU32 register_or_refcount;
+
+    //
+    // True if a client or an event was ever allocated on this fd.
+    // If false, RMAPI cleanup is skipped.
+    //
+    NvBool bCleanupRmapi;
 };

 // Forward define the gpu ops structures
 typedef struct gpuSession                           *nvgpuSessionHandle_t;
 typedef struct gpuDevice                            *nvgpuDeviceHandle_t;
 typedef struct gpuAddressSpace                      *nvgpuAddressSpaceHandle_t;
+typedef struct gpuTsg                               *nvgpuTsgHandle_t;
+typedef struct UvmGpuTsgAllocParams_tag              nvgpuTsgAllocParams_t;
 typedef struct gpuChannel                           *nvgpuChannelHandle_t;
 typedef struct UvmGpuChannelInfo_tag                *nvgpuChannelInfo_t;
 typedef struct UvmGpuChannelAllocParams_tag          nvgpuChannelAllocParams_t;
@@ -488,6 +534,7 @@ typedef struct UvmGpuAddressSpaceInfo_tag           *nvgpuAddressSpaceInfo_t;
 typedef struct UvmGpuAllocInfo_tag                  *nvgpuAllocInfo_t;
 typedef struct UvmGpuP2PCapsParams_tag              *nvgpuP2PCapsParams_t;
 typedef struct UvmGpuFbInfo_tag                     *nvgpuFbInfo_t;
+typedef struct UvmGpuNvlinkInfo_tag                 *nvgpuNvlinkInfo_t;
 typedef struct UvmGpuEccInfo_tag                    *nvgpuEccInfo_t;
 typedef struct UvmGpuFaultInfo_tag                  *nvgpuFaultInfo_t;
 typedef struct UvmGpuAccessCntrInfo_tag             *nvgpuAccessCntrInfo_t;
@@ -498,14 +545,16 @@ typedef struct UvmPmaAllocationOptions_tag          *nvgpuPmaAllocationOptions_t
 typedef struct UvmPmaStatistics_tag                 *nvgpuPmaStatistics_t;
 typedef struct UvmGpuMemoryInfo_tag                 *nvgpuMemoryInfo_t;
 typedef struct UvmGpuExternalMappingInfo_tag        *nvgpuExternalMappingInfo_t;
+typedef struct UvmGpuExternalPhysAddrInfo_tag       *nvgpuExternalPhysAddrInfo_t;
 typedef struct UvmGpuChannelResourceInfo_tag        *nvgpuChannelResourceInfo_t;
 typedef struct UvmGpuChannelInstanceInfo_tag        *nvgpuChannelInstanceInfo_t;
 typedef struct UvmGpuChannelResourceBindParams_tag  *nvgpuChannelResourceBindParams_t;
 typedef struct UvmGpuPagingChannelAllocParams_tag    nvgpuPagingChannelAllocParams_t;
 typedef struct UvmGpuPagingChannel_tag              *nvgpuPagingChannelHandle_t;
 typedef struct UvmGpuPagingChannelInfo_tag          *nvgpuPagingChannelInfo_t;
-typedef NV_STATUS (*nvPmaEvictPagesCallback)(void *, NvU32, NvU64 *, NvU32, NvU64, NvU64);
-typedef NV_STATUS (*nvPmaEvictRangeCallback)(void *, NvU64, NvU64);
+typedef enum   UvmPmaGpuMemoryType_tag               nvgpuGpuMemoryType_t;
+typedef NV_STATUS (*nvPmaEvictPagesCallback)(void *, NvU64, NvU64 *, NvU32, NvU64, NvU64, nvgpuGpuMemoryType_t);
+typedef NV_STATUS (*nvPmaEvictRangeCallback)(void *, NvU64, NvU64, nvgpuGpuMemoryType_t);

 /*
 * flags
@@ -520,7 +569,7 @@ typedef NV_STATUS (*nvPmaEvictRangeCallback)(void *, NvU64, NvU64);
 #define NV_FLAG_USES_MSIX              0x0040
 #define NV_FLAG_PASSTHRU               0x0080
 #define NV_FLAG_SUSPENDED              0x0100
-// Unused                              0x0200
+#define NV_FLAG_SOC_IGPU               0x0200
 // Unused                              0x0400
 #define NV_FLAG_PERSISTENT_SW_STATE    0x0800
 #define NV_FLAG_IN_RECOVERY            0x1000
@@ -557,11 +606,14 @@ typedef enum
    NV_POWER_STATE_RUNNING
 } nv_power_state_t;

-typedef enum
+typedef struct
 {
-    NV_FIRMWARE_GSP,
-    NV_FIRMWARE_GSP_LOG
-} nv_firmware_t;
+    const char *vidmem_power_status;
+    const char *dynamic_power_status;
+    const char *gc6_support;
+    const char *gcoff_support;
+    const char *s0ix_status;
+} nv_power_info_t;

 #define NV_PRIMARY_VGA(nv)      ((nv)->primary_vga)

@@ -569,18 +621,25 @@ typedef enum
 #define NV_IS_SOC_DISPLAY_DEVICE(nv)    \
        ((nv)->flags & NV_FLAG_SOC_DISPLAY)

+#define NV_IS_SOC_IGPU_DEVICE(nv)    \
+        ((nv)->flags & NV_FLAG_SOC_IGPU)
+
 #define NV_IS_DEVICE_IN_SURPRISE_REMOVAL(nv)    \
        (((nv)->flags & NV_FLAG_IN_SURPRISE_REMOVAL) != 0)

-#define NV_SOC_IS_ISO_IOMMU_PRESENT(nv)     \
-        ((nv)->iso_iommu_present)
-
 /*
- * NVIDIA ACPI event ID to be passed into the core NVIDIA driver for
- * AC/DC event.
+ * For console setup by EFI GOP, the base address is BAR1.
+ * For console setup by VBIOS, the base address is BAR2 + 16MB.
 */
-#define NV_SYSTEM_ACPI_BATTERY_POWER_EVENT   0x8002
+#define NV_IS_CONSOLE_MAPPED(nv, addr)  \
+        (((addr) == (nv)->bars[NV_GPU_BAR_INDEX_FB].cpu_address) || \
+         ((addr) == ((nv)->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address + 0x1000000)))

+#define NV_SOC_IS_ISO_IOMMU_PRESENT(nv)     \
+        ((nv)->iommus.iso_iommu_present)
+
+#define NV_SOC_IS_NISO_IOMMU_PRESENT(nv)     \
+        ((nv)->iommus.niso_iommu_present)
 /*
 * GPU add/remove events
 */
@@ -592,8 +651,6 @@ typedef enum
 * to core NVIDIA driver for ACPI events.
 */
 #define NV_SYSTEM_ACPI_EVENT_VALUE_DISPLAY_SWITCH_DEFAULT    0
-#define NV_SYSTEM_ACPI_EVENT_VALUE_POWER_EVENT_AC            0
-#define NV_SYSTEM_ACPI_EVENT_VALUE_POWER_EVENT_BATTERY       1
 #define NV_SYSTEM_ACPI_EVENT_VALUE_DOCK_EVENT_UNDOCKED       0
 #define NV_SYSTEM_ACPI_EVENT_VALUE_DOCK_EVENT_DOCKED         1

@@ -604,14 +661,18 @@ typedef enum
 #define NV_EVAL_ACPI_METHOD_NVIF     0x01
 #define NV_EVAL_ACPI_METHOD_WMMX     0x02

-#define NV_I2C_CMD_READ              1
-#define NV_I2C_CMD_WRITE             2
-#define NV_I2C_CMD_SMBUS_READ        3
-#define NV_I2C_CMD_SMBUS_WRITE       4
-#define NV_I2C_CMD_SMBUS_QUICK_WRITE 5
-#define NV_I2C_CMD_SMBUS_QUICK_READ  6
-#define NV_I2C_CMD_SMBUS_BLOCK_READ  7
-#define NV_I2C_CMD_SMBUS_BLOCK_WRITE 8
+typedef enum {
+    NV_I2C_CMD_READ = 1,
+    NV_I2C_CMD_WRITE,
+    NV_I2C_CMD_SMBUS_READ,
+    NV_I2C_CMD_SMBUS_WRITE,
+    NV_I2C_CMD_SMBUS_QUICK_WRITE,
+    NV_I2C_CMD_SMBUS_QUICK_READ,
+    NV_I2C_CMD_SMBUS_BLOCK_READ,
+    NV_I2C_CMD_SMBUS_BLOCK_WRITE,
+    NV_I2C_CMD_BLOCK_READ,
+    NV_I2C_CMD_BLOCK_WRITE
+} nv_i2c_cmd_t;

 // Flags needed by OSAllocPagesNode
 #define NV_ALLOC_PAGES_NODE_NONE                0x0
@@ -624,27 +685,38 @@ typedef enum
 #define NV_GET_NV_STATE(pGpu) \
    (nv_state_t *)((pGpu) ? (pGpu)->pOsGpuInfo : NULL)

-#define IS_REG_OFFSET(nv, offset, length)                                       \
-    (((offset) >= (nv)->regs->cpu_address) &&                                   \
-    (((offset) + ((length)-1)) <=                                               \
-        (nv)->regs->cpu_address + ((nv)->regs->size-1)))
+static inline NvBool IS_REG_OFFSET(nv_state_t *nv, NvU64 offset, NvU64 length)
+{
+    return ((offset >= nv->regs->cpu_address) &&
+            ((offset + (length - 1)) >= offset) &&
+            ((offset + (length - 1)) <= (nv->regs->cpu_address + (nv->regs->size - 1))));
+}

-#define IS_FB_OFFSET(nv, offset, length)                                        \
-    (((nv)->fb) && ((offset) >= (nv)->fb->cpu_address) &&                       \
-    (((offset) + ((length)-1)) <= (nv)->fb->cpu_address + ((nv)->fb->size-1)))
+static inline NvBool IS_FB_OFFSET(nv_state_t *nv, NvU64 offset, NvU64 length)
+{
+    return  ((nv->fb) && (nv->fb->size != 0) &&
+             (offset >= nv->fb->cpu_address) &&
+             ((offset + (length - 1)) >= offset) &&
+             ((offset + (length - 1)) <= (nv->fb->cpu_address + (nv->fb->size - 1))));
+}

-#define IS_UD_OFFSET(nv, offset, length)                                        \
-    (((nv)->ud.cpu_address != 0) && ((nv)->ud.size != 0) &&                     \
-    ((offset) >= (nv)->ud.cpu_address) &&                                       \
-    (((offset) + ((length)-1)) <= (nv)->ud.cpu_address + ((nv)->ud.size-1)))
+static inline NvBool IS_UD_OFFSET(nv_state_t *nv, NvU64 offset, NvU64 length)
+{
+    return ((nv->ud.cpu_address != 0) && (nv->ud.size != 0) &&
+            (offset >= nv->ud.cpu_address) &&
+            ((offset + (length - 1)) >= offset) &&
+            ((offset + (length - 1)) <= (nv->ud.cpu_address + (nv->ud.size - 1))));
+}

-#define IS_IMEM_OFFSET(nv, offset, length)                                      \
-    (((nv)->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address != 0) &&                    \
-     ((nv)->bars[NV_GPU_BAR_INDEX_IMEM].size != 0) &&                           \
-     ((offset) >= (nv)->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address) &&             \
-     (((offset) + ((length) - 1)) <=                                            \
-        (nv)->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address +                         \
-            ((nv)->bars[NV_GPU_BAR_INDEX_IMEM].size - 1)))
+static inline NvBool IS_IMEM_OFFSET(nv_state_t *nv, NvU64 offset, NvU64 length)
+{
+    return ((nv->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address != 0) &&
+            (nv->bars[NV_GPU_BAR_INDEX_IMEM].size != 0) &&
+            (offset >= nv->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address) &&
+            ((offset + (length - 1)) >= offset) &&
+            ((offset + (length - 1)) <= (nv->bars[NV_GPU_BAR_INDEX_IMEM].cpu_address +
+                                         (nv->bars[NV_GPU_BAR_INDEX_IMEM].size - 1))));
+}

 #define NV_RM_MAX_MSIX_LINES  8

@@ -712,8 +784,8 @@ nv_state_t*  NV_API_CALL  nv_get_ctl_state       (void);

 void   NV_API_CALL  nv_set_dma_address_size      (nv_state_t *, NvU32 );

-NV_STATUS  NV_API_CALL  nv_alias_pages           (nv_state_t *, NvU32, NvU32, NvU32, NvU64, NvU64 *, void **);
-NV_STATUS  NV_API_CALL  nv_alloc_pages           (nv_state_t *, NvU32, NvBool, NvU32, NvBool, NvBool, NvU64 *, void **);
+NV_STATUS  NV_API_CALL  nv_alias_pages           (nv_state_t *, NvU32, NvU64, NvU32, NvU32, NvU64, NvU64 *, NvBool, void **);
+NV_STATUS  NV_API_CALL  nv_alloc_pages           (nv_state_t *, NvU32, NvU64, NvBool, NvU32, NvBool, NvBool, NvS32, NvU64 *, void **);
 NV_STATUS  NV_API_CALL  nv_free_pages            (nv_state_t *, NvU32, NvBool, NvU32, void *);

 NV_STATUS  NV_API_CALL  nv_register_user_pages   (nv_state_t *, NvU64, NvU64 *, void *, void **);
@@ -730,8 +802,6 @@ NV_STATUS NV_API_CALL   nv_register_phys_pages   (nv_state_t *, NvU64 *, NvU64,
 void      NV_API_CALL   nv_unregister_phys_pages (nv_state_t *, void *);

 NV_STATUS  NV_API_CALL  nv_dma_map_sgt           (nv_dma_device_t *, NvU64, NvU64 *, NvU32, void **);
-NV_STATUS  NV_API_CALL  nv_dma_map_pages         (nv_dma_device_t *, NvU64, NvU64 *, NvBool, NvU32, void **);
-NV_STATUS  NV_API_CALL  nv_dma_unmap_pages       (nv_dma_device_t *, NvU64, NvU64 *, void **);

 NV_STATUS  NV_API_CALL  nv_dma_map_alloc         (nv_dma_device_t *, NvU64, NvU64 *, NvBool, void **);
 NV_STATUS  NV_API_CALL  nv_dma_unmap_alloc       (nv_dma_device_t *, NvU64, NvU64 *, void **);
@@ -758,6 +828,7 @@ void   NV_API_CALL  nv_acpi_methods_init         (NvU32 *);
 void   NV_API_CALL  nv_acpi_methods_uninit       (void);

 NV_STATUS  NV_API_CALL  nv_acpi_method           (NvU32, NvU32, NvU32, void *, NvU16, NvU32 *, void *, NvU16 *);
+NV_STATUS  NV_API_CALL  nv_acpi_d3cold_dsm_for_upstream_port (nv_state_t *, NvU8 *, NvU32, NvU32, NvU32 *);
 NV_STATUS  NV_API_CALL  nv_acpi_dsm_method       (nv_state_t *, NvU8 *, NvU32, NvBool, NvU32, void *, NvU16, NvU32 *, void *, NvU16 *);
 NV_STATUS  NV_API_CALL  nv_acpi_ddc_method       (nv_state_t *, void *, NvU32 *, NvBool);
 NV_STATUS  NV_API_CALL  nv_acpi_dod_method       (nv_state_t *, NvU32 *, NvU32 *);
@@ -775,13 +846,14 @@ NV_STATUS  NV_API_CALL  nv_pci_trigger_recovery  (nv_state_t *);
 NvBool     NV_API_CALL  nv_requires_dma_remap    (nv_state_t *);

 NvBool     NV_API_CALL  nv_is_rm_firmware_active(nv_state_t *);
-const void*NV_API_CALL  nv_get_firmware(nv_state_t *, nv_firmware_t, const void **, NvU32 *);
+const void*NV_API_CALL  nv_get_firmware(nv_state_t *, nv_firmware_type_t, nv_firmware_chip_family_t, const void **, NvU32 *);
 void       NV_API_CALL  nv_put_firmware(const void *);

 nv_file_private_t* NV_API_CALL nv_get_file_private(NvS32, NvBool, void **);
 void               NV_API_CALL nv_put_file_private(void *);

-NV_STATUS NV_API_CALL nv_get_device_memory_config(nv_state_t *, NvU64 *, NvU64 *, NvU32 *, NvU32 *, NvS32 *);
+NV_STATUS NV_API_CALL nv_get_device_memory_config(nv_state_t *, NvU64 *, NvU64 *, NvU64 *, NvU32 *, NvS32 *);
+NV_STATUS NV_API_CALL nv_get_egm_info(nv_state_t *, NvU64 *, NvU64 *, NvS32 *);

 NV_STATUS NV_API_CALL nv_get_ibmnpu_genreg_info(nv_state_t *, NvU64 *, NvU64 *, void**);
 NV_STATUS NV_API_CALL nv_get_ibmnpu_relaxed_ordering_mode(nv_state_t *nv, NvBool *mode);
@@ -816,27 +888,29 @@ NV_STATUS NV_API_CALL nv_acquire_fabric_mgmt_cap (int, int*);
 int       NV_API_CALL nv_cap_drv_init(void);
 void      NV_API_CALL nv_cap_drv_exit(void);
 NvBool    NV_API_CALL nv_is_gpu_accessible(nv_state_t *);
-
-NvU32     NV_API_CALL nv_get_os_type(void);
+NvBool    NV_API_CALL nv_match_gpu_os_info(nv_state_t *, void *);

 void      NV_API_CALL nv_get_updated_emu_seg(NvU32 *start, NvU32 *end);
+void      NV_API_CALL nv_get_screen_info(nv_state_t *, NvU64 *, NvU32 *, NvU32 *, NvU32 *, NvU32 *, NvU64 *);
+
 struct dma_buf;
 typedef struct nv_dma_buf nv_dma_buf_t;
 struct drm_gem_object;

 NV_STATUS NV_API_CALL nv_dma_import_sgt  (nv_dma_device_t *, struct sg_table *, struct drm_gem_object *);
 void NV_API_CALL nv_dma_release_sgt(struct sg_table *, struct drm_gem_object *);
-NV_STATUS NV_API_CALL nv_dma_import_dma_buf      (nv_dma_device_t *, struct dma_buf *, NvU32 *, void **, struct sg_table **, nv_dma_buf_t **);
-NV_STATUS NV_API_CALL nv_dma_import_from_fd      (nv_dma_device_t *, NvS32, NvU32 *, void **, struct sg_table **, nv_dma_buf_t **);
-void      NV_API_CALL nv_dma_release_dma_buf     (void *, nv_dma_buf_t *);
+NV_STATUS NV_API_CALL nv_dma_import_dma_buf      (nv_dma_device_t *, struct dma_buf *, NvU32 *, struct sg_table **, nv_dma_buf_t **);
+NV_STATUS NV_API_CALL nv_dma_import_from_fd      (nv_dma_device_t *, NvS32, NvU32 *, struct sg_table **, nv_dma_buf_t **);
+void      NV_API_CALL nv_dma_release_dma_buf     (nv_dma_buf_t *);

 void      NV_API_CALL nv_schedule_uvm_isr        (nv_state_t *);

+NV_STATUS NV_API_CALL nv_schedule_uvm_drain_p2p  (NvU8 *);
+void      NV_API_CALL nv_schedule_uvm_resume_p2p (NvU8 *);

 NvBool    NV_API_CALL nv_platform_supports_s0ix  (void);
 NvBool    NV_API_CALL nv_s2idle_pm_configured    (void);

-
 NvBool    NV_API_CALL nv_is_chassis_notebook      (void);
 void      NV_API_CALL nv_allow_runtime_suspend    (nv_state_t *nv);
 void      NV_API_CALL nv_disallow_runtime_suspend (nv_state_t *nv);
@@ -846,44 +920,7 @@ typedef void (*nvTegraDceClientIpcCallback)(NvU32, NvU32, NvU32, void *, void *)
 NV_STATUS NV_API_CALL nv_get_num_phys_pages      (void *, NvU32 *);
 NV_STATUS NV_API_CALL nv_get_phys_pages          (void *, void *, NvU32 *);

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+void      NV_API_CALL nv_get_disp_smmu_stream_ids (nv_state_t *, NvU32 *, NvU32 *);

 /*
 * ---------------------------------------------------------------------------
@@ -911,6 +948,7 @@ NV_STATUS  NV_API_CALL  rm_ioctl                 (nvidia_stack_t *, nv_state_t *
 NvBool     NV_API_CALL  rm_isr                   (nvidia_stack_t *, nv_state_t *, NvU32 *);
 void       NV_API_CALL  rm_isr_bh                (nvidia_stack_t *, nv_state_t *);
 void       NV_API_CALL  rm_isr_bh_unlocked       (nvidia_stack_t *, nv_state_t *);
+NvBool     NV_API_CALL  rm_is_msix_allowed       (nvidia_stack_t *, nv_state_t *);
 NV_STATUS  NV_API_CALL  rm_power_management      (nvidia_stack_t *, nv_state_t *, nv_pm_action_t);
 NV_STATUS  NV_API_CALL  rm_stop_user_channels    (nvidia_stack_t *, nv_state_t *);
 NV_STATUS  NV_API_CALL  rm_restart_user_channels (nvidia_stack_t *, nv_state_t *);
@@ -929,6 +967,8 @@ NV_STATUS  NV_API_CALL  rm_write_registry_string (nvidia_stack_t *, nv_state_t *
 void       NV_API_CALL  rm_parse_option_string   (nvidia_stack_t *, const char *);
 char*      NV_API_CALL  rm_remove_spaces         (const char *);
 char*      NV_API_CALL  rm_string_token          (char **, const char);
+void       NV_API_CALL  rm_vgpu_vfio_set_driver_vm(nvidia_stack_t *, NvBool);
+NV_STATUS  NV_API_CALL  rm_get_adapter_status_external(nvidia_stack_t *, nv_state_t *);

 NV_STATUS  NV_API_CALL  rm_run_rc_callback       (nvidia_stack_t *, nv_state_t *);
 void       NV_API_CALL  rm_execute_work_item     (nvidia_stack_t *, void *);
@@ -945,39 +985,41 @@ NvBool     NV_API_CALL  rm_is_supported_pci_device(NvU8   pci_class,

 void       NV_API_CALL  rm_i2c_remove_adapters    (nvidia_stack_t *, nv_state_t *);
 NvBool     NV_API_CALL  rm_i2c_is_smbus_capable   (nvidia_stack_t *, nv_state_t *, void *);
-NV_STATUS  NV_API_CALL  rm_i2c_transfer           (nvidia_stack_t *, nv_state_t *, void *, NvU8, NvU8, NvU8, NvU32, NvU8 *);
+NV_STATUS  NV_API_CALL  rm_i2c_transfer           (nvidia_stack_t *, nv_state_t *, void *, nv_i2c_cmd_t, NvU8, NvU8, NvU32, NvU8 *);

 NV_STATUS  NV_API_CALL  rm_perform_version_check  (nvidia_stack_t *, void *, NvU32);

-NV_STATUS  NV_API_CALL  rm_system_event           (nvidia_stack_t *, NvU32, NvU32);
+void       NV_API_CALL  rm_power_source_change_event        (nvidia_stack_t *, NvU32);
+
+void       NV_API_CALL  rm_request_dnotifier_state          (nvidia_stack_t *, nv_state_t *);

 void       NV_API_CALL  rm_disable_gpu_state_persistence    (nvidia_stack_t *sp, nv_state_t *);
 NV_STATUS  NV_API_CALL  rm_p2p_init_mapping       (nvidia_stack_t *, NvU64, NvU64 *, NvU64 *, NvU64 *, NvU64 *, NvU64, NvU64, NvU64, NvU64, void (*)(void *), void *);
 NV_STATUS  NV_API_CALL  rm_p2p_destroy_mapping    (nvidia_stack_t *, NvU64);
 NV_STATUS  NV_API_CALL  rm_p2p_get_pages          (nvidia_stack_t *, NvU64, NvU32, NvU64, NvU64, NvU64 *, NvU32 *, NvU32 *, NvU32 *, NvU8 **, void *);
 NV_STATUS  NV_API_CALL  rm_p2p_get_gpu_info       (nvidia_stack_t *, NvU64, NvU64, NvU8 **, void **);
-NV_STATUS  NV_API_CALL  rm_p2p_get_pages_persistent (nvidia_stack_t *,  NvU64, NvU64, void **, NvU64 *, NvU32 *, void *, void *);
+NV_STATUS  NV_API_CALL  rm_p2p_get_pages_persistent (nvidia_stack_t *,  NvU64, NvU64, void **, NvU64 *, NvU32 *, void *, void *, void **);
 NV_STATUS  NV_API_CALL  rm_p2p_register_callback  (nvidia_stack_t *, NvU64, NvU64, NvU64, void *, void (*)(void *), void *);
 NV_STATUS  NV_API_CALL  rm_p2p_put_pages          (nvidia_stack_t *, NvU64, NvU32, NvU64, void *);
-NV_STATUS  NV_API_CALL  rm_p2p_put_pages_persistent(nvidia_stack_t *, void *, void *);
-NV_STATUS  NV_API_CALL  rm_p2p_dma_map_pages      (nvidia_stack_t *, nv_dma_device_t *, NvU8 *, NvU32, NvU32, NvU64 *, void **);
-NV_STATUS  NV_API_CALL  rm_dma_buf_dup_mem_handle (nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvHandle, NvHandle, void *, NvHandle, NvU64, NvU64, NvHandle *);
+NV_STATUS  NV_API_CALL  rm_p2p_put_pages_persistent(nvidia_stack_t *, void *, void *, void *);
+NV_STATUS  NV_API_CALL  rm_p2p_dma_map_pages      (nvidia_stack_t *, nv_dma_device_t *, NvU8 *, NvU64, NvU32, NvU64 *, void **);
+NV_STATUS  NV_API_CALL  rm_dma_buf_dup_mem_handle (nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvHandle, NvHandle, void *, NvHandle, NvU64, NvU64, NvHandle *, void **);
 void       NV_API_CALL  rm_dma_buf_undup_mem_handle(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle);
-NV_STATUS  NV_API_CALL  rm_dma_buf_map_mem_handle (nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvU64, NvU64, NvU64 *);
-NV_STATUS  NV_API_CALL  rm_dma_buf_unmap_mem_handle(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvU64, NvU64);
-NV_STATUS  NV_API_CALL  rm_dma_buf_get_client_and_device(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle *, NvHandle *, NvHandle *, void **);
+NV_STATUS  NV_API_CALL  rm_dma_buf_map_mem_handle (nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, MemoryRange, void *, NvBool, MemoryArea *);
+void       NV_API_CALL  rm_dma_buf_unmap_mem_handle(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, void *, NvBool, MemoryArea);
+NV_STATUS  NV_API_CALL  rm_dma_buf_get_client_and_device(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvHandle *, NvHandle *, NvHandle *, void **, NvBool *);
 void       NV_API_CALL  rm_dma_buf_put_client_and_device(nvidia_stack_t *, nv_state_t *, NvHandle, NvHandle, NvHandle, void *);
 NV_STATUS  NV_API_CALL  rm_log_gpu_crash          (nv_stack_t *, nv_state_t *);

 void       NV_API_CALL rm_kernel_rmapi_op(nvidia_stack_t *sp, void *ops_cmd);
 NvBool     NV_API_CALL rm_get_device_remove_flag(nvidia_stack_t *sp, NvU32 gpu_id);
 NV_STATUS  NV_API_CALL rm_gpu_copy_mmu_faults(nvidia_stack_t *, nv_state_t *, NvU32 *);
-NV_STATUS  NV_API_CALL rm_gpu_copy_mmu_faults_unlocked(nvidia_stack_t *, nv_state_t *, NvU32 *);
+NV_STATUS  NV_API_CALL rm_gpu_handle_mmu_faults(nvidia_stack_t *, nv_state_t *, NvU32 *);
 NvBool     NV_API_CALL rm_gpu_need_4k_page_isolation(nv_state_t *);
 NvBool     NV_API_CALL rm_is_chipset_io_coherent(nv_stack_t *);
 NvBool     NV_API_CALL rm_init_event_locks(nvidia_stack_t *, nv_state_t *);
 void       NV_API_CALL rm_destroy_event_locks(nvidia_stack_t *, nv_state_t *);
-NV_STATUS  NV_API_CALL rm_get_gpu_numa_info(nvidia_stack_t *, nv_state_t *, NvS32 *, NvU64 *, NvU64 *, NvU64 *, NvU32 *);
+NV_STATUS  NV_API_CALL rm_get_gpu_numa_info(nvidia_stack_t *, nv_state_t *, nv_ioctl_numa_info_t *);
 NV_STATUS  NV_API_CALL rm_gpu_numa_online(nvidia_stack_t *, nv_state_t *);
 NV_STATUS  NV_API_CALL rm_gpu_numa_offline(nvidia_stack_t *, nv_state_t *);
 NvBool     NV_API_CALL rm_is_device_sequestered(nvidia_stack_t *, nv_state_t *);
@@ -992,30 +1034,34 @@ void       NV_API_CALL rm_cleanup_dynamic_power_management(nvidia_stack_t *, nv_
 void       NV_API_CALL rm_enable_dynamic_power_management(nvidia_stack_t *, nv_state_t *);
 NV_STATUS  NV_API_CALL rm_ref_dynamic_power(nvidia_stack_t *, nv_state_t *, nv_dynamic_power_mode_t);
 void       NV_API_CALL rm_unref_dynamic_power(nvidia_stack_t *, nv_state_t *, nv_dynamic_power_mode_t);
-NV_STATUS  NV_API_CALL rm_transition_dynamic_power(nvidia_stack_t *, nv_state_t *, NvBool);
-const char* NV_API_CALL rm_get_vidmem_power_status(nvidia_stack_t *, nv_state_t *);
-const char* NV_API_CALL rm_get_dynamic_power_management_status(nvidia_stack_t *, nv_state_t *);
-const char* NV_API_CALL rm_get_gpu_gcx_support(nvidia_stack_t *, nv_state_t *, NvBool);
+NV_STATUS  NV_API_CALL rm_transition_dynamic_power(nvidia_stack_t *, nv_state_t *, NvBool, NvBool *);
+void       NV_API_CALL rm_get_power_info(nvidia_stack_t *, nv_state_t *, nv_power_info_t *);

 void       NV_API_CALL rm_acpi_notify(nvidia_stack_t *, nv_state_t *, NvU32);
-NV_STATUS  NV_API_CALL rm_get_clientnvpcf_power_limits(nvidia_stack_t *, nv_state_t *, NvU32 *, NvU32 *);
+void       NV_API_CALL rm_acpi_nvpcf_notify(nvidia_stack_t *);
+
+NvBool     NV_API_CALL rm_is_altstack_in_use(void);

 /* vGPU VFIO specific functions */
 NV_STATUS  NV_API_CALL  nv_vgpu_create_request(nvidia_stack_t *, nv_state_t *, const NvU8 *, NvU32, NvU16 *, NvU32);
 NV_STATUS  NV_API_CALL  nv_vgpu_delete(nvidia_stack_t *, const NvU8 *, NvU16);
-NV_STATUS  NV_API_CALL  nv_vgpu_get_type_ids(nvidia_stack_t *, nv_state_t *, NvU32 *, NvU32 **, NvBool);
+NV_STATUS  NV_API_CALL  nv_vgpu_get_type_ids(nvidia_stack_t *, nv_state_t *, NvU32 *, NvU32 *, NvBool, NvU8, NvBool);
 NV_STATUS  NV_API_CALL  nv_vgpu_get_type_info(nvidia_stack_t *, nv_state_t *, NvU32, char *, int, NvU8);
-NV_STATUS  NV_API_CALL  nv_vgpu_get_bar_info(nvidia_stack_t *, nv_state_t *, const NvU8 *, NvU64 *, NvU32, void *);
-NV_STATUS  NV_API_CALL  nv_vgpu_start(nvidia_stack_t *, const NvU8 *, void *, NvS32 *, NvU8 *, NvU32);
-NV_STATUS  NV_API_CALL  nv_vgpu_get_sparse_mmap(nvidia_stack_t *, nv_state_t *, const NvU8 *, NvU64 **, NvU64 **, NvU32 *);
+NV_STATUS  NV_API_CALL  nv_vgpu_get_bar_info(nvidia_stack_t *, nv_state_t *, const NvU8 *, NvU64 *,
+                                             NvU64 *, NvU64 *, NvU32 *, NvBool *, NvU8 *);
+NV_STATUS  NV_API_CALL  nv_vgpu_get_hbm_info(nvidia_stack_t *, nv_state_t *, const NvU8 *, NvU64 *, NvU64 *);
 NV_STATUS  NV_API_CALL  nv_vgpu_process_vf_info(nvidia_stack_t *, nv_state_t *, NvU8, NvU32, NvU8, NvU8, NvU8, NvBool, void *);
-NV_STATUS  NV_API_CALL  nv_vgpu_update_request(nvidia_stack_t *, const NvU8 *, NvU32, NvU64 *, NvU64 *, const char *);
-NV_STATUS  NV_API_CALL  nv_gpu_bind_event(nvidia_stack_t *);
+NV_STATUS  NV_API_CALL  nv_gpu_bind_event(nvidia_stack_t *, NvU32, NvBool *);
+NV_STATUS  NV_API_CALL  nv_gpu_unbind_event(nvidia_stack_t *, NvU32, NvBool *);

 NV_STATUS NV_API_CALL nv_get_usermap_access_params(nv_state_t*, nv_usermap_access_params_t*);
 nv_soc_irq_type_t NV_API_CALL nv_get_current_irq_type(nv_state_t*);
 void       NV_API_CALL  nv_flush_coherent_cpu_cache_range(nv_state_t *nv, NvU64 cpu_virtual, NvU64 size);

+#if defined(NV_VMWARE)
+const void* NV_API_CALL rm_get_firmware(nv_firmware_type_t fw_type, const void **fw_buf, NvU32 *fw_size);
+#endif
+
 /* Callbacks should occur roughly every 10ms. */
 #define NV_SNAPSHOT_TIMER_HZ 100
 void NV_API_CALL nv_start_snapshot_timer(void (*snapshot_callback)(void *context));
@@ -1027,17 +1073,18 @@ static inline const NvU8 *nv_get_cached_uuid(nv_state_t *nv)
    return nv->nv_uuid_cache.valid ? nv->nv_uuid_cache.uuid : NULL;
 }

+/* nano second resolution timer callback structure */
+typedef struct nv_nano_timer nv_nano_timer_t;

+/* nano timer functions */
+void        NV_API_CALL nv_create_nano_timer(nv_state_t *, void *pTmrEvent, nv_nano_timer_t **);
+void        NV_API_CALL nv_start_nano_timer(nv_state_t *nv, nv_nano_timer_t *, NvU64 timens);
+NV_STATUS   NV_API_CALL rm_run_nano_timer_callback(nvidia_stack_t *, nv_state_t *, void *pTmrEvent);
+void        NV_API_CALL nv_cancel_nano_timer(nv_state_t *, nv_nano_timer_t *);
+void        NV_API_CALL nv_destroy_nano_timer(nv_state_t *nv, nv_nano_timer_t *);

-
-
-
-
-
-
-
-
-
+// Host1x specific functions.
+NV_STATUS NV_API_CALL nv_get_syncpoint_aperture(NvU32, NvU64 *, NvU64 *, NvU32 *);

 #if defined(NVCPU_X86_64)

--- a/kernel-open/common/inc/nv_speculation_barrier.h
+++ b/kernel-open/common/inc/nv_speculation_barrier.h
@@ -86,7 +86,7 @@
 /* Not currently implemented for MSVC/ARM64. See bug 3366890. */
 #   define nv_speculation_barrier()
 #   define speculation_barrier() nv_speculation_barrier()
-#elif defined(NVCPU_NVRISCV64) && NVOS_IS_LIBOS
+#elif defined(NVCPU_IS_RISCV64)
 #   define nv_speculation_barrier()
 #else
 #error "Unknown compiler/chip family"
--- a/kernel-open/common/inc/nv_uvm_interface.h
+++ b/kernel-open/common/inc/nv_uvm_interface.h
--- a/kernel-open/common/inc/nv_uvm_types.h
+++ b/kernel-open/common/inc/nv_uvm_types.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2014-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2014-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -39,12 +39,13 @@
 // are multiple BIG page sizes in RM. These defines are used as flags to "0"
 // should be OK when user is not sure which pagesize allocation it wants
 //
-#define UVM_PAGE_SIZE_DEFAULT    0x0
-#define UVM_PAGE_SIZE_4K         0x1000
-#define UVM_PAGE_SIZE_64K        0x10000
-#define UVM_PAGE_SIZE_128K       0x20000
-#define UVM_PAGE_SIZE_2M         0x200000
-#define UVM_PAGE_SIZE_512M       0x20000000
+#define UVM_PAGE_SIZE_DEFAULT    0x0ULL
+#define UVM_PAGE_SIZE_4K         0x1000ULL
+#define UVM_PAGE_SIZE_64K        0x10000ULL
+#define UVM_PAGE_SIZE_128K       0x20000ULL
+#define UVM_PAGE_SIZE_2M         0x200000ULL
+#define UVM_PAGE_SIZE_512M       0x20000000ULL
+#define UVM_PAGE_SIZE_256G       0x4000000000ULL

 //
 // When modifying flags, make sure they are compatible with the mirrored
@@ -92,6 +93,7 @@ typedef unsigned long long UvmGpuPointer;
 typedef struct uvmGpuSession_tag       *uvmGpuSessionHandle;       // gpuSessionHandle
 typedef struct uvmGpuDevice_tag        *uvmGpuDeviceHandle;        // gpuDeviceHandle
 typedef struct uvmGpuAddressSpace_tag  *uvmGpuAddressSpaceHandle;  // gpuAddressSpaceHandle
+typedef struct uvmGpuTsg_tag           *uvmGpuTsgHandle;           // gpuTsgHandle
 typedef struct uvmGpuChannel_tag       *uvmGpuChannelHandle;       // gpuChannelHandle
 typedef struct uvmGpuCopyEngine_tag    *uvmGpuCopyEngineHandle;    // gpuObjectHandle

@@ -103,6 +105,10 @@ typedef struct UvmGpuMemoryInfo_tag
    // Out: Set to TRUE, if the allocation is in sysmem.
    NvBool sysmem;

+    // Out: Set to TRUE, if this allocation is treated as EGM.
+    //      sysmem is also TRUE when egm is TRUE.
+    NvBool egm;
+
    // Out: Set to TRUE, if the allocation is a constructed
    //      under a Device or Subdevice.
    //      All permutations of sysmem and deviceDescendant are valid.
@@ -110,7 +116,7 @@ typedef struct UvmGpuMemoryInfo_tag
    NvBool deviceDescendant;

    // Out: Page size associated with the phys alloc.
-    NvU32 pageSize;
+    NvU64 pageSize;

    // Out: Set to TRUE, if the allocation is contiguous.
    NvBool contig;
@@ -124,6 +130,10 @@ typedef struct UvmGpuMemoryInfo_tag

    // Out: Uuid of the GPU to which the allocation belongs.
    //      This is only valid if deviceDescendant is NV_TRUE.
+    //      When egm is NV_TRUE, this is also the UUID of the GPU
+    //      for which EGM is local.
+    //      If the GPU has SMC enabled, the UUID is the GI UUID.
+    //      Otherwise, it is the UUID for the physical GPU.
    //      Note: If the allocation is owned by a device in
    //      an SLI group and the allocation is broadcast
    //      across the SLI group, this UUID will be any one
@@ -217,12 +227,6 @@ typedef struct UvmGpuChannelInstanceInfo_tag
    // Out: Type of the engine the channel is bound to
    NvU32 channelEngineType;

-    // Out: Channel handle required to ring the doorbell
-    NvU32 workSubmissionToken;
-
-    // Out: Address of the doorbell
-    volatile NvU32 *workSubmissionOffset;
-
    // Out: Channel handle to be used in the CLEAR_FAULTED method
    NvU32 clearFaultedToken;

@@ -231,6 +235,10 @@ typedef struct UvmGpuChannelInstanceInfo_tag
    // Ampere+ GPUs
    volatile NvU32 *pChramChannelRegister;

+    // Out: Address of the Runlist PRI Base Register required to ring the
+    // doorbell after clearing the faulted bit.
+    volatile NvU32 *pRunlistPRIBaseRegister;
+
    // Out: SMC engine id to which the GR channel is bound, or zero if the GPU
    // does not support SMC or it is a CE channel
    NvU32 smcEngineId;
@@ -260,6 +268,7 @@ typedef struct UvmGpuChannelInfo_tag

    // The errorNotifier is filled out when the channel hits an RC error.
    NvNotification    *errorNotifier;
+    NvNotification    *keyRotationNotifier;

    NvU32              hwRunlistId;
    NvU32              hwChannelId;
@@ -282,6 +291,16 @@ typedef struct UvmGpuChannelInfo_tag
    // to kick off the new work.
    //
    volatile NvU32    *pWorkSubmissionToken;
+
+    // GPU VAs of both GPFIFO and GPPUT are needed in Confidential Computing
+    // so a channel can be controlled via another channel (SEC2 or WLC/LCIC)
+    NvU64              gpFifoGpuVa;
+    NvU64              gpPutGpuVa;
+    NvU64              gpGetGpuVa;
+    // GPU VA of work submission offset is needed in Confidential Computing
+    // so CE channels can ring doorbell of other channels as required for
+    // WLC/LCIC work submission
+    NvU64              workSubmissionOffsetGpuVa;
 } UvmGpuChannelInfo;

 typedef enum
@@ -294,6 +313,17 @@ typedef enum
    UVM_BUFFER_LOCATION_VID  = 2,
 } UVM_BUFFER_LOCATION;

+typedef struct UvmGpuTsgAllocParams_tag
+{
+    // Interpreted as UVM_GPU_CHANNEL_ENGINE_TYPE
+    NvU32 engineType;
+
+    // Index of the engine the TSG is bound to.
+    // Ignored if engineType is anything other than
+    // UVM_GPU_CHANNEL_ENGINE_TYPE_CE.
+    NvU32 engineIndex;
+} UvmGpuTsgAllocParams;
+
 typedef struct UvmGpuChannelAllocParams_tag
 {
    NvU32 numGpFifoEntries;
@@ -301,13 +331,6 @@ typedef struct UvmGpuChannelAllocParams_tag
    // The next two fields store UVM_BUFFER_LOCATION values
    NvU32 gpFifoLoc;
    NvU32 gpPutLoc;
-
-    // Index of the engine the channel will be bound to
-    // ignored if engineType is anything other than UVM_GPU_CHANNEL_ENGINE_TYPE_CE
-    NvU32 engineIndex;
-
-    // interpreted as UVM_GPU_CHANNEL_ENGINE_TYPE
-    NvU32 engineType;
 } UvmGpuChannelAllocParams;

 typedef struct UvmGpuPagingChannelAllocParams_tag
@@ -319,7 +342,7 @@ typedef struct UvmGpuPagingChannelAllocParams_tag

 // The max number of Copy Engines supported by a GPU.
 // The gpu ops build has a static assert that this is the correct number.
-#define UVM_COPY_ENGINE_COUNT_MAX 10
+#define UVM_COPY_ENGINE_COUNT_MAX 64

 typedef struct
 {
@@ -372,48 +395,23 @@ typedef enum
    UVM_LINK_TYPE_NVLINK_1,
    UVM_LINK_TYPE_NVLINK_2,
    UVM_LINK_TYPE_NVLINK_3,
-
-
-
-
+    UVM_LINK_TYPE_NVLINK_4,
+    UVM_LINK_TYPE_NVLINK_5,
+    UVM_LINK_TYPE_C2C,
 } UVM_LINK_TYPE;

 typedef struct UvmGpuCaps_tag
 {
-    NvU32    sysmemLink;            // UVM_LINK_TYPE
-    NvU32    sysmemLinkRateMBps;    // See UvmGpuP2PCapsParams::totalLinkLineRateMBps
+    // If numaEnabled is NV_TRUE, then the system address of allocated GPU
+    // memory can be converted to struct pages. See
+    // UvmGpuInfo::systemMemoryWindowStart.
    NvBool   numaEnabled;
    NvU32    numaNodeId;
-
-    // On ATS systems, GPUs connected to different CPU sockets can have peer
-    // traffic. They are called indirect peers. However, indirect peers are
-    // mapped using sysmem aperture. In order to disambiguate the location of a
-    // specific memory address, each GPU maps its memory to a different window
-    // in the System Physical Address (SPA) space. The following fields contain
-    // the base + size of such window for the GPU. systemMemoryWindowSize
-    // different than 0 indicates that the window is valid.
-    //
-    // - If the window is valid, then we can map GPU memory to the CPU as
-    // cache-coherent by adding the GPU address to the window start.
-    // - If numaEnabled is NV_TRUE, then we can also convert the system
-    // addresses of allocated GPU memory to struct pages.
-    //
-    // TODO: Bug 1986868: fix window start computation for SIMICS
-    NvU64    systemMemoryWindowStart;
-    NvU64    systemMemoryWindowSize;
-
-    // This tells if the GPU is connected to NVSwitch. On systems with NVSwitch
-    // all GPUs are connected to it. If connectedToSwitch is NV_TRUE,
-    // nvswitchMemoryWindowStart tells the base address for the GPU in the
-    // NVSwitch address space. It is used when creating PTEs of memory mappings
-    // to NVSwitch peers.
-    NvBool   connectedToSwitch;
-    NvU64    nvswitchMemoryWindowStart;
 } UvmGpuCaps;

 typedef struct UvmGpuAddressSpaceInfo_tag
 {
-    NvU32           bigPageSize;
+    NvU64           bigPageSize;

    NvBool          atsEnabled;

@@ -433,19 +431,15 @@ typedef struct UvmGpuAddressSpaceInfo_tag

 typedef struct UvmGpuAllocInfo_tag
 {
-    NvU64   rangeBegin;             // Allocation will be made between
-    NvU64   rangeEnd;               // rangeBegin & rangeEnd both included
    NvU64   gpuPhysOffset;          // Returns gpuPhysOffset if contiguous requested
-    NvU32   pageSize;               // default is RM big page size - 64K or 128 K" else use 4K or 2M
-    NvU64   alignment;              // Alignment of allocation
+    NvU64   pageSize;               // default is RM big page size - 64K or 128 K" else use 4K or 2M
+    NvU64   alignment;              // Virtual alignment
    NvBool  bContiguousPhysAlloc;   // Flag to request contiguous physical allocation
    NvBool  bMemGrowsDown;          // Causes RM to reserve physical heap from top of FB
    NvBool  bPersistentVidmem;      // Causes RM to allocate persistent video memory
    NvHandle hPhysHandle;           // Handle for phys allocation either provided or retrieved
-
-
-
-
+    NvBool   bUnprotected;            // Allocation to be made in unprotected memory whenever
+                                      // SEV or GPU CC modes are enabled. Ignored otherwise
 } UvmGpuAllocInfo;

 typedef enum
@@ -526,6 +520,13 @@ typedef struct UvmGpuExternalMappingInfo_tag
    // In: Size of the buffer to store PTEs (in bytes).
    NvU64 pteBufferSize;

+    // In: Page size for mapping
+    //     If this field is passed as 0, the page size
+    //     of the allocation is used for mapping.
+    //     nvUvmInterfaceGetExternalAllocPtes must pass
+    //     this field as zero.
+    NvU64 mappingPageSize;
+
    // In: Pointer to a buffer to store PTEs.
    // Out: The interface will fill the buffer with PTEs
    NvU64 *pteBuffer;
@@ -542,12 +543,46 @@ typedef struct UvmGpuExternalMappingInfo_tag
    NvU32 pteSize;
 } UvmGpuExternalMappingInfo;

+typedef struct UvmGpuExternalPhysAddrInfo_tag
+{
+    // In: Virtual permissions. Returns
+    // NV_ERR_INVALID_ACCESS_TYPE if input is
+    // inaccurate
+    UvmRmGpuMappingType mappingType;
+
+    // In: Size of the buffer to store PhysAddrs (in bytes).
+    NvU64 physAddrBufferSize;
+
+    // In: Page size for mapping
+    //     If this field is passed as 0, the page size
+    //     of the allocation is used for mapping.
+    //     nvUvmInterfaceGetExternalAllocPtes must pass
+    //     this field as zero.
+    NvU64 mappingPageSize;
+
+    // In: Pointer to a buffer to store PhysAddrs.
+    // Out: The interface will fill the buffer with PhysAddrs
+    NvU64 *physAddrBuffer;
+
+    // Out: Number of PhysAddrs filled in to the buffer.
+    NvU64 numWrittenPhysAddrs;
+
+    // Out: Number of PhysAddrs remaining to be filled
+    //      if the buffer is not sufficient to accommodate
+    //      requested PhysAddrs.
+    NvU64 numRemainingPhysAddrs;
+} UvmGpuExternalPhysAddrInfo;
+
 typedef struct UvmGpuP2PCapsParams_tag
 {
    // Out: peerId[i] contains gpu[i]'s peer id of gpu[1 - i]. Only defined if
    // the GPUs are direct peers.
    NvU32 peerIds[2];

+    // Out: peerId[i] contains gpu[i]'s EGM peer id of gpu[1 - i]. Only defined
+    // if the GPUs are direct peers and EGM enabled in the system.
+    NvU32 egmPeerIds[2];
+
    // Out: UVM_LINK_TYPE
    NvU32 p2pLink;

@@ -563,11 +598,6 @@ typedef struct UvmGpuP2PCapsParams_tag
    // second, not taking into account the protocols overhead. The reported
    // bandwidth for indirect peers is zero.
    NvU32 totalLinkLineRateMBps;
-
-    // Out: True if the peers have a indirect link to communicate. On P9
-    // systems, this is true if peers are connected to different NPUs that
-    // forward the requests between them.
-    NvU32 indirectAccess      : 1;
 } UvmGpuP2PCapsParams;

 // Platform-wide information
@@ -576,10 +606,11 @@ typedef struct UvmPlatformInfo_tag
    // Out: ATS (Address Translation Services) is supported
    NvBool atsSupported;

-
-
-
-
+    // Out: True if HW trusted execution, such as AMD's SEV-SNP or Intel's TDX,
+    // is enabled in the VM, indicating that Confidential Computing must be
+    // also enabled in the GPU(s); these two security features are either both
+    // enabled, or both disabled.
+    NvBool confComputingEnabled;
 } UvmPlatformInfo;

 typedef struct UvmGpuClientInfo_tag
@@ -589,23 +620,21 @@ typedef struct UvmGpuClientInfo_tag
    NvHandle hSmcPartRef;
 } UvmGpuClientInfo;

+typedef enum
+{
+    UVM_GPU_CONF_COMPUTE_MODE_NONE,
+    UVM_GPU_CONF_COMPUTE_MODE_APM,
+    UVM_GPU_CONF_COMPUTE_MODE_HCC,
+    UVM_GPU_CONF_COMPUTE_MODE_COUNT
+} UvmGpuConfComputeMode;

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+typedef struct UvmGpuConfComputeCaps_tag
+{
+    // Out: GPU's confidential compute mode
+    UvmGpuConfComputeMode mode;
+    // Is key rotation enabled for UVM keys
+    NvBool bKeyRotationEnabled;
+} UvmGpuConfComputeCaps;

 #define UVM_GPU_NAME_LENGTH 0x40

@@ -614,7 +643,8 @@ typedef struct UvmGpuInfo_tag
    // Printable gpu name
    char name[UVM_GPU_NAME_LENGTH];

-    // Uuid of this gpu
+    // Uuid of the physical GPU or GI UUID if nvUvmInterfaceGetGpuInfo()
+    // requested information for a valid SMC partition.
    NvProcessorUuid uuid;

    // Gpu architecture; NV2080_CTRL_MC_ARCH_INFO_ARCHITECTURE_*
@@ -660,6 +690,9 @@ typedef struct UvmGpuInfo_tag
    // Maximum number of TPCs per GPC
    NvU32 maxTpcPerGpcCount;

+    // Number of access counter buffers.
+    NvU32 accessCntrBufferCount;
+
    // NV_TRUE if SMC is enabled on this GPU.
    NvBool smcEnabled;

@@ -671,9 +704,47 @@ typedef struct UvmGpuInfo_tag

    UvmGpuClientInfo smcUserClientInfo;

+    // Confidential Compute capabilities of this GPU
+    UvmGpuConfComputeCaps gpuConfComputeCaps;

+    // UVM_LINK_TYPE
+    NvU32 sysmemLink;

+    // See UvmGpuP2PCapsParams::totalLinkLineRateMBps
+    NvU32 sysmemLinkRateMBps;

+    // On coherent systems each GPU maps its memory to a window in the System
+    // Physical Address (SPA) space. The following fields describe that window.
+    //
+    // systemMemoryWindowSize > 0 indicates that the window is valid. meaning
+    // that GPU memory can be mapped by the CPU as cache-coherent by adding the
+    // GPU address to the window start.
+    NvU64 systemMemoryWindowStart;
+    NvU64 systemMemoryWindowSize;
+
+    // This tells if the GPU is connected to NVSwitch. On systems with NVSwitch
+    // all GPUs are connected to it. If connectedToSwitch is NV_TRUE,
+    // nvswitchMemoryWindowStart tells the base address for the GPU in the
+    // NVSwitch address space. It is used when creating PTEs of memory mappings
+    // to NVSwitch peers.
+    NvBool connectedToSwitch;
+    NvU64 nvswitchMemoryWindowStart;
+
+    // local EGM properties
+    // NV_TRUE if EGM is enabled
+    NvBool   egmEnabled;
+
+    // Peer ID to reach local EGM when EGM is enabled
+    NvU8     egmPeerId;
+
+    // EGM base address to offset in the GMMU PTE entry for EGM mappings
+    NvU64    egmBaseAddr;
+
+    // If connectedToSwitch is NV_TRUE,
+    // nvswitchEgmMemoryWindowStart tells the base address for the GPU's EGM memory in the
+    // NVSwitch address space. It is used when creating PTEs of GPU memory mappings
+    // to NVSwitch peers.
+    NvU64 nvswitchEgmMemoryWindowStart;

 } UvmGpuInfo;

@@ -683,9 +754,12 @@ typedef struct UvmGpuFbInfo_tag
    // RM regions that are not registered with PMA either.
    NvU64 maxAllocatableAddress;

-    NvU32 heapSize;         // RAM in KB available for user allocations
-    NvU32 reservedHeapSize; // RAM in KB reserved for internal RM allocation
-    NvBool bZeroFb;         // Zero FB mode enabled.
+    NvU32  heapSize;           // RAM in KB available for user allocations
+    NvU32  reservedHeapSize;   // RAM in KB reserved for internal RM allocation
+    NvBool bZeroFb;            // Zero FB mode enabled.
+    NvU64  maxVidmemPageSize;  // Largest GPU page size to access vidmem.
+    NvBool bStaticBar1Enabled; // Static BAR1 mode is enabled
+    NvU64  staticBar1Size;     // The size of the static mapping
 } UvmGpuFbInfo;

 typedef struct UvmGpuEccInfo_tag
@@ -697,6 +771,15 @@ typedef struct UvmGpuEccInfo_tag
    NvBool   bEccEnabled;
 } UvmGpuEccInfo;

+typedef struct UvmGpuNvlinkInfo_tag
+{
+    unsigned nvlinkMask;
+    unsigned nvlinkOffset;
+    void    *nvlinkReadLocation;
+    NvBool  *nvlinkErrorNotifier;
+    NvBool   bNvlinkRecoveryEnabled;
+} UvmGpuNvlinkInfo;
+
 typedef struct UvmPmaAllocationOptions_tag
 {
    NvU32 flags;
@@ -717,11 +800,9 @@ typedef struct UvmPmaStatistics_tag
    volatile NvU64 numPages2m;                // PMA-wide 2MB pages count across all regions
    volatile NvU64 numFreePages64k;           // PMA-wide free 64KB page count across all regions
    volatile NvU64 numFreePages2m;            // PMA-wide free 2MB pages count across all regions
-
-
-
-
-
+    volatile NvU64 numPages2mProtected;       // PMA-wide 2MB pages count in protected memory
+    volatile NvU64 numFreePages64kProtected;  // PMA-wide free 64KB page count in protected memory
+    volatile NvU64 numFreePages2mProtected;   // PMA-wide free 2MB pages count in protected memory
 } UvmPmaStatistics;

 /*******************************************************************************
@@ -765,14 +846,14 @@ typedef NV_STATUS (*uvmEventResume_t) (void);
 /*******************************************************************************
    uvmEventStartDevice
    This function will be called by the GPU driver once it has finished its
-    initialization to tell the UVM driver that this GPU has come up.
+    initialization to tell the UVM driver that this physical GPU has come up.
 */
 typedef NV_STATUS (*uvmEventStartDevice_t) (const NvProcessorUuid *pGpuUuidStruct);

 /*******************************************************************************
    uvmEventStopDevice
-    This function will be called by the GPU driver to let UVM know that a GPU
-    is going down.
+    This function will be called by the GPU driver to let UVM know that a
+    physical GPU is going down.
 */
 typedef NV_STATUS (*uvmEventStopDevice_t) (const NvProcessorUuid *pGpuUuidStruct);

@@ -803,7 +884,7 @@ typedef NV_STATUS (*uvmEventServiceInterrupt_t) (void *pDeviceObject,
 /*******************************************************************************
    uvmEventIsrTopHalf_t
    This function will be called by the GPU driver to let UVM know
-    that an interrupt has occurred.
+    that an interrupt has occurred on the given physical GPU.

    Returns:
        NV_OK if the UVM driver handled the interrupt
@@ -815,6 +896,41 @@ typedef NV_STATUS (*uvmEventIsrTopHalf_t) (const NvProcessorUuid *pGpuUuidStruct
 typedef void (*uvmEventIsrTopHalf_t) (void);
 #endif

+/*******************************************************************************
+    uvmEventDrainP2P
+    This function will be called by the GPU driver to signal to UVM that the
+    GPU has encountered an uncontained error, and all peer work must be drained
+    to recover.  When it is called, the following assumptions/guarantees are
+    valid/made:
+
+      * Impacted user channels have been preempted and disabled
+      * UVM channels are still running normally and will continue to do
+        so unless an unrecoverable error is hit on said channels
+      * UVM must not return from this function until all enqueued work on
+      * peer channels has drained
+      * In the context of this function call, RM will still service faults
+      * UVM must prevent new peer work from being enqueued until the
+        uvmEventResumeP2P callback is issued
+
+    Returns:
+        NV_OK if UVM has idled peer work and will prevent new peer workloads.
+        NV_ERR_TIMEOUT if peer work was unable to be drained within a timeout
+        XXX NV_ERR_* for any other failure (TBD)
+
+*/
+typedef NV_STATUS (*uvmEventDrainP2P_t) (const NvProcessorUuid *pGpuUuidStruct);
+
+/*******************************************************************************
+    uvmEventResumeP2P
+    This function will be called by the GPU driver to signal to UVM that the
+    GPU has recovered from the previously reported uncontained NVLINK error.
+    When it is called, the following assumptions/guarantees are valid/made:
+
+      * UVM is again allowed to enqueue peer work
+      * UVM channels are still running normally
+*/
+typedef NV_STATUS (*uvmEventResumeP2P_t) (const NvProcessorUuid *pGpuUuidStruct);
+
 struct UvmOpsUvmEvents
 {
    uvmEventSuspend_t     suspend;
@@ -827,26 +943,91 @@ struct UvmOpsUvmEvents
    uvmEventWddmRestartAfterTimeout_t wddmRestartAfterTimeout;
    uvmEventServiceInterrupt_t serviceInterrupt;
 #endif
+    uvmEventDrainP2P_t drainP2P;
+    uvmEventResumeP2P_t resumeP2P;
 };

+#define UVM_CSL_SIGN_AUTH_TAG_SIZE_BYTES 32
+#define UVM_CSL_CRYPT_AUTH_TAG_SIZE_BYTES 16
+
+typedef union UvmFaultMetadataPacket_tag
+{
+    struct {
+        NvU8   authTag[UVM_CSL_CRYPT_AUTH_TAG_SIZE_BYTES];
+        NvBool valid;
+    };
+    // padding to 32Bytes
+    NvU8 _padding[32];
+} UvmFaultMetadataPacket;
+
+// This struct shall not be accessed nor modified directly by UVM as it is
+// entirely managed by the RM layer
+typedef struct UvmCslContext_tag
+{
+    struct ccslContext_t *ctx;
+    void *nvidia_stack;
+} UvmCslContext;
+
 typedef struct UvmGpuFaultInfo_tag
 {
    struct
    {
-        // Register mappings obtained from RM
+        // Fault buffer GET register mapping.
+        //
+        // When Confidential Computing is enabled, GET refers to the shadow
+        // buffer (see bufferAddress below), and not to the actual HW buffer.
+        // In this setup, writes of GET (by UVM) do not result on re-evaluation
+        // of any interrupt condition.
        volatile NvU32* pFaultBufferGet;
+
+        // Fault buffer PUT register mapping.
+        //
+        // When Confidential Computing is enabled, PUT refers to the shadow
+        // buffer (see bufferAddress below), and not to the actual HW buffer.
+        // In this setup, writes of PUT (by GSP-RM) do not result on
+        // re-evaluation of any interrupt condition.
        volatile NvU32* pFaultBufferPut;
-        // Note: this variable is deprecated since buffer overflow is not a separate
-        // register from future chips.
+
+        // Note: this variable is deprecated since buffer overflow is not a
+        // separate register from future chips.
        volatile NvU32* pFaultBufferInfo;
+
+        // Register mapping used to clear a replayable fault interrupt in
+        // Turing+ GPUs.
        volatile NvU32* pPmcIntr;
+
+        // Register mapping used to enable replayable fault interrupts.
        volatile NvU32* pPmcIntrEnSet;
+
+        // Register mapping used to disable replayable fault interrupts.
        volatile NvU32* pPmcIntrEnClear;
+
+        // Register used to enable, or disable, faults on prefetches.
        volatile NvU32* pPrefetchCtrl;
+
+        // Replayable fault interrupt mask identifier.
        NvU32 replayableFaultMask;
-        // fault buffer cpu mapping and size
-        void* bufferAddress;
+
+        // Fault buffer CPU mapping
+        void*  bufferAddress;
+        //
+        // When Confidential Computing is disabled, the mapping points to the
+        // actual HW fault buffer.
+        //
+        // When Confidential Computing is enabled, the mapping points to a
+        // copy of the HW fault buffer. This "shadow buffer" is maintained
+        // by GSP-RM.
+
+        // Size, in bytes, of the fault buffer pointed by bufferAddress.
        NvU32  bufferSize;
+        // Mapping pointing to the start of the fault buffer metadata containing
+        // a 16Byte authentication tag and a valid byte. Always NULL when
+        // Confidential Computing is disabled.
+        UvmFaultMetadataPacket *bufferMetadata;
+
+        // CSL context used for performing decryption of replayable faults when
+        // Confidential Computing is enabled.
+        UvmCslContext cslCtx;
    } replayable;
    struct
    {
@@ -865,10 +1046,24 @@ typedef struct UvmGpuFaultInfo_tag

        // Preallocated stack for functions called from the UVM isr bottom half
        void *isr_bh_sp;
+
+        // Used only when Hopper Confidential Compute is enabled
+        // Register mappings obtained from RM
+        volatile NvU32* pFaultBufferPut;
+
+        // Used only when Hopper Confidential Compute is enabled
+        // Cached get index of the non-replayable shadow buffer
+        NvU32 shadowBufferGet;
+
+        // See replayable.bufferMetadata
+        UvmFaultMetadataPacket  *shadowBufferMetadata;
    } nonReplayable;
    NvHandle faultBufferHandle;
+    struct Device *pDevice;
 } UvmGpuFaultInfo;

+struct Device;
+
 typedef struct UvmGpuPagingChannel_tag
 {
    struct gpuDevice  *device;
@@ -876,6 +1071,7 @@ typedef struct UvmGpuPagingChannel_tag
    NvHandle          channelHandle;
    NvHandle          errorNotifierHandle;
    void              *pushStreamSp;
+    struct Device     *pDevice;
 } UvmGpuPagingChannel, *UvmGpuPagingChannelHandle;

 typedef struct UvmGpuAccessCntrInfo_tag
@@ -899,12 +1095,6 @@ typedef struct UvmGpuAccessCntrInfo_tag
    void* bufferAddress;
    NvU32  bufferSize;
    NvHandle accessCntrBufferHandle;
-
-    // The Notification address in the access counter notification msg does not
-    // contain the correct upper bits 63-47 for GPA-based notifications. RM
-    // provides us with the correct offset to be added.
-    // See Bug 1803015
-    NvU64 baseDmaSysmemAddr;
 } UvmGpuAccessCntrInfo;

 typedef enum
@@ -936,7 +1126,18 @@ typedef struct UvmGpuAccessCntrConfig_tag
    NvU32 threshold;
 } UvmGpuAccessCntrConfig;

+//
+// When modifying this enum, make sure they are compatible with the mirrored
+// MEMORY_PROTECTION enum in phys_mem_allocator.h.
+//
+typedef enum UvmPmaGpuMemoryType_tag
+{
+    UVM_PMA_GPU_MEMORY_TYPE_UNPROTECTED = 0,
+    UVM_PMA_GPU_MEMORY_TYPE_PROTECTED   = 1
+} UVM_PMA_GPU_MEMORY_TYPE;
+
 typedef UvmGpuChannelInfo gpuChannelInfo;
+typedef UvmGpuTsgAllocParams gpuTsgAllocParams;
 typedef UvmGpuChannelAllocParams gpuChannelAllocParams;
 typedef UvmGpuCaps gpuCaps;
 typedef UvmGpuCopyEngineCaps gpuCeCaps;
@@ -951,20 +1152,45 @@ typedef UvmGpuAccessCntrConfig gpuAccessCntrConfig;
 typedef UvmGpuFaultInfo gpuFaultInfo;
 typedef UvmGpuMemoryInfo gpuMemoryInfo;
 typedef UvmGpuExternalMappingInfo gpuExternalMappingInfo;
+typedef UvmGpuExternalPhysAddrInfo gpuExternalPhysAddrInfo;
 typedef UvmGpuChannelResourceInfo gpuChannelResourceInfo;
 typedef UvmGpuChannelInstanceInfo gpuChannelInstanceInfo;
 typedef UvmGpuChannelResourceBindParams gpuChannelResourceBindParams;
 typedef UvmGpuFbInfo gpuFbInfo;
 typedef UvmGpuEccInfo gpuEccInfo;
+typedef UvmGpuNvlinkInfo gpuNvlinkInfo;
 typedef UvmGpuPagingChannel *gpuPagingChannelHandle;
 typedef UvmGpuPagingChannelInfo gpuPagingChannelInfo;
 typedef UvmGpuPagingChannelAllocParams gpuPagingChannelAllocParams;
 typedef UvmPmaAllocationOptions gpuPmaAllocationOptions;

+typedef struct UvmCslIv
+{
+    NvU8 iv[12];
+    NvU8 fresh;
+} UvmCslIv;

+typedef enum UvmCslOperation
+{
+    UVM_CSL_OPERATION_ENCRYPT,
+    UVM_CSL_OPERATION_DECRYPT
+} UvmCslOperation;

-
-
-
+typedef enum UVM_KEY_ROTATION_STATUS {
+    // Key rotation complete/not in progress
+    UVM_KEY_ROTATION_STATUS_IDLE = 0,
+    // RM is waiting for clients to report their channels are idle for key rotation
+    UVM_KEY_ROTATION_STATUS_PENDING = 1,
+    // Key rotation is in progress
+    UVM_KEY_ROTATION_STATUS_IN_PROGRESS = 2,
+    // Key rotation timeout failure, RM will RC non-idle channels.
+    // UVM should never see this status value.
+    UVM_KEY_ROTATION_STATUS_FAILED_TIMEOUT = 3,
+    // Key rotation failed because upper threshold was crossed, RM will RC non-idle channels
+    UVM_KEY_ROTATION_STATUS_FAILED_THRESHOLD = 4,
+    // Internal RM failure while rotating keys for a certain channel, RM will RC the channel.
+    UVM_KEY_ROTATION_STATUS_FAILED_ROTATION = 5,
+    UVM_KEY_ROTATION_STATUS_MAX_COUNT = 6,
+} UVM_KEY_ROTATION_STATUS;

 #endif // _NV_UVM_TYPES_H_
--- a/kernel-open/common/inc/nvgputypes.h
+++ b/kernel-open/common/inc/nvgputypes.h
@@ -150,9 +150,7 @@ typedef struct NvSyncPointFenceRec {
 |*                                                                           *|
 \***************************************************************************/

-#if !defined(XAPIGEN)   /* NvOffset is XAPIGEN builtin type, so skip typedef */
 typedef NvU64           NvOffset; /* GPU address                             */
-#endif

 #define NvOffset_HI32(n)  ((NvU32)(((NvU64)(n)) >> 32))
 #define NvOffset_LO32(n)  ((NvU32)((NvU64)(n)))
--- a/kernel-open/common/inc/nvkms-api-types.h
+++ b/kernel-open/common/inc/nvkms-api-types.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2014-2015 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2014-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -29,6 +29,7 @@
 #include <nvlimits.h>

 #define NVKMS_MAX_SUBDEVICES                  NV_MAX_SUBDEVICES
+#define NVKMS_MAX_HEADS_PER_DISP              NV_MAX_HEADS

 #define NVKMS_LEFT                            0
 #define NVKMS_RIGHT                           1
@@ -44,6 +45,13 @@

 #define NVKMS_DEVICE_ID_TEGRA                 0x0000ffff

+#define NVKMS_MAX_SUPERFRAME_VIEWS            4
+
+#define NVKMS_LOG2_LUT_ARRAY_SIZE             10
+#define NVKMS_LUT_ARRAY_SIZE                  (1 << NVKMS_LOG2_LUT_ARRAY_SIZE)
+
+#define NVKMS_OLUT_FP_NORM_SCALE_DEFAULT      0xffffffff
+
 typedef NvU32 NvKmsDeviceHandle;
 typedef NvU32 NvKmsDispHandle;
 typedef NvU32 NvKmsConnectorHandle;
@@ -52,6 +60,7 @@ typedef NvU32 NvKmsFrameLockHandle;
 typedef NvU32 NvKmsDeferredRequestFifoHandle;
 typedef NvU32 NvKmsSwapGroupHandle;
 typedef NvU32 NvKmsVblankSyncObjectHandle;
+typedef NvU32 NvKmsVblankSemControlHandle;

 struct NvKmsSize {
    NvU16 width;
@@ -178,6 +187,14 @@ enum NvKmsEventType {
    NVKMS_EVENT_TYPE_FLIP_OCCURRED,
 };

+enum NvKmsFlipResult {
+    NV_KMS_FLIP_RESULT_SUCCESS = 0,    /* Success */
+    NV_KMS_FLIP_RESULT_INVALID_PARAMS, /* Parameter validation failed */
+    NV_KMS_FLIP_RESULT_IN_PROGRESS,    /* Flip would fail because an outstanding
+                                          flip containing changes that cannot be
+                                          queued is in progress */
+};
+
 typedef enum {
    NV_EVO_SCALER_1TAP      = 0,
    NV_EVO_SCALER_2TAPS     = 1,
@@ -220,6 +237,90 @@ struct NvKmsUsageBounds {
    } layer[NVKMS_MAX_LAYERS_PER_HEAD];
 };

+/*!
+ * Per-component arrays of NvU16s describing the LUT; used for both the input
+ * LUT and output LUT.
+ */
+struct NvKmsLutRamps {
+    NvU16 red[NVKMS_LUT_ARRAY_SIZE];   /*! in */
+    NvU16 green[NVKMS_LUT_ARRAY_SIZE]; /*! in */
+    NvU16 blue[NVKMS_LUT_ARRAY_SIZE];  /*! in */
+};
+
+/* Datatypes for LUT capabilities */
+enum NvKmsLUTFormat {
+    /*
+     * Normalized fixed-point format mapping [0, 1] to [0x0, 0xFFFF].
+     */
+    NVKMS_LUT_FORMAT_UNORM16,
+
+    /*
+     * Half-precision floating point.
+     */
+    NVKMS_LUT_FORMAT_FP16,
+
+    /*
+     * 14-bit fixed-point format required to work around hardware bug 813188.
+     *
+     * To convert from UNORM16 to UNORM14_WAR_813188:
+     * unorm14_war_813188 = ((unorm16 >> 2) & ~7) + 0x6000
+     */
+    NVKMS_LUT_FORMAT_UNORM14_WAR_813188
+};
+
+enum NvKmsLUTVssSupport {
+    NVKMS_LUT_VSS_NOT_SUPPORTED,
+    NVKMS_LUT_VSS_SUPPORTED,
+    NVKMS_LUT_VSS_REQUIRED,
+};
+
+enum NvKmsLUTVssType {
+    NVKMS_LUT_VSS_TYPE_NONE,
+    NVKMS_LUT_VSS_TYPE_LINEAR,
+    NVKMS_LUT_VSS_TYPE_LOGARITHMIC,
+};
+
+struct NvKmsLUTCaps {
+    /*! Whether this layer or head on this device supports this LUT stage. */
+    NvBool supported;
+
+    /*! Whether this LUT supports VSS. */
+    enum NvKmsLUTVssSupport vssSupport;
+
+    /*!
+     * The type of VSS segmenting this LUT uses.
+     */
+    enum NvKmsLUTVssType vssType;
+
+    /*!
+     * Expected number of VSS segments.
+     */
+    NvU32 vssSegments;
+
+    /*!
+     * Expected number of LUT entries.
+     */
+    NvU32 lutEntries;
+
+    /*!
+     * Format for each of the LUT entries.
+     */
+    enum NvKmsLUTFormat entryFormat;
+};
+
+/* each LUT entry uses this many bytes */
+#define NVKMS_LUT_CAPS_LUT_ENTRY_SIZE (4 * sizeof(NvU16))
+
+/* if the LUT surface uses VSS, size of the VSS header */
+#define NVKMS_LUT_VSS_HEADER_SIZE (4 * NVKMS_LUT_CAPS_LUT_ENTRY_SIZE)
+
+struct NvKmsLUTSurfaceParams {
+    NvKmsSurfaceHandle surfaceHandle;
+    NvU64 offset NV_ALIGN_BYTES(8);
+    NvU32 vssSegments;
+    NvU32 lutEntries;
+};
+
 /*
 * A 3x4 row-major colorspace conversion matrix.
 *
@@ -415,9 +516,9 @@ struct NvKmsLayerCapabilities {
    NvBool supportsWindowMode              :1;

    /*!
-     * Whether layer supports HDR pipe.
+     * Whether layer supports ICtCp pipe.
     */
-    NvBool supportsHDR                     :1;
+    NvBool supportsICtCp                   :1;


    /*!
@@ -438,6 +539,10 @@ struct NvKmsLayerCapabilities {
     * still expected to honor the NvKmsUsageBounds for each head.
     */
    NvU64 supportedSurfaceMemoryFormats NV_ALIGN_BYTES(8);
+
+    /* Capabilities for each LUT stage in the EVO3 precomp pipeline. */
+    struct NvKmsLUTCaps ilut;
+    struct NvKmsLUTCaps tmo;
 };

 /*!
@@ -530,4 +635,148 @@ typedef struct {
    NvBool noncoherent;
 } NvKmsDispIOCoherencyModes;

+enum NvKmsInputColorRange {
+    /*
+     * If DEFAULT is provided, driver will assume full range for RGB formats
+     * and limited range for YUV formats.
+     */
+    NVKMS_INPUT_COLORRANGE_DEFAULT = 0,
+
+    NVKMS_INPUT_COLORRANGE_LIMITED = 1,
+
+    NVKMS_INPUT_COLORRANGE_FULL = 2,
+};
+
+enum NvKmsInputColorSpace {
+    /* Unknown colorspace; no de-gamma will be applied */
+    NVKMS_INPUT_COLORSPACE_NONE = 0,
+
+    /* Linear, Rec.709 [-0.5, 7.5) */
+    NVKMS_INPUT_COLORSPACE_SCRGB_LINEAR = 1,
+
+    /* PQ, Rec.2020 unity */
+    NVKMS_INPUT_COLORSPACE_BT2100_PQ = 2,
+};
+
+enum NvKmsOutputColorimetry {
+    NVKMS_OUTPUT_COLORIMETRY_DEFAULT = 0,
+
+    NVKMS_OUTPUT_COLORIMETRY_BT2100 = 1,
+};
+
+enum NvKmsOutputTf {
+    /*
+     * NVKMS itself won't apply any OETF (clients are still
+     * free to provide a custom OLUT)
+     */
+    NVKMS_OUTPUT_TF_NONE = 0,
+    NVKMS_OUTPUT_TF_TRADITIONAL_GAMMA_SDR = 1,
+    NVKMS_OUTPUT_TF_PQ = 2,
+};
+
+/*!
+ * EOTF Data Byte 1 as per CTA-861-G spec.
+ * This is expected to match exactly with the spec.
+ */
+enum NvKmsInfoFrameEOTF {
+    NVKMS_INFOFRAME_EOTF_SDR_GAMMA = 0,
+    NVKMS_INFOFRAME_EOTF_HDR_GAMMA = 1,
+    NVKMS_INFOFRAME_EOTF_ST2084 = 2,
+    NVKMS_INFOFRAME_EOTF_HLG = 3,
+};
+
+/*!
+ * HDR Static Metadata Type1 Descriptor as per CEA-861.3 spec.
+ * This is expected to match exactly with the spec.
+ */
+struct NvKmsHDRStaticMetadata {
+    /*!
+     * Color primaries of the data.
+     * These are coded as unsigned 16-bit values in units of 0.00002,
+     * where 0x0000 represents zero and 0xC350 represents 1.0000.
+     */
+    struct {
+        NvU16 x, y;
+    } displayPrimaries[3];
+
+    /*!
+     * White point of colorspace data.
+     * These are coded as unsigned  16-bit values in units of 0.00002,
+     * where 0x0000 represents zero and 0xC350 represents 1.0000.
+     */
+    struct {
+        NvU16 x, y;
+    } whitePoint;
+
+    /**
+     * Maximum mastering display luminance.
+     * This value is coded as an unsigned 16-bit value in units of 1 cd/m2,
+     * where 0x0001 represents 1 cd/m2 and 0xFFFF represents 65535 cd/m2.
+     */
+    NvU16 maxDisplayMasteringLuminance;
+
+    /*!
+     * Minimum mastering display luminance.
+     * This value is coded as an unsigned 16-bit value in units of
+     * 0.0001 cd/m2, where 0x0001 represents 0.0001 cd/m2 and 0xFFFF
+     * represents 6.5535 cd/m2.
+     */
+    NvU16 minDisplayMasteringLuminance;
+
+    /*!
+     * Maximum content light level.
+     * This value is coded as an unsigned 16-bit value in units of 1 cd/m2,
+     * where 0x0001 represents 1 cd/m2 and 0xFFFF represents 65535 cd/m2.
+     */
+    NvU16 maxCLL;
+
+    /*!
+     * Maximum frame-average light level.
+     * This value is coded as an unsigned 16-bit value in units of 1 cd/m2,
+     * where 0x0001 represents 1 cd/m2 and 0xFFFF represents 65535 cd/m2.
+     */
+    NvU16 maxFALL;
+};
+
+/*!
+ * A superframe is made of two or more video streams that are combined in
+ * a specific way. A DP serializer (an external device connected to a Tegra
+ * ARM SOC over DP or HDMI) can receive a video stream comprising multiple
+ * videos combined into a single frame and then split it into multiple
+ * video streams. The following structure describes the number of views
+ * and dimensions of each view inside a superframe.
+ */
+struct NvKmsSuperframeInfo {
+    NvU8 numViews;
+    struct {
+        /* x offset inside superframe at which this view starts */
+        NvU16 x;
+
+        /* y offset inside superframe at which this view starts */
+        NvU16 y;
+
+        /* Horizontal active width in pixels for this view */
+        NvU16 width;
+
+        /* Vertical active height in lines for this view */
+        NvU16 height;
+    } view[NVKMS_MAX_SUPERFRAME_VIEWS];
+};
+
+/* Fields within NvKmsVblankSemControlDataOneHead::flags */
+#define NVKMS_VBLANK_SEM_CONTROL_SWAP_INTERVAL          15:0
+
+struct NvKmsVblankSemControlDataOneHead {
+    NvU32 requestCounterAccel;
+    NvU32 requestCounter;
+    NvU32 flags;
+
+    NvU32 semaphore;
+    NvU64 vblankCount NV_ALIGN_BYTES(8);
+};
+
+struct NvKmsVblankSemControlData {
+    struct NvKmsVblankSemControlDataOneHead head[NV_MAX_HEADS];
+};
+
 #endif /* NVKMS_API_TYPES_H */
--- a/kernel-open/common/inc/nvkms-format.h
+++ b/kernel-open/common/inc/nvkms-format.h
@@ -86,8 +86,9 @@ enum NvKmsSurfaceMemoryFormat {
    NvKmsSurfaceMemoryFormatY12___V12U12_N420 = 32,
    NvKmsSurfaceMemoryFormatY8___U8___V8_N444 = 33,
    NvKmsSurfaceMemoryFormatY8___U8___V8_N420 = 34,
+    NvKmsSurfaceMemoryFormatRF16GF16BF16XF16  = 35,
    NvKmsSurfaceMemoryFormatMin = NvKmsSurfaceMemoryFormatI8,
-    NvKmsSurfaceMemoryFormatMax = NvKmsSurfaceMemoryFormatY8___U8___V8_N420,
+    NvKmsSurfaceMemoryFormatMax = NvKmsSurfaceMemoryFormatRF16GF16BF16XF16,
 };

 typedef struct NvKmsSurfaceMemoryFormatInfo {
--- a/kernel-open/common/inc/nvkms-kapi.h
+++ b/kernel-open/common/inc/nvkms-kapi.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2015 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2015-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -49,6 +49,8 @@ struct NvKmsKapiDevice;
 struct NvKmsKapiMemory;
 struct NvKmsKapiSurface;
 struct NvKmsKapiChannelEvent;
+struct NvKmsKapiSemaphoreSurface;
+struct NvKmsKapiSemaphoreSurfaceCallback;

 typedef NvU32 NvKmsKapiConnector;
 typedef NvU32 NvKmsKapiDisplay;
@@ -67,6 +69,14 @@ typedef NvU32 NvKmsKapiDisplay;
 */
 typedef void NvKmsChannelEventProc(void *dataPtr, NvU32 dataU32);

+/*
+ * Note: Same as above, this function must not call back into NVKMS-KAPI, nor
+ * directly into RM. Doing so could cause deadlocks given the notification
+ * function will most likely be called from within RM's interrupt handler
+ * callchain.
+ */
+typedef void NvKmsSemaphoreSurfaceCallbackProc(void *pData);
+
 /** @} */

 /**
@@ -114,6 +124,14 @@ struct NvKmsKapiDisplayMode {
 #define NVKMS_KAPI_LAYER_INVALID_IDX           0xff
 #define NVKMS_KAPI_LAYER_PRIMARY_IDX              0

+struct NvKmsKapiLutCaps {
+    struct {
+        struct NvKmsLUTCaps ilut;
+        struct NvKmsLUTCaps tmo;
+    } layer[NVKMS_KAPI_LAYER_MAX];
+    struct NvKmsLUTCaps olut;
+};
+
 struct NvKmsKapiDeviceResourcesInfo {

    NvU32 numHeads;
@@ -126,6 +144,11 @@ struct NvKmsKapiDeviceResourcesInfo {
        NvU32 validCursorCompositionModes;
        NvU64 supportedCursorSurfaceMemoryFormats;

+        struct {
+            NvU64 maxSubmittedOffset;
+            NvU64 stride;
+        } semsurf;
+
        struct {
            NvU16 validRRTransforms;
            NvU32 validCompositionModes;
@@ -143,12 +166,19 @@ struct NvKmsKapiDeviceResourcesInfo {

        NvU32 hasVideoMemory;

+        NvU32 numDisplaySemaphores;
+
        NvU8  genericPageKind;

        NvBool  supportsSyncpts;
+
+        NvBool requiresVrrSemaphores;
    } caps;

    NvU64 supportedSurfaceMemoryFormats[NVKMS_KAPI_LAYER_MAX];
+    NvBool supportsICtCp[NVKMS_KAPI_LAYER_MAX];
+
+    struct NvKmsKapiLutCaps lutCaps;
 };

 #define NVKMS_KAPI_LAYER_MASK(layerType) (1 << (layerType))
@@ -164,8 +194,6 @@ struct NvKmsKapiConnectorInfo {

    NvU32 physicalIndex;

-    NvU32 headMask;
-
    NvKmsConnectorSignalFormat signalFormat;
    NvKmsConnectorType         type;

@@ -193,20 +221,29 @@ struct NvKmsKapiStaticDisplayInfo {
    NvU32  numPossibleClones;
    NvKmsKapiDisplay possibleCloneHandles[NVKMS_KAPI_MAX_CLONE_DISPLAYS];

+    NvU32 headMask;
 };

-struct NvKmsKapiSyncpt {
+struct NvKmsKapiSyncParams {
+    union {
+        struct {
+            /*!
+             * Possible syncpt use case in kapi.
+             * For pre-syncpt, use only id and value
+             * and for post-syncpt, use only fd.
+             */
+            NvU32   preSyncptId;
+            NvU32   preSyncptValue;
+        } syncpt;

-    /*!
-     * Possible syncpt use case in kapi.
-     * For pre-syncpt, use only id and value
-     * and for post-syncpt, use only fd.
-     */
-    NvBool  preSyncptSpecified;
-    NvU32   preSyncptId;
-    NvU32   preSyncptValue;
+        struct {
+            NvU32 index;
+        } semaphore;
+    } u;

-    NvBool  postSyncptRequested;
+    NvBool preSyncptSpecified;
+    NvBool postSyncptRequested;
+    NvBool semaphoreSpecified;
 };

 struct NvKmsKapiLayerConfig {
@@ -216,7 +253,14 @@ struct NvKmsKapiLayerConfig {
        NvU8 surfaceAlpha;
    } compParams;
    struct NvKmsRRParams rrParams;
-    struct NvKmsKapiSyncpt syncptParams;
+    struct NvKmsKapiSyncParams syncParams;
+
+    struct {
+        struct NvKmsHDRStaticMetadata val;
+        NvBool enabled;
+    } hdrMetadata;
+
+    enum NvKmsOutputTf tf;

    NvU8 minPresentInterval;
    NvBool tearing;
@@ -226,16 +270,56 @@ struct NvKmsKapiLayerConfig {

    NvS16 dstX, dstY;
    NvU16 dstWidth, dstHeight;
+
+    enum NvKmsInputColorSpace inputColorSpace;
+
+    struct {
+        NvBool enabled;
+        struct NvKmsKapiSurface *lutSurface;
+        NvU64 offset;
+        NvU32 vssSegments;
+        NvU32 lutEntries;
+    } ilut;
+
+    struct {
+        NvBool enabled;
+        struct NvKmsKapiSurface *lutSurface;
+        NvU64 offset;
+        NvU32 vssSegments;
+        NvU32 lutEntries;
+    } tmo;
+
+    struct NvKmsCscMatrix csc;
+    NvBool cscUseMain;
+
+    struct {
+        struct NvKmsCscMatrix lmsCtm;
+        struct NvKmsCscMatrix lmsToItpCtm;
+        struct NvKmsCscMatrix itpToLmsCtm;
+        struct NvKmsCscMatrix blendCtm;
+        struct {
+            NvBool lmsCtm      : 1;
+            NvBool lmsToItpCtm : 1;
+            NvBool itpToLmsCtm : 1;
+            NvBool blendCtm    : 1;
+        } enabled;
+    } matrixOverrides;
 };

 struct NvKmsKapiLayerRequestedConfig {
    struct NvKmsKapiLayerConfig config;
    struct {
-        NvBool surfaceChanged : 1;
-        NvBool srcXYChanged   : 1;
-        NvBool srcWHChanged   : 1;
-        NvBool dstXYChanged   : 1;
-        NvBool dstWHChanged   : 1;
+        NvBool surfaceChanged          : 1;
+        NvBool srcXYChanged            : 1;
+        NvBool srcWHChanged            : 1;
+        NvBool dstXYChanged            : 1;
+        NvBool dstWHChanged            : 1;
+        NvBool cscChanged              : 1;
+        NvBool tfChanged               : 1;
+        NvBool hdrMetadataChanged      : 1;
+        NvBool matrixOverridesChanged  : 1;
+        NvBool ilutChanged             : 1;
+        NvBool tmoChanged              : 1;
    } flags;
 };

@@ -277,14 +361,54 @@ struct NvKmsKapiHeadModeSetConfig {
    NvKmsKapiDisplay displays[NVKMS_KAPI_MAX_CLONE_DISPLAYS];

    struct NvKmsKapiDisplayMode mode;
+
+    NvBool vrrEnabled;
+
+    struct {
+        NvBool enabled;
+        enum NvKmsInfoFrameEOTF eotf;
+        struct NvKmsHDRStaticMetadata staticMetadata;
+    } hdrInfoFrame;
+
+    enum NvKmsOutputColorimetry colorimetry;
+
+    struct {
+        struct {
+            NvU32 depth;
+            NvU32 start;
+            NvU32 end;
+            struct NvKmsLutRamps *pRamps;
+        } input;
+
+        struct {
+            NvBool enabled;
+            struct NvKmsLutRamps *pRamps;
+        } output;
+    } lut;
+
+    struct {
+        NvBool enabled;
+        struct NvKmsKapiSurface *lutSurface;
+        NvU64 offset;
+        NvU32 vssSegments;
+        NvU32 lutEntries;
+    } olut;
+
+    NvU32 olutFpNormScale;
 };

 struct NvKmsKapiHeadRequestedConfig {
    struct NvKmsKapiHeadModeSetConfig modeSetConfig;
    struct {
-        NvBool activeChanged   : 1;
-        NvBool displaysChanged : 1;
-        NvBool modeChanged     : 1;
+        NvBool activeChanged          : 1;
+        NvBool displaysChanged        : 1;
+        NvBool modeChanged            : 1;
+        NvBool hdrInfoFrameChanged    : 1;
+        NvBool colorimetryChanged     : 1;
+        NvBool legacyIlutChanged      : 1;
+        NvBool legacyOlutChanged      : 1;
+        NvBool olutChanged            : 1;
+        NvBool olutFpNormScaleChanged : 1;
    } flags;

    struct NvKmsKapiCursorRequestedConfig cursorRequestedConfig;
@@ -309,6 +433,9 @@ struct NvKmsKapiHeadReplyConfig {
 };

 struct NvKmsKapiModeSetReplyConfig {
+    enum NvKmsFlipResult flipResult;
+    NvBool vrrFlip;
+    NvS32 vrrSemaphoreIndex;
    struct NvKmsKapiHeadReplyConfig
        headReplyConfig[NVKMS_KAPI_MAX_HEADS];
 };
@@ -368,6 +495,9 @@ struct NvKmsKapiDynamicDisplayParams {
    /* [OUT] Connection status */
    NvU32 connected;

+    /* [OUT] VRR status */
+    NvBool vrrSupported;
+
    /* [IN/OUT] EDID of connected monitor/ Input to override EDID */
    struct {
        NvU16  bufferSize;
@@ -416,6 +546,20 @@ struct NvKmsKapiCreateSurfaceParams {
    NvU8 log2GobsPerBlockY;
 };

+enum NvKmsKapiAllocationType {
+    NVKMS_KAPI_ALLOCATION_TYPE_SCANOUT = 0,
+    NVKMS_KAPI_ALLOCATION_TYPE_NOTIFIER = 1,
+    NVKMS_KAPI_ALLOCATION_TYPE_OFFSCREEN = 2,
+};
+
+typedef enum NvKmsKapiRegisterWaiterResultRec {
+    NVKMS_KAPI_REG_WAITER_FAILED,
+    NVKMS_KAPI_REG_WAITER_SUCCESS,
+    NVKMS_KAPI_REG_WAITER_ALREADY_SIGNALLED,
+} NvKmsKapiRegisterWaiterResult;
+
+typedef void NvKmsKapiSuspendResumeCallbackFunc(NvBool suspend);
+
 struct NvKmsKapiFunctionsTable {

    /*!
@@ -478,6 +622,75 @@ struct NvKmsKapiFunctionsTable {
     */
    void (*releaseOwnership)(struct NvKmsKapiDevice *device);

+    /*!
+     * Grant modeset permissions for a display to fd. Only one (dispIndex, head,
+     * display) is currently supported.
+     *
+     * \param [in]  fd         fd from opening /dev/nvidia-modeset.
+     *
+     * \param [in]  device     A device returned by allocateDevice().
+     *
+     * \param [in]  head       head of display.
+     *
+     * \param [in]  display    The display to grant.
+     *
+     * \return NV_TRUE on success, NV_FALSE on failure.
+     */
+    NvBool (*grantPermissions)
+    (
+        NvS32 fd,
+        struct NvKmsKapiDevice *device,
+        NvU32 head,
+        NvKmsKapiDisplay display
+    );
+
+    /*!
+     * Revoke modeset permissions previously granted. Only one (dispIndex,
+     * head, display) is currently supported.
+     *
+     * \param [in]  device     A device returned by allocateDevice().
+     *
+     * \param [in]  head       head of display.
+     *
+     * \param [in]  display    The display to revoke.
+     *
+     * \return NV_TRUE on success, NV_FALSE on failure.
+     */
+    NvBool (*revokePermissions)
+    (
+        struct NvKmsKapiDevice *device,
+        NvU32 head,
+        NvKmsKapiDisplay display
+    );
+
+    /*!
+     * Grant modeset sub-owner permissions to fd. This is used by clients to
+     * convert drm 'master' permissions into nvkms sub-owner permission.
+     *
+     * \param [in]  fd         fd from opening /dev/nvidia-modeset.
+     *
+     * \param [in]  device     A device returned by allocateDevice().
+     *
+     * \return NV_TRUE on success, NV_FALSE on failure.
+     */
+    NvBool (*grantSubOwnership)
+    (
+        NvS32 fd,
+        struct NvKmsKapiDevice *device
+    );
+
+    /*!
+     * Revoke sub-owner permissions previously granted.
+     *
+     * \param [in]  device     A device returned by allocateDevice().
+     *
+     * \return NV_TRUE on success, NV_FALSE on failure.
+     */
+    NvBool (*revokeSubOwnership)
+    (
+        struct NvKmsKapiDevice *device
+    );
+
    /*!
     * Registers for notification, via
     * NvKmsKapiAllocateDeviceParams::eventCallback, of the events specified
@@ -609,6 +822,8 @@ struct NvKmsKapiFunctionsTable {
     * \param [in] device  A device allocated using allocateDevice().
     *
     * \param [in] layout  BlockLinear or Pitch.
+     * 
+     * \param [in] type    Allocation type.
     *
     * \param [in] size    Size, in bytes, of the memory to allocate.
     *
@@ -624,6 +839,7 @@ struct NvKmsKapiFunctionsTable {
    (
        struct NvKmsKapiDevice *device,
        enum NvKmsSurfaceMemoryLayout layout,
+        enum NvKmsKapiAllocationType type,
        NvU64 size,
        NvU8 *compressible
    );
@@ -637,6 +853,8 @@ struct NvKmsKapiFunctionsTable {
     * \param [in] device  A device allocated using allocateDevice().
     *
     * \param [in] layout  BlockLinear or Pitch.
+     * 
+     * \param [in] type    Allocation type.
     *
     * \param [in] size    Size, in bytes, of the memory to allocate.
     *
@@ -652,6 +870,7 @@ struct NvKmsKapiFunctionsTable {
    (
        struct NvKmsKapiDevice *device,
        enum NvKmsSurfaceMemoryLayout layout,
+        enum NvKmsKapiAllocationType type,
        NvU64 size,
        NvU8 *compressible
    );
@@ -1042,6 +1261,299 @@ struct NvKmsKapiFunctionsTable {
                                       NvP64 dmaBuf,
                                       NvU32 limit);

+    /*!
+     * Import a semaphore surface allocated elsewhere to NVKMS and return a
+     * handle to the new object.
+     *
+     * \param [in] device            A device allocated using allocateDevice().
+     *
+     * \param [in] nvKmsParamsUser   Userspace pointer to driver-specific
+     *                               parameters describing the semaphore
+     *                               surface being imported.
+     *
+     * \param [in] nvKmsParamsSize   Size of the driver-specific parameter
+     *                               struct.
+     *
+     * \param [out] pSemaphoreMap    Returns a CPU mapping of the semaphore
+     *                               surface's semaphore memory to the client.
+     *
+     * \param [out] pMaxSubmittedMap Returns a CPU mapping of the semaphore
+     *                               surface's semaphore memory to the client.
+     *
+     * \return struct NvKmsKapiSemaphoreSurface* on success, NULL on failure.
+     */
+    struct NvKmsKapiSemaphoreSurface* (*importSemaphoreSurface)
+    (
+         struct NvKmsKapiDevice *device,
+         NvU64 nvKmsParamsUser,
+         NvU64 nvKmsParamsSize,
+         void **pSemaphoreMap,
+         void **pMaxSubmittedMap
+    );
+
+    /*!
+     * Free an imported semaphore surface.
+     *
+     * \param [in]  device              The device passed to
+     *                                  importSemaphoreSurface() when creating
+     *                                  semaphoreSurface.
+     *
+     * \param [in]  semaphoreSurface    A semaphore surface returned by
+     *                                  importSemaphoreSurface().
+     */
+    void (*freeSemaphoreSurface)
+    (
+        struct NvKmsKapiDevice *device,
+        struct NvKmsKapiSemaphoreSurface *semaphoreSurface
+    );
+
+    /*!
+     * Register a callback to be called when a semaphore reaches a value.
+     *
+     * The callback will be called when the semaphore at index in
+     * semaphoreSurface reaches the value wait_value.  The callback will
+     * be called at most once and is automatically unregistered when called.
+     * It may also be unregistered (i.e., cancelled) explicitly using the
+     * unregisterSemaphoreSurfaceCallback() function. To avoid leaking the
+     * memory used to track the registered callback, callers must ensure one
+     * of these methods of unregistration is used for every successful
+     * callback registration that returns a non-NULL pCallbackHandle.
+     *
+     * \param [in]  device              The device passed to
+     *                                  importSemaphoreSurface() when creating
+     *                                  semaphoreSurface.
+     *
+     * \param [in]  semaphoreSurface    A semaphore surface returned by
+     *                                  importSemaphoreSurface().
+     *
+     * \param [in]  pCallback           A pointer to the function to call when
+     *                                  the specified value is reached. NULL
+     *                                  means no callback.
+     *
+     * \param [in]  pData               Arbitrary data to be passed back to the
+     *                                  callback as its sole parameter.
+     *
+     * \param [in]  index               The index of the semaphore within
+     *                                  semaphoreSurface.
+     *
+     * \param [in]  wait_value          The value the semaphore must reach or
+     *                                  exceed before the callback is called.
+     *
+     * \param [in]  new_value           The value the semaphore will be set to
+     *                                  when it reaches or exceeds <wait_value>.
+     *                                  0 means do not update the value.
+     *
+     * \param [out] pCallbackHandle     On success, the value pointed to will
+     *                                  contain an opaque handle to the
+     *                                  registered callback that may be used to
+     *                                  cancel it if needed. Unused if pCallback
+     *                                  is NULL.
+     *
+     * \return NVKMS_KAPI_REG_WAITER_SUCCESS if the waiter was registered or if
+     *         no callback was requested and the semaphore at <index> has
+     *         already reached or exceeded <wait_value>
+     *
+     *         NVKMS_KAPI_REG_WAITER_ALREADY_SIGNALLED if a callback was
+     *         requested and the semaphore at <index> has already reached or
+     *         exceeded <wait_value>
+     *
+     *         NVKMS_KAPI_REG_WAITER_FAILED if waiter registration failed.
+     */
+    NvKmsKapiRegisterWaiterResult
+    (*registerSemaphoreSurfaceCallback)
+    (
+        struct NvKmsKapiDevice *device,
+        struct NvKmsKapiSemaphoreSurface *semaphoreSurface,
+        NvKmsSemaphoreSurfaceCallbackProc *pCallback,
+        void *pData,
+        NvU64 index,
+        NvU64 wait_value,
+        NvU64 new_value,
+        struct NvKmsKapiSemaphoreSurfaceCallback **pCallbackHandle
+    );
+
+    /*!
+     * Unregister a callback registered via registerSemaphoreSurfaceCallback()
+     *
+     * If the callback has not yet been called, this function will cancel the
+     * callback and free its associated resources.
+     *
+     * Note this function treats the callback handle as a pointer. While this
+     * function does not dereference that pointer itself, the underlying call
+     * to RM does within a properly guarded critical section that first ensures
+     * it is not in the process of being used within a callback. This means
+     * the callstack must take into consideration that pointers are not in
+     * general unique handles if they may have been freed, since a subsequent
+     * malloc could return the same pointer value at that point. This callchain
+     * avoids that by leveraging the behavior of the underlying RM APIs:
+     *
+     * 1) A callback handle is referenced relative to its corresponding
+     *    (semaphore surface, index, wait_value) tuple here and within RM. It
+     *    is not a valid handle outside of that scope.
+     *
+     * 2) A callback can not be registered against an already-reached value
+     *    for a given semaphore surface index.
+     *
+     * 3) A given callback handle can not be registered twice against the same
+     *    (semaphore surface, index, wait_value) tuple, so unregistration will
+     *    never race with registration at the RM level, and would only race at
+     *    a higher level if used incorrectly. Since this is kernel code, we
+     *    can safely assume there won't be malicious clients purposely misuing
+     *    the API, but the burden is placed on the caller to ensure its usage
+     *    does not lead to races at higher levels.
+     *
+     * These factors considered together ensure any valid registered handle is
+     * either still in the relevant waiter list and refers to the same event/
+     * callback as when it was registered, or has been removed from the list
+     * as part of a critical section that also destroys the list itself and
+     * makes future lookups in that list impossible, and hence eliminates the
+     * chance of comparing a stale handle with a new handle of the same value
+     * as part of a lookup.
+     *
+     * \param [in]  device              The device passed to
+     *                                  importSemaphoreSurface() when creating
+     *                                  semaphoreSurface.
+     *
+     * \param [in]  semaphoreSurface    The semaphore surface passed to
+     *                                  registerSemaphoreSurfaceCallback() when
+     *                                  registering the callback.
+     *
+     * \param [in]  index               The index passed to
+     *                                  registerSemaphoreSurfaceCallback() when
+     *                                  registering the callback.
+     *
+     * \param [in]  wait_value          The wait_value passed to
+     *                                  registerSemaphoreSurfaceCallback() when
+     *                                  registering the callback.
+     *
+     * \param [in]  callbackHandle      The callback handle returned by
+     *                                  registerSemaphoreSurfaceCallback().
+     */
+    NvBool
+    (*unregisterSemaphoreSurfaceCallback)
+    (
+        struct NvKmsKapiDevice *device,
+        struct NvKmsKapiSemaphoreSurface *semaphoreSurface,
+        NvU64 index,
+        NvU64 wait_value,
+        struct NvKmsKapiSemaphoreSurfaceCallback *callbackHandle
+    );
+
+    /*!
+     * Update the value of a semaphore surface from the CPU.
+     *
+     * Update the semaphore value at the specified index from the CPU, then
+     * wake up any pending CPU waiters associated with that index that are
+     * waiting on it reaching a value <= the new value.
+     */
+    NvBool
+    (*setSemaphoreSurfaceValue)
+    (
+        struct NvKmsKapiDevice *device,
+        struct NvKmsKapiSemaphoreSurface *semaphoreSurface,
+        NvU64 index,
+        NvU64 new_value
+    );
+
+    /*!
+     * Set the callback function for suspending and resuming the display system.
+     */
+    void
+    (*setSuspendResumeCallback)
+    (
+        NvKmsKapiSuspendResumeCallbackFunc *function
+    );
+
+    /*!
+     * Immediately initialize the specified display semaphore to the pending state.
+     *
+     * Must be called prior to applying a mode set that utilizes the specified
+     * display semaphore for synchronization.
+     *
+     * \param [in] device         The device which will utilize the semaphore.
+     *
+     * \param [in] semaphoreIndex Index of the desired semaphore within the
+     *                            NVKMS semaphore pool. Must be less than
+     *                            NvKmsKapiDeviceResourcesInfo::caps::numDisplaySemaphores
+     *                            for the specified device.
+     */
+    NvBool
+    (*tryInitDisplaySemaphore)
+    (
+        struct NvKmsKapiDevice *device,
+        NvU32 semaphoreIndex
+    );
+
+    /*!
+     * Immediately set the specified display semaphore to the displayable state.
+     *
+     * Must be called after \ref tryInitDisplaySemaphore to indicate a mode
+     * configuration change that utilizes the specified display semaphore for
+     * synchronization may proceed.
+     *
+     * \param [in] device         The device which will utilize the semaphore.
+     *
+     * \param [in] semaphoreIndex Index of the desired semaphore within the
+     *                            NVKMS semaphore pool. Must be less than
+     *                            NvKmsKapiDeviceResourcesInfo::caps::numDisplaySemaphores
+     *                            for the specified device.
+     */
+    void
+    (*signalDisplaySemaphore)
+    (
+        struct NvKmsKapiDevice *device,
+        NvU32 semaphoreIndex
+    );
+
+    /*!
+     * Immediately cancel use of a display semaphore by resetting its value to
+     * its initial state.
+     *
+     * This can be used by clients to restore a semaphore to a consistent state
+     * when they have prepared it for use by previously calling
+     * \ref tryInitDisplaySemaphore() on it, but are then prevented from
+     * submitting the associated hardware operations to consume it due to the
+     * subsequent failure of some software or hardware operation.
+     *
+     * \param [in] device         The device which will utilize the semaphore.
+     *
+     * \param [in] semaphoreIndex Index of the desired semaphore within the
+     *                            NVKMS semaphore pool. Must be less than
+     *                            NvKmsKapiDeviceResourcesInfo::caps::numDisplaySemaphores
+     *                            for the specified device.
+     */
+    void
+    (*cancelDisplaySemaphore)
+    (
+        struct NvKmsKapiDevice *device,
+        NvU32 semaphoreIndex
+    );
+
+    /*!
+     * Signal the VRR semaphore at the specified index from the CPU.
+     * If device does not support VRR semaphores, this is a no-op.
+     * Returns true if signal is success or no-op, otherwise returns false.
+     *
+     * \param [in]  device  A device allocated using allocateDevice().
+     *
+     * \param [in]  index   The VRR semaphore index to be signalled.
+     */
+    NvBool
+    (*signalVrrSemaphore)
+    (
+        struct NvKmsKapiDevice *device,
+        NvS32 index
+    );
+
+    /*
+     * Notify NVKMS that the system's framebuffer console has been disabled and
+     * the reserved allocation for the old framebuffer console can be unmapped.
+     */
+    void
+    (*framebufferConsoleDisabled)
+    (
+        struct NvKmsKapiDevice *device
+    );
 };

 /** @} */
@@ -1056,6 +1568,20 @@ NvBool nvKmsKapiGetFunctionsTable
    struct NvKmsKapiFunctionsTable *funcsTable
 );

+NvU32 nvKmsKapiF16ToF32(NvU16 a);
+
+NvU16 nvKmsKapiF32ToF16(NvU32 a);
+
+NvU32 nvKmsKapiF32Mul(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32Div(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32Add(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32ToUI32RMinMag(NvU32 a, NvBool exact);
+
+NvU32 nvKmsKapiUI32ToF32(NvU32 a);
+
 /** @} */

 #endif /* defined(__NVKMS_KAPI_H__) */
--- a/kernel-open/common/inc/nvlimits.h
+++ b/kernel-open/common/inc/nvlimits.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2017 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2017-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -25,35 +25,34 @@

 //
 // This file was generated with FINN, an NVIDIA coding tool.
-// Source file: nvlimits.finn
+// Source file:      nvlimits.finn
 //




-
-
-
-
-
-
-
 /*
 * This is the maximum number of GPUs supported in a single system.
 */
-#define NV_MAX_DEVICES          32
+#define NV_MAX_DEVICES                32

 /*
 * This is the maximum number of subdevices within a single device.
 */
-#define NV_MAX_SUBDEVICES       8
+#define NV_MAX_SUBDEVICES             8

 /*
 * This is the maximum length of the process name string.
 */
-#define NV_PROC_NAME_MAX_LENGTH 100U
+#define NV_PROC_NAME_MAX_LENGTH       100U

 /*
 * This is the maximum number of heads per GPU.
 */
-#define NV_MAX_HEADS            4
+#define NV_MAX_HEADS                  4
+
+/*
+ * Maximum length of a MIG device UUID. It is a 36-byte UUID string plus a
+ * 4-byte prefix and NUL terminator: 'M' 'I' 'G' '-' UUID '\0x0'
+ */
+#define NV_MIG_DEVICE_UUID_STR_LENGTH 41U
--- a/kernel-open/common/inc/nvmisc.h
+++ b/kernel-open/common/inc/nvmisc.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 1993-2020 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -67,6 +67,9 @@ extern "C" {
 #define NVBIT64(b)                NVBIT_TYPE(b, NvU64)
 #endif

+//Concatenate 2 32bit values to a 64bit value
+#define NV_CONCAT_32_TO_64(hi, lo) ((((NvU64)hi) << 32) | ((NvU64)lo))
+
 // Helper macro's for 32 bit bitmasks
 #define NV_BITMASK32_ELEMENT_SIZE            (sizeof(NvU32) << 3)
 #define NV_BITMASK32_IDX(chId)               (((chId) & ~(0x1F)) >> 5)  
@@ -234,12 +237,14 @@ extern "C" {
 #define DRF_EXTENT(drf)         (drf##_HIGH_FIELD)
 #define DRF_SHIFT(drf)          ((drf##_LOW_FIELD) % 32U)
 #define DRF_SHIFT_RT(drf)       ((drf##_HIGH_FIELD) % 32U)
+#define DRF_SIZE(drf)           ((drf##_HIGH_FIELD)-(drf##_LOW_FIELD)+1U)
 #define DRF_MASK(drf)           (0xFFFFFFFFU >> (31U - ((drf##_HIGH_FIELD) % 32U) + ((drf##_LOW_FIELD) % 32U)))
 #else
 #define DRF_BASE(drf)           (NV_FALSE?drf)  // much better
 #define DRF_EXTENT(drf)         (NV_TRUE?drf)  // much better
 #define DRF_SHIFT(drf)          (((NvU32)DRF_BASE(drf)) % 32U)
 #define DRF_SHIFT_RT(drf)       (((NvU32)DRF_EXTENT(drf)) % 32U)
+#define DRF_SIZE(drf)           (DRF_EXTENT(drf)-DRF_BASE(drf)+1U)
 #define DRF_MASK(drf)           (0xFFFFFFFFU>>(31U - DRF_SHIFT_RT(drf) + DRF_SHIFT(drf)))
 #endif
 #define DRF_DEF(d,r,f,c)        (((NvU32)(NV ## d ## r ## f ## c))<<DRF_SHIFT(NV ## d ## r ## f))
@@ -249,12 +254,12 @@ extern "C" {
 #define DRF_EXTENT(drf)         (1?drf)  // much better
 #define DRF_SHIFT(drf)          ((DRF_ISBIT(0,drf)) % 32)
 #define DRF_SHIFT_RT(drf)       ((DRF_ISBIT(1,drf)) % 32)
+#define DRF_SIZE(drf)           (DRF_EXTENT(drf)-DRF_BASE(drf)+1U)
 #define DRF_MASK(drf)           (0xFFFFFFFFU>>(31-((DRF_ISBIT(1,drf)) % 32)+((DRF_ISBIT(0,drf)) % 32)))
 #define DRF_DEF(d,r,f,c)        ((NV ## d ## r ## f ## c)<<DRF_SHIFT(NV ## d ## r ## f))
 #define DRF_NUM(d,r,f,n)        (((n)&DRF_MASK(NV ## d ## r ## f))<<DRF_SHIFT(NV ## d ## r ## f))
 #endif
 #define DRF_SHIFTMASK(drf)      (DRF_MASK(drf)<<(DRF_SHIFT(drf)))
-#define DRF_SIZE(drf)           (DRF_EXTENT(drf)-DRF_BASE(drf)+1U)

 #define DRF_VAL(d,r,f,v)        (((v)>>DRF_SHIFT(NV ## d ## r ## f))&DRF_MASK(NV ## d ## r ## f))
 #endif
@@ -492,6 +497,23 @@ do                                                      \
 //
 #define NV_TWO_N_MINUS_ONE(n) (((1ULL<<(n/2))<<((n+1)/2))-1)

+//
+// Create a 64b bitmask with n bits set
+// This is the same as ((1ULL<<n) - 1), but it doesn't overflow for n=64
+//
+// ...
+// n=-1, 0x0000000000000000
+// n=0,  0x0000000000000000
+// n=1,  0x0000000000000001
+// ...
+// n=63, 0x7FFFFFFFFFFFFFFF
+// n=64, 0xFFFFFFFFFFFFFFFF
+// n=65, 0xFFFFFFFFFFFFFFFF
+// n=66, 0xFFFFFFFFFFFFFFFF
+// ...
+//
+#define NV_BITMASK64(n) ((n<1) ? 0ULL : (NV_U64_MAX>>((n>64) ? 0 : (64-n))))
+
 #define DRF_READ_1WORD_BS(d,r,f,v) \
    ((DRF_EXTENT_MW(NV##d##r##f)<8)?DRF_READ_1BYTE_BS(NV##d##r##f,(v)): \
    ((DRF_EXTENT_MW(NV##d##r##f)<16)?DRF_READ_2BYTE_BS(NV##d##r##f,(v)): \
@@ -572,6 +594,13 @@ nvMaskPos32(const NvU32 mask, const NvU32 bitIdx)
    n32 = BIT_IDX_32(LOWESTBIT(n32));\
 }

+// Destructive operation on n64
+#define LOWESTBITIDX_64(n64)         \
+{                                    \
+    n64 = BIT_IDX_64(LOWESTBIT(n64));\
+}
+
+
 // Destructive operation on n32
 #define HIGHESTBITIDX_32(n32)   \
 {                               \
@@ -692,6 +721,42 @@ nvPrevPow2_U64(const NvU64 x )
    }                                                       \
 }

+//
+// Bug 4851259: Newly added functions must be hidden from certain HS-signed
+// ucode compilers to avoid signature mismatch.
+//
+#ifndef NVDEC_1_0
+/*!
+ * Returns the position of nth set bit in the given mask.
+ *
+ * Returns -1 if mask has fewer than n bits set.
+ *
+ * n is 0 indexed and has valid values 0..31 inclusive, so "zeroth" set bit is
+ * the first set LSB.
+ *
+ * Example, if mask = 0x000000F0u and n = 1, the return value will be 5.
+ * Example, if mask = 0x000000F0u and n = 4, the return value will be -1.
+ */
+static NV_FORCEINLINE NvS32
+nvGetNthSetBitIndex32(NvU32 mask, NvU32 n)
+{
+    NvU32 seenSetBitsCount = 0;
+    NvS32 index;
+    FOR_EACH_INDEX_IN_MASK(32, index, mask)
+    {
+        if (seenSetBitsCount == n)
+        {
+            return index;
+        }
+        ++seenSetBitsCount;
+    }
+    FOR_EACH_INDEX_IN_MASK_END;
+
+    return -1;
+}
+
+#endif // NVDEC_1_0
+
 //
 // Size to use when declaring variable-sized arrays
 //
@@ -907,6 +972,24 @@ static NV_FORCEINLINE void *NV_NVUPTR_TO_PTR(NvUPtr address)
    return uAddr.p;
 }

+// Get bit at pos (k) from x
+#define NV_BIT_GET(k, x)                       (((x) >> (k)) & 1)
+// Get bit at pos (n) from (hi) if >= 64, otherwise from (lo). This is paired with NV_BIT_SET_128 which sets the bit.
+#define NV_BIT_GET_128(n, lo, hi)              (((n) < 64) ? NV_BIT_GET((n), (lo)) : NV_BIT_GET((n) - 64, (hi)))
+//
+// Set the bit at pos (b) for U64 which is < 128. Since the (b) can be >= 64, we need 2 U64 to store this.
+// Use (lo) if (b) is less than 64, and (hi) if >= 64.
+//
+#define NV_BIT_SET_128(b, lo, hi)              { nvAssert( (b) < 128 ); if ( (b) < 64 ) (lo) |= NVBIT64(b); else (hi) |= NVBIT64( b & 0x3F ); }
+//
+// Clear the bit at pos (b) for U64 which is < 128.
+// Use (lo) if (b) is less than 64, and (hi) if >= 64.
+//
+#define NV_BIT_CLEAR_128(b, lo, hi)            { nvAssert( (b) < 128 ); if ( (b) < 64 ) (lo) &= ~NVBIT64(b); else (hi) &= ~NVBIT64( b & 0x3F ); }
+
+// Get the number of elements the specified fixed-size array
+#define NV_ARRAY_ELEMENTS(x)                   ((sizeof(x)/sizeof((x)[0])))
+
 #ifdef __cplusplus
 }
 #endif //__cplusplus
--- a/kernel-open/common/inc/nvstatus.h
+++ b/kernel-open/common/inc/nvstatus.h
@@ -24,11 +24,6 @@
 #ifndef SDK_NVSTATUS_H
 #define SDK_NVSTATUS_H

-/* XAPIGEN - this file is not suitable for (nor needed by) xapigen.         */
-/*           Rather than #ifdef out every such include in every sdk         */
-/*           file, punt here.                                               */
-#if !defined(XAPIGEN)        /* rest of file */
-
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -125,6 +120,4 @@ const char *nvstatusToString(NV_STATUS nvStatusIn);
 }
 #endif

-#endif // XAPIGEN
-
 #endif /* SDK_NVSTATUS_H */
--- a/kernel-open/common/inc/nvstatuscodes.h
+++ b/kernel-open/common/inc/nvstatuscodes.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2014-2020 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2014-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -24,11 +24,6 @@
 #ifndef SDK_NVSTATUSCODES_H
 #define SDK_NVSTATUSCODES_H

-/* XAPIGEN - this file is not suitable for (nor needed by) xapigen.         */
-/*           Rather than #ifdef out every such include in every sdk         */
-/*           file, punt here.                                               */
-#if !defined(XAPIGEN)        /* rest of file */
-
 NV_STATUS_CODE(NV_OK,                                  0x00000000, "Success")
 NV_STATUS_CODE(NV_ERR_GENERIC,                         0x0000FFFF, "Failure: Generic Error")

@@ -153,6 +148,13 @@ NV_STATUS_CODE(NV_ERR_NVLINK_CLOCK_ERROR,              0x00000076, "Nvlink Clock
 NV_STATUS_CODE(NV_ERR_NVLINK_TRAINING_ERROR,           0x00000077, "Nvlink Training Error")
 NV_STATUS_CODE(NV_ERR_NVLINK_CONFIGURATION_ERROR,      0x00000078, "Nvlink Configuration Error")
 NV_STATUS_CODE(NV_ERR_RISCV_ERROR,                     0x00000079, "Generic RISC-V assert or halt")
+NV_STATUS_CODE(NV_ERR_FABRIC_MANAGER_NOT_PRESENT,      0x0000007A, "Fabric Manager is not loaded")
+NV_STATUS_CODE(NV_ERR_ALREADY_SIGNALLED,               0x0000007B, "Semaphore Surface value already >= requested wait value")
+NV_STATUS_CODE(NV_ERR_QUEUE_TASK_SLOT_NOT_AVAILABLE,   0x0000007C, "PMU RPC error due to no queue slot available for this event")
+NV_STATUS_CODE(NV_ERR_KEY_ROTATION_IN_PROGRESS,        0x0000007D, "Operation not allowed as key rotation is in progress")
+NV_STATUS_CODE(NV_ERR_TEST_ONLY_CODE_NOT_ENABLED,      0x0000007E, "Test-only code path not enabled")
+NV_STATUS_CODE(NV_ERR_SECURE_BOOT_FAILED,              0x0000007F, "GFW secure boot failed")
+NV_STATUS_CODE(NV_ERR_INSUFFICIENT_ZBC_ENTRY,          0x00000080, "No more ZBC entry for the client")

 // Warnings:
 NV_STATUS_CODE(NV_WARN_HOT_SWITCH,                     0x00010001, "WARNING Hot switch")
@@ -164,6 +166,4 @@ NV_STATUS_CODE(NV_WARN_NOTHING_TO_DO,                  0x00010006, "WARNING Noth
 NV_STATUS_CODE(NV_WARN_NULL_OBJECT,                    0x00010007, "WARNING NULL object found")
 NV_STATUS_CODE(NV_WARN_OUT_OF_RANGE,                   0x00010008, "WARNING value out of range")

-#endif // XAPIGEN
-
 #endif /* SDK_NVSTATUSCODES_H */
--- a/kernel-open/common/inc/nvtypes.h
+++ b/kernel-open/common/inc/nvtypes.h
@@ -145,7 +145,18 @@ typedef   signed short     NvS16; /* -32768 to 32767                         */
 #endif

 // Macro to build an NvU32 from four bytes, listed from msb to lsb
-#define NvU32_BUILD(a, b, c, d) (((a) << 24) | ((b) << 16) | ((c) << 8) | (d))
+#define NvU32_BUILD(a, b, c, d) \
+    ((NvU32)( \
+     (((NvU32)(a) & 0xff) << 24) | \
+     (((NvU32)(b) & 0xff) << 16) | \
+     (((NvU32)(c) & 0xff) << 8)  | \
+     (((NvU32)(d) & 0xff))))
+
+// Macro to build an NvU64 from two DWORDS, listed from msb to lsb
+#define NvU64_BUILD(a, b) \
+    ((NvU64)( \
+     (((NvU64)(a) & ~0U) << 32) | \
+     (((NvU64)(b) & ~0U))))

 #if NVTYPES_USE_STDINT
 typedef uint32_t           NvV32; /* "void": enumerated or multiple fields   */
@@ -513,6 +524,12 @@ typedef struct
 // place to re-locate these from nvos.h which cannot be included by a number
 // of builds that need them

+#if defined(__GNUC__) || defined(__clang__) || defined(__INTEL_COMPILER)
+    #define NV_ATTRIBUTE_UNUSED __attribute__((__unused__))
+#else
+    #define NV_ATTRIBUTE_UNUSED
+#endif
+
 #if defined(_MSC_VER)

    #if _MSC_VER >= 1310
@@ -536,8 +553,6 @@ typedef struct

    #define NV_FORCERESULTCHECK

-    #define NV_ATTRIBUTE_UNUSED
-
    #define NV_FORMAT_PRINTF(_f, _a)

 #else // ! defined(_MSC_VER)
@@ -635,12 +650,6 @@ typedef struct
        #define NV_FORCERESULTCHECK
    #endif

-    #if defined(__GNUC__) || defined(__clang__) || defined(__INTEL_COMPILER)
-        #define NV_ATTRIBUTE_UNUSED __attribute__((__unused__))
-    #else
-        #define NV_ATTRIBUTE_UNUSED
-    #endif
-
    /*
     * Functions decorated with NV_FORMAT_PRINTF(f, a) have a format string at
     * parameter number 'f' and variadic arguments start at parameter number 'a'.
--- a/kernel-open/common/inc/os-interface.h
+++ b/kernel-open/common/inc/os-interface.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 1999-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 1999-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -40,8 +40,11 @@
 #include "nv_stdarg.h"
 #include <nv-kernel-interface-api.h>
 #include <os/nv_memory_type.h>
+#include <os/nv_memory_area.h>
 #include <nv-caps.h>

+#include "rs_access.h"
+


 typedef struct
@@ -67,7 +70,6 @@ typedef struct os_wait_queue os_wait_queue;
 * ---------------------------------------------------------------------------
 */

-NvU64       NV_API_CALL  os_get_num_phys_pages       (void);
 NV_STATUS   NV_API_CALL  os_alloc_mem                (void **, NvU64);
 void        NV_API_CALL  os_free_mem                 (void *);
 NV_STATUS   NV_API_CALL  os_get_current_time         (NvU32 *, NvU32 *);
@@ -103,9 +105,10 @@ NvBool      NV_API_CALL  os_pci_remove_supported     (void);
 void        NV_API_CALL  os_pci_remove               (void *);
 void*       NV_API_CALL  os_map_kernel_space         (NvU64, NvU64, NvU32);
 void        NV_API_CALL  os_unmap_kernel_space       (void *, NvU64);
-void*       NV_API_CALL  os_map_user_space           (NvU64, NvU64, NvU32, NvU32, void **);
+#if defined(NV_VMWARE)
+void*       NV_API_CALL  os_map_user_space           (MemoryArea *, NvU32, NvU32, void **);
 void        NV_API_CALL  os_unmap_user_space         (void *, NvU64, void *);
-NV_STATUS   NV_API_CALL  os_flush_cpu_cache          (void);
+#endif
 NV_STATUS   NV_API_CALL  os_flush_cpu_cache_all      (void);
 NV_STATUS   NV_API_CALL  os_flush_user_cache         (void);
 void        NV_API_CALL  os_flush_cpu_write_combine_buffer(void);
@@ -116,7 +119,7 @@ void        NV_API_CALL  os_io_write_byte            (NvU32, NvU8);
 void        NV_API_CALL  os_io_write_word            (NvU32, NvU16);
 void        NV_API_CALL  os_io_write_dword           (NvU32, NvU32);
 NvBool      NV_API_CALL  os_is_administrator         (void);
-NvBool      NV_API_CALL  os_allow_priority_override  (void);
+NvBool      NV_API_CALL  os_check_access             (RsAccessRight accessRight);
 void        NV_API_CALL  os_dbg_init                 (void);
 void        NV_API_CALL  os_dbg_breakpoint           (void);
 void        NV_API_CALL  os_dbg_set_level            (NvU32);
@@ -125,13 +128,15 @@ NvU32       NV_API_CALL  os_get_cpu_number           (void);
 void        NV_API_CALL  os_disable_console_access   (void);
 void        NV_API_CALL  os_enable_console_access    (void);
 NV_STATUS   NV_API_CALL  os_registry_init            (void);
+NvU64       NV_API_CALL  os_get_max_user_va          (void);
 NV_STATUS   NV_API_CALL  os_schedule                 (void);
 NV_STATUS   NV_API_CALL  os_alloc_spinlock           (void **);
 void        NV_API_CALL  os_free_spinlock            (void *);
 NvU64       NV_API_CALL  os_acquire_spinlock         (void *);
 void        NV_API_CALL  os_release_spinlock         (void *, NvU64);
 NV_STATUS   NV_API_CALL  os_queue_work_item          (struct os_work_queue *, void *);
-NV_STATUS   NV_API_CALL  os_flush_work_queue         (struct os_work_queue *);
+NV_STATUS   NV_API_CALL  os_flush_work_queue         (struct os_work_queue *, NvBool);
+NvBool      NV_API_CALL  os_is_queue_flush_ongoing   (struct os_work_queue *);
 NV_STATUS   NV_API_CALL  os_alloc_mutex              (void **);
 void        NV_API_CALL  os_free_mutex               (void *);
 NV_STATUS   NV_API_CALL  os_acquire_mutex            (void *);
@@ -142,8 +147,17 @@ void        NV_API_CALL  os_free_semaphore           (void *);
 NV_STATUS   NV_API_CALL  os_acquire_semaphore        (void *);
 NV_STATUS   NV_API_CALL  os_cond_acquire_semaphore   (void *);
 NV_STATUS   NV_API_CALL  os_release_semaphore        (void *);
+void*       NV_API_CALL  os_alloc_rwlock             (void);
+void        NV_API_CALL  os_free_rwlock              (void *);
+NV_STATUS   NV_API_CALL  os_acquire_rwlock_read      (void *);
+NV_STATUS   NV_API_CALL  os_acquire_rwlock_write     (void *);
+NV_STATUS   NV_API_CALL  os_cond_acquire_rwlock_read (void *);
+NV_STATUS   NV_API_CALL  os_cond_acquire_rwlock_write(void *);
+void        NV_API_CALL  os_release_rwlock_read      (void *);
+void        NV_API_CALL  os_release_rwlock_write     (void *);
 NvBool      NV_API_CALL  os_semaphore_may_sleep      (void);
 NV_STATUS   NV_API_CALL  os_get_version_info         (os_version_info*);
+NV_STATUS   NV_API_CALL  os_get_is_openrm            (NvBool *);
 NvBool      NV_API_CALL  os_is_isr                   (void);
 NvBool      NV_API_CALL  os_pat_supported            (void);
 void        NV_API_CALL  os_dump_stack               (void);
@@ -153,10 +167,9 @@ NvBool      NV_API_CALL  os_is_vgx_hyper             (void);
 NV_STATUS   NV_API_CALL  os_inject_vgx_msi           (NvU16, NvU64, NvU32);
 NvBool      NV_API_CALL  os_is_grid_supported        (void);
 NvU32       NV_API_CALL  os_get_grid_csp_support     (void);
-void        NV_API_CALL  os_get_screen_info          (NvU64 *, NvU16 *, NvU16 *, NvU16 *, NvU16 *, NvU64, NvU64);
 void        NV_API_CALL  os_bug_check                (NvU32, const char *);
 NV_STATUS   NV_API_CALL  os_lock_user_pages          (void *, NvU64, void **, NvU32);
-NV_STATUS   NV_API_CALL  os_lookup_user_io_memory    (void *, NvU64, NvU64 **, void**);
+NV_STATUS   NV_API_CALL  os_lookup_user_io_memory    (void *, NvU64, NvU64 **);
 NV_STATUS   NV_API_CALL  os_unlock_user_pages        (NvU64, void *);
 NV_STATUS   NV_API_CALL  os_match_mmap_offset        (void *, NvU64, NvU64 *);
 NV_STATUS   NV_API_CALL  os_get_euid                 (NvU32 *);
@@ -172,7 +185,6 @@ NV_STATUS   NV_API_CALL  os_put_page                 (NvU64 address);
 NvU32       NV_API_CALL  os_get_page_refcount        (NvU64 address);
 NvU32       NV_API_CALL  os_count_tail_pages         (NvU64 address);
 void        NV_API_CALL  os_free_pages_phys          (NvU64, NvU32);
-NV_STATUS   NV_API_CALL  os_call_nv_vmbus            (NvU32, void *);
 NV_STATUS   NV_API_CALL  os_open_temporary_file      (void **);
 void        NV_API_CALL  os_close_file               (void *);
 NV_STATUS   NV_API_CALL  os_write_file               (void *, NvU8 *, NvU64, NvU64);
@@ -180,7 +192,7 @@ NV_STATUS   NV_API_CALL  os_read_file                (void *, NvU8 *, NvU64, NvU
 NV_STATUS   NV_API_CALL  os_open_readonly_file       (const char *, void **);
 NV_STATUS   NV_API_CALL  os_open_and_read_file       (const char *, NvU8 *, NvU64);
 NvBool      NV_API_CALL  os_is_nvswitch_present      (void);
-void        NV_API_CALL  os_get_random_bytes         (NvU8 *, NvU16);
+NV_STATUS   NV_API_CALL  os_get_random_bytes         (NvU8 *, NvU16);
 NV_STATUS   NV_API_CALL  os_alloc_wait_queue         (os_wait_queue **);
 void        NV_API_CALL  os_free_wait_queue          (os_wait_queue *);
 void        NV_API_CALL  os_wait_uninterruptible     (os_wait_queue *);
@@ -192,27 +204,32 @@ nv_cap_t*   NV_API_CALL  os_nv_cap_create_file_entry  (nv_cap_t *, const char *,
 void        NV_API_CALL  os_nv_cap_destroy_entry      (nv_cap_t *);
 int         NV_API_CALL  os_nv_cap_validate_and_dup_fd(const nv_cap_t *, int);
 void        NV_API_CALL  os_nv_cap_close_fd           (int);
+NvS32       NV_API_CALL  os_imex_channel_get          (NvU64);
+NvS32       NV_API_CALL  os_imex_channel_count        (void);

-
-
-
-
-
-
-
-
-
-
-
-
-
+enum os_pci_req_atomics_type {
+    OS_INTF_PCIE_REQ_ATOMICS_32BIT,
+    OS_INTF_PCIE_REQ_ATOMICS_64BIT,
+    OS_INTF_PCIE_REQ_ATOMICS_128BIT
+};
+NV_STATUS   NV_API_CALL  os_enable_pci_req_atomics   (void *, enum os_pci_req_atomics_type);
+NV_STATUS   NV_API_CALL  os_get_numa_node_memory_usage (NvS32, NvU64 *, NvU64 *);
+NV_STATUS   NV_API_CALL  os_numa_add_gpu_memory      (void *, NvU64, NvU64, NvU32 *);
+NV_STATUS   NV_API_CALL  os_numa_remove_gpu_memory   (void *, NvU64, NvU64, NvU32); 
+NV_STATUS   NV_API_CALL  os_offline_page_at_address(NvU64 address);
+void*       NV_API_CALL  os_get_pid_info(void);
+void        NV_API_CALL  os_put_pid_info(void *pid_info);
+NV_STATUS   NV_API_CALL  os_find_ns_pid(void *pid_info, NvU32 *ns_pid);

 extern NvU32 os_page_size;
 extern NvU64 os_page_mask;
 extern NvU8  os_page_shift;
-extern NvU32 os_sev_status;
-extern NvBool os_sev_enabled;
+extern NvBool os_cc_enabled;
+extern NvBool os_cc_sev_snp_enabled;
+extern NvBool os_cc_snp_vtom_enabled;
+extern NvBool os_cc_tdx_enabled;
 extern NvBool os_dma_buf_enabled;
+extern NvBool os_imex_channel_is_supported;

 /*
 * ---------------------------------------------------------------------------
@@ -245,11 +262,4 @@ int  NV_API_CALL  nv_printf(NvU32 debuglevel, const char *printf_format, ...);
 #define NV_LOCK_USER_PAGES_FLAGS_WRITE_NO                  0x00000000
 #define NV_LOCK_USER_PAGES_FLAGS_WRITE_YES                 0x00000001

-
-
-
-
-
-
-
 #endif /* OS_INTERFACE_H */
--- a/kernel-open/common/inc/os/nv_memory_area.h
+++ b/kernel-open/common/inc/os/nv_memory_area.h
@@ -0,0 +1,104 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef NV_MEMORY_AREA_H
+#define NV_MEMORY_AREA_H
+
+typedef struct MemoryRange
+{
+    NvU64 start;
+    NvU64 size;
+} MemoryRange;
+
+typedef struct MemoryArea
+{
+    MemoryRange *pRanges;
+    NvU64 numRanges;
+} MemoryArea;
+
+static inline NvU64 memareaSize(MemoryArea memArea)
+{
+    NvU64 size = 0;
+    NvU64 idx = 0;
+    for (idx = 0; idx < memArea.numRanges; idx++)
+    {
+        size += memArea.pRanges[idx].size;
+    }
+    return size;
+}
+
+static inline MemoryRange
+mrangeMake
+(
+    NvU64 start,
+    NvU64 size
+)
+{
+    MemoryRange range;
+    range.start = start;
+    range.size = size;
+    return range;
+}
+
+static inline NvU64
+mrangeLimit
+(
+    MemoryRange a
+)
+{
+    return a.start + a.size;
+}
+
+static inline NvBool
+mrangeIntersects
+(
+    MemoryRange a,
+    MemoryRange b
+)
+{
+    return ((a.start >= b.start) && (a.start < mrangeLimit(b))) ||
+        ((b.start >= a.start) && (b.start < mrangeLimit(a)));
+}
+
+static inline NvBool
+mrangeContains
+(
+    MemoryRange outer,
+    MemoryRange inner
+)
+{
+    return (inner.start >= outer.start) && (mrangeLimit(inner) <= mrangeLimit(outer));
+}
+
+static inline MemoryRange
+mrangeOffset
+(
+    MemoryRange range,
+    NvU64 amt
+)
+{
+    range.start += amt;
+    return range;
+}
+
+#endif /* NV_MEMORY_AREA_H */
--- a/kernel-open/common/inc/rm-gpu-ops.h
+++ b/kernel-open/common/inc/rm-gpu-ops.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 1999-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 1999-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -37,7 +37,7 @@ NV_STATUS  NV_API_CALL  rm_gpu_ops_create_session (nvidia_stack_t *, nvgpuSessio
 NV_STATUS  NV_API_CALL  rm_gpu_ops_destroy_session (nvidia_stack_t *, nvgpuSessionHandle_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_device_create (nvidia_stack_t *, nvgpuSessionHandle_t, const nvgpuInfo_t *, const NvProcessorUuid *, nvgpuDeviceHandle_t *, NvBool);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_device_destroy (nvidia_stack_t *, nvgpuDeviceHandle_t);
-NV_STATUS  NV_API_CALL  rm_gpu_ops_address_space_create(nvidia_stack_t *, nvgpuDeviceHandle_t, unsigned long long, unsigned long long, nvgpuAddressSpaceHandle_t *, nvgpuAddressSpaceInfo_t);
+NV_STATUS  NV_API_CALL  rm_gpu_ops_address_space_create(nvidia_stack_t *, nvgpuDeviceHandle_t, unsigned long long, unsigned long long, NvBool, nvgpuAddressSpaceHandle_t *, nvgpuAddressSpaceInfo_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_dup_address_space(nvidia_stack_t *, nvgpuDeviceHandle_t, NvHandle, NvHandle, nvgpuAddressSpaceHandle_t *, nvgpuAddressSpaceInfo_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_address_space_destroy(nvidia_stack_t *, nvgpuAddressSpaceHandle_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_memory_alloc_fb(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvLength, NvU64 *, nvgpuAllocInfo_t);
@@ -45,7 +45,6 @@ NV_STATUS  NV_API_CALL  rm_gpu_ops_memory_alloc_fb(nvidia_stack_t *, nvgpuAddres
 NV_STATUS  NV_API_CALL  rm_gpu_ops_pma_alloc_pages(nvidia_stack_t *, void *, NvLength, NvU32 , nvgpuPmaAllocationOptions_t, NvU64 *);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_pma_free_pages(nvidia_stack_t *, void *, NvU64 *, NvLength , NvU32, NvU32);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_pma_pin_pages(nvidia_stack_t *, void *, NvU64 *, NvLength , NvU32, NvU32);
-NV_STATUS  NV_API_CALL  rm_gpu_ops_pma_unpin_pages(nvidia_stack_t *, void *, NvU64 *, NvLength , NvU32);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_get_pma_object(nvidia_stack_t *, nvgpuDeviceHandle_t, void **, const nvgpuPmaStatistics_t *);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_pma_register_callbacks(nvidia_stack_t *sp, void *, nvPmaEvictPagesCallback, nvPmaEvictRangeCallback, void *);
 void       NV_API_CALL  rm_gpu_ops_pma_unregister_callbacks(nvidia_stack_t *sp, void *);
@@ -56,14 +55,16 @@ NV_STATUS  NV_API_CALL  rm_gpu_ops_get_p2p_caps(nvidia_stack_t *, nvgpuDeviceHan

 NV_STATUS  NV_API_CALL  rm_gpu_ops_memory_cpu_map(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, NvLength, void **, NvU32);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_memory_cpu_ummap(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, void*);
-NV_STATUS  NV_API_CALL  rm_gpu_ops_channel_allocate(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, const nvgpuChannelAllocParams_t *, nvgpuChannelHandle_t *, nvgpuChannelInfo_t);
+NV_STATUS  NV_API_CALL  rm_gpu_ops_tsg_allocate(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, const nvgpuTsgAllocParams_t *, nvgpuTsgHandle_t *);
+NV_STATUS  NV_API_CALL  rm_gpu_ops_tsg_destroy(nvidia_stack_t *, nvgpuTsgHandle_t);
+NV_STATUS  NV_API_CALL  rm_gpu_ops_channel_allocate(nvidia_stack_t *, const nvgpuTsgHandle_t, const nvgpuChannelAllocParams_t *, nvgpuChannelHandle_t *, nvgpuChannelInfo_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_channel_destroy(nvidia_stack_t *, nvgpuChannelHandle_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_memory_free(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64);
 NV_STATUS  NV_API_CALL rm_gpu_ops_query_caps(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuCaps_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_query_ces_caps(nvidia_stack_t *sp, nvgpuDeviceHandle_t, nvgpuCesCaps_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_get_gpu_info(nvidia_stack_t *, const NvProcessorUuid *pUuid, const nvgpuClientInfo_t *, nvgpuInfo_t *);
 NV_STATUS  NV_API_CALL rm_gpu_ops_service_device_interrupts_rm(nvidia_stack_t *, nvgpuDeviceHandle_t);
-NV_STATUS  NV_API_CALL rm_gpu_ops_dup_allocation(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, nvgpuAddressSpaceHandle_t, NvU64 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_dup_allocation(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, nvgpuAddressSpaceHandle_t, NvU64, NvU64 *);

 NV_STATUS  NV_API_CALL  rm_gpu_ops_dup_memory (nvidia_stack_t *, nvgpuDeviceHandle_t, NvHandle, NvHandle, NvHandle *, nvgpuMemoryInfo_t);

@@ -74,17 +75,21 @@ NV_STATUS NV_API_CALL rm_gpu_ops_own_page_fault_intr(nvidia_stack_t *, nvgpuDevi
 NV_STATUS  NV_API_CALL rm_gpu_ops_init_fault_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuFaultInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_destroy_fault_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuFaultInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_get_non_replayable_faults(nvidia_stack_t *, nvgpuFaultInfo_t, void *, NvU32 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_flush_replayable_fault_buffer(nvidia_stack_t *, nvgpuFaultInfo_t, NvBool);
+NV_STATUS  NV_API_CALL rm_gpu_ops_toggle_prefetch_faults(nvidia_stack_t *, nvgpuFaultInfo_t, NvBool);
 NV_STATUS  NV_API_CALL rm_gpu_ops_has_pending_non_replayable_faults(nvidia_stack_t *, nvgpuFaultInfo_t, NvBool *);
-NV_STATUS  NV_API_CALL rm_gpu_ops_init_access_cntr_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuAccessCntrInfo_t);
+NV_STATUS  NV_API_CALL rm_gpu_ops_init_access_cntr_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuAccessCntrInfo_t, NvU32);
 NV_STATUS  NV_API_CALL rm_gpu_ops_destroy_access_cntr_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuAccessCntrInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_own_access_cntr_intr(nvidia_stack_t *, nvgpuSessionHandle_t, nvgpuAccessCntrInfo_t, NvBool);
 NV_STATUS  NV_API_CALL rm_gpu_ops_enable_access_cntr(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuAccessCntrInfo_t, nvgpuAccessCntrConfig_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_disable_access_cntr(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuAccessCntrInfo_t);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_set_page_directory (nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, unsigned, NvBool, NvU32);
 NV_STATUS  NV_API_CALL  rm_gpu_ops_unset_page_directory (nvidia_stack_t *, nvgpuAddressSpaceHandle_t);
+NV_STATUS  NV_API_CALL rm_gpu_ops_get_nvlink_info(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuNvlinkInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_p2p_object_create(nvidia_stack_t *, nvgpuDeviceHandle_t, nvgpuDeviceHandle_t, NvHandle *);
 void       NV_API_CALL rm_gpu_ops_p2p_object_destroy(nvidia_stack_t *, nvgpuSessionHandle_t, NvHandle);
 NV_STATUS  NV_API_CALL rm_gpu_ops_get_external_alloc_ptes(nvidia_stack_t*, nvgpuAddressSpaceHandle_t, NvHandle, NvU64, NvU64, nvgpuExternalMappingInfo_t);
+NV_STATUS  NV_API_CALL rm_gpu_ops_get_external_alloc_phys_addrs(nvidia_stack_t*, nvgpuAddressSpaceHandle_t, NvHandle, NvU64, NvU64, nvgpuExternalPhysAddrInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_retain_channel(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvHandle, NvHandle, void **, nvgpuChannelInstanceInfo_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_bind_channel_resources(nvidia_stack_t *, void *, nvgpuChannelResourceBindParams_t);
 void       NV_API_CALL rm_gpu_ops_release_channel(nvidia_stack_t *, void *);
@@ -97,14 +102,18 @@ void       NV_API_CALL rm_gpu_ops_paging_channel_destroy(nvidia_stack_t *, nvgpu
 NV_STATUS  NV_API_CALL rm_gpu_ops_paging_channels_map(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, nvgpuDeviceHandle_t, NvU64 *);
 void       NV_API_CALL rm_gpu_ops_paging_channels_unmap(nvidia_stack_t *, nvgpuAddressSpaceHandle_t, NvU64, nvgpuDeviceHandle_t);
 NV_STATUS  NV_API_CALL rm_gpu_ops_paging_channel_push_stream(nvidia_stack_t *, nvgpuPagingChannelHandle_t, char *, NvU32);
+void       NV_API_CALL rm_gpu_ops_report_fatal_error(nvidia_stack_t *, NV_STATUS error);

-
-
-
-
-
-
-
-
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_context_init(nvidia_stack_t *, struct ccslContext_t **, nvgpuChannelHandle_t);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_context_clear(nvidia_stack_t *, struct ccslContext_t *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_rotate_key(nvidia_stack_t *, UvmCslContext *[], NvU32);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_rotate_iv(nvidia_stack_t *, struct ccslContext_t *, NvU8);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_encrypt(nvidia_stack_t *, struct ccslContext_t *, NvU32, NvU8 const *, NvU8 *, NvU8 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_encrypt_with_iv(nvidia_stack_t *, struct ccslContext_t *, NvU32, NvU8 const *, NvU8*, NvU8 *, NvU8 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_decrypt(nvidia_stack_t *, struct ccslContext_t *, NvU32, NvU8 const *, NvU8 const *, NvU32, NvU8 *, NvU8 const *, NvU32, NvU8 const *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_sign(nvidia_stack_t *, struct ccslContext_t *, NvU32, NvU8 const *, NvU8 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_query_message_pool(nvidia_stack_t *, struct ccslContext_t *, NvU8, NvU64 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_increment_iv(nvidia_stack_t *, struct ccslContext_t *, NvU8, NvU64, NvU8 *);
+NV_STATUS  NV_API_CALL rm_gpu_ops_ccsl_log_encryption(nvidia_stack_t *, struct ccslContext_t *, NvU8, NvU32);

 #endif
--- a/kernel-open/common/inc/rs_access.h
+++ b/kernel-open/common/inc/rs_access.h
@@ -0,0 +1,276 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2019-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+#pragma once
+
+#include <nvtypes.h>
+#if defined(_MSC_VER)
+#pragma warning(disable:4324)
+#endif
+
+//
+// This file was generated with FINN, an NVIDIA coding tool.
+// Source file:      rs_access.finn
+//
+
+
+
+
+#include "nvtypes.h"
+#include "nvmisc.h"
+
+
+/****************************************************************************/
+/*                       Access right definitions                           */
+/****************************************************************************/
+
+//
+// The meaning of each access right is documented in
+//   resman/docs/rmapi/resource_server/rm_capabilities.adoc
+//
+// RS_ACCESS_COUNT is the number of access rights that have been defined
+// and are in use. All integers in the range [0, RS_ACCESS_COUNT) should
+// represent valid access rights.
+//
+// When adding a new access right, don't forget to update
+//   1) The descriptions in the resman/docs/rmapi/resource_server/rm_capabilities.adoc
+//   2) RS_ACCESS_COUNT, defined below
+//   3) The declaration of g_rsAccessMetadata in rs_access_rights.c
+//   4) The list of access rights in drivers/common/chip-config/Chipcontrols.pm
+//   5) Any relevant access right callbacks
+//
+
+#define RS_ACCESS_DUP_OBJECT 0U
+#define RS_ACCESS_NICE       1U
+#define RS_ACCESS_DEBUG      2U
+#define RS_ACCESS_PERFMON    3U
+#define RS_ACCESS_COUNT      4U
+
+
+/****************************************************************************/
+/*                     Access right data structures                         */
+/****************************************************************************/
+
+/*!
+ * @brief A type that can be used to represent any access right.
+ */
+typedef NvU16 RsAccessRight;
+
+/*!
+ * @brief An internal type used to represent one limb in an access right mask.
+ */
+typedef NvU32 RsAccessLimb;
+#define SDK_RS_ACCESS_LIMB_BITS 32
+
+/*!
+ * @brief The number of limbs in the RS_ACCESS_MASK struct.
+ */
+#define SDK_RS_ACCESS_MAX_LIMBS 1
+
+/*!
+ * @brief The maximum number of possible access rights supported by the
+ *        current data structure definition.
+ *
+ * You probably want RS_ACCESS_COUNT instead, which is the number of actual
+ * access rights defined.
+ */
+#define SDK_RS_ACCESS_MAX_COUNT (0x20) /* finn: Evaluated from "(SDK_RS_ACCESS_LIMB_BITS * SDK_RS_ACCESS_MAX_LIMBS)" */
+
+/**
+ * @brief A struct representing a set of access rights.
+ *
+ * Note that the values of bit positions larger than RS_ACCESS_COUNT is
+ * undefined, and should not be assumed to be 0 (see RS_ACCESS_MASK_FILL).
+ */
+typedef struct RS_ACCESS_MASK {
+    RsAccessLimb limbs[SDK_RS_ACCESS_MAX_LIMBS];
+} RS_ACCESS_MASK;
+
+/**
+ * @brief A struct representing auxiliary information about each access right.
+ */
+typedef struct RS_ACCESS_INFO {
+    NvU32 flags;
+} RS_ACCESS_INFO;
+
+
+/****************************************************************************/
+/*                           Access right macros                            */
+/****************************************************************************/
+
+#define SDK_RS_ACCESS_LIMB_INDEX(index) ((index) / SDK_RS_ACCESS_LIMB_BITS)
+#define SDK_RS_ACCESS_LIMB_POS(index)   ((index) % SDK_RS_ACCESS_LIMB_BITS)
+
+#define SDK_RS_ACCESS_LIMB_ELT(pAccessMask, index) \
+    ((pAccessMask)->limbs[SDK_RS_ACCESS_LIMB_INDEX(index)])
+#define SDK_RS_ACCESS_OFFSET_MASK(index) \
+    NVBIT_TYPE(SDK_RS_ACCESS_LIMB_POS(index), RsAccessLimb)
+
+/*!
+ * @brief Checks that accessRight represents a valid access right.
+ *
+ * The valid range of access rights is [0, RS_ACCESS_COUNT).
+ *
+ * @param[in] accessRight The access right value to check
+ *
+ * @return true if accessRight is valid
+ * @return false otherwise
+ */
+#define RS_ACCESS_BOUNDS_CHECK(accessRight) \
+    (accessRight < RS_ACCESS_COUNT)
+
+/*!
+ * @brief Test whether an access right is present in a set
+ *
+ * @param[in] pAccessMask The set of access rights to read
+ * @param[in] index The access right to examine
+ *
+ * @return NV_TRUE if the access right specified by index was present in the set,
+ *         and NV_FALSE otherwise
+ */
+#define RS_ACCESS_MASK_TEST(pAccessMask, index) \
+    (RS_ACCESS_BOUNDS_CHECK(index) && \
+        (SDK_RS_ACCESS_LIMB_ELT(pAccessMask, index) & SDK_RS_ACCESS_OFFSET_MASK(index)) != 0)
+
+/*!
+ * @brief Add an access right to a mask
+ *
+ * @param[in] pAccessMask The set of access rights to modify
+ * @param[in] index The access right to set
+ */
+#define RS_ACCESS_MASK_ADD(pAccessMask, index) \
+    do \
+    { \
+        if (RS_ACCESS_BOUNDS_CHECK(index)) { \
+            SDK_RS_ACCESS_LIMB_ELT(pAccessMask, index) |= SDK_RS_ACCESS_OFFSET_MASK(index); \
+        } \
+    } while (NV_FALSE)
+
+/*!
+ * @brief Remove an access right from a mask
+ *
+ * @param[in] pAccessMask The set of access rights to modify
+ * @param[in] index The access right to unset
+ */
+#define RS_ACCESS_MASK_REMOVE(pAccessMask, index) \
+    do \
+    { \
+        if (RS_ACCESS_BOUNDS_CHECK(index)) { \
+            SDK_RS_ACCESS_LIMB_ELT(pAccessMask, index) &= ~SDK_RS_ACCESS_OFFSET_MASK(index); \
+        } \
+    } while (NV_FALSE)
+
+/*!
+ * @brief Performs an in-place union between two access right masks
+ *
+ * @param[in,out] pMaskOut The access rights mask to be updated
+ * @param[in] pMaskIn The set of access rights to be added to pMaskOut
+ */
+#define RS_ACCESS_MASK_UNION(pMaskOut, pMaskIn) \
+    do \
+    { \
+        NvLength limb; \
+        for (limb = 0; limb < SDK_RS_ACCESS_MAX_LIMBS; limb++) \
+        { \
+            SDK_RS_ACCESS_LIMB_ELT(pMaskOut, limb) |= SDK_RS_ACCESS_LIMB_ELT(pMaskIn, limb); \
+        } \
+    } while (NV_FALSE)
+
+/*!
+ * @brief Performs an in-place subtract of one mask's rights from another
+ *
+ * @param[in,out] pMaskOut The access rights mask to be updated
+ * @param[in] pMaskIn The set of access rights to be removed from pMaskOut
+ */
+#define RS_ACCESS_MASK_SUBTRACT(pMaskOut, pMaskIn) \
+    do \
+    { \
+        NvLength limb; \
+        for (limb = 0; limb < SDK_RS_ACCESS_MAX_LIMBS; limb++) \
+        { \
+            SDK_RS_ACCESS_LIMB_ELT(pMaskOut, limb) &= ~SDK_RS_ACCESS_LIMB_ELT(pMaskIn, limb); \
+        } \
+    } while (NV_FALSE)
+
+/*!
+ * @brief Removes all rights from an access rights mask
+ *
+ * @param[in,out] pAccessMask The access rights mask to be updated
+ */
+#define RS_ACCESS_MASK_CLEAR(pAccessMask) \
+    do \
+    { \
+        portMemSet(pAccessMask, 0, sizeof(*pAccessMask)); \
+    } while (NV_FALSE)
+
+/*!
+ * @brief Adds all rights to an access rights mask
+ *
+ * @param[in,out] pAccessMask The access rights mask to be updated
+ */
+#define RS_ACCESS_MASK_FILL(pAccessMask) \
+    do \
+    { \
+        portMemSet(pAccessMask, 0xff, sizeof(*pAccessMask)); \
+    } while (NV_FALSE)
+
+
+/****************************************************************************/
+/*                           Share definitions                              */
+/****************************************************************************/
+
+//
+// The usage of Share Policy and the meaning of each share type is documented in
+//   resman/docs/rmapi/resource_server/rm_capabilities.adoc
+//
+#define RS_SHARE_TYPE_NONE              (0U)
+#define RS_SHARE_TYPE_ALL               (1U)
+#define RS_SHARE_TYPE_OS_SECURITY_TOKEN (2U)
+#define RS_SHARE_TYPE_CLIENT            (3U)
+#define RS_SHARE_TYPE_PID               (4U)
+#define RS_SHARE_TYPE_SMC_PARTITION     (5U)
+#define RS_SHARE_TYPE_GPU               (6U)
+#define RS_SHARE_TYPE_FM_CLIENT         (7U)
+// Must be last. Update when a new SHARE_TYPE is added
+#define RS_SHARE_TYPE_MAX               (8U)
+
+
+//
+// Use Revoke to remove an existing policy from the list.
+// Allow is based on OR logic, Require is based on AND logic.
+// To share a right, at least one Allow (non-Require) must match, and all Require must pass.
+// If Compose is specified, policies will be added to the list. Otherwise, they will replace the list.
+//
+#define RS_SHARE_ACTION_FLAG_REVOKE      NVBIT(0)
+#define RS_SHARE_ACTION_FLAG_REQUIRE     NVBIT(1)
+#define RS_SHARE_ACTION_FLAG_COMPOSE     NVBIT(2)
+
+/****************************************************************************/
+/*                       Share flag data structures                         */
+/****************************************************************************/
+
+typedef struct RS_SHARE_POLICY {
+    NvU32          target;
+    RS_ACCESS_MASK accessMask;
+    NvU16          type;                         ///< RS_SHARE_TYPE_
+    NvU8           action;                        ///< RS_SHARE_ACTION_
+} RS_SHARE_POLICY;
--- a/kernel-open/conftest.sh
+++ b/kernel-open/conftest.sh
--- a/kernel-open/count-lines.mk
+++ b/kernel-open/count-lines.mk
@@ -0,0 +1,25 @@
+count:
+	@echo "conftests:$(words $(ALL_CONFTESTS))" \
+		  "objects:$(words $(NV_OBJECTS_DEPEND_ON_CONFTEST))" \
+		  "modules:$(words $(NV_KERNEL_MODULES))"
+
+.PHONY: count
+
+# Include the top-level makefile to get $(NV_KERNEL_MODULES)
+include Makefile
+
+# Set $(src) for the to-be-included nvidia*.Kbuild files
+src := $(CURDIR)
+
+# Include nvidia*.Kbuild and append the nvidia*-y objects to ALL_OBJECTS
+$(foreach _module, $(NV_KERNEL_MODULES),          \
+     $(eval include $(_module)/$(_module).Kbuild) \
+ )
+
+# Concatenate all of the conftest lists; use $(sort ) to remove duplicates
+ALL_CONFTESTS := $(sort $(NV_CONFTEST_FUNCTION_COMPILE_TESTS) \
+                        $(NV_CONFTEST_GENERIC_COMPILE_TESTS)  \
+                        $(NV_CONFTEST_MACRO_COMPILE_TESTS)    \
+                        $(NV_CONFTEST_SYMBOL_COMPILE_TESTS)   \
+                        $(NV_CONFTEST_TYPE_COMPILE_TESTS)     \
+                  )
--- a/kernel-open/header-presence-tests.mk
+++ b/kernel-open/header-presence-tests.mk
@@ -0,0 +1,106 @@
+# Each of these headers is checked for presence with a test #include; a
+# corresponding #define will be generated in conftest/headers.h.
+NV_HEADER_PRESENCE_TESTS = \
+  asm/system.h \
+  drm/drmP.h \
+  drm/drm_aperture.h \
+  drm/drm_auth.h \
+  drm/drm_gem.h \
+  drm/drm_crtc.h \
+  drm/drm_color_mgmt.h \
+  drm/drm_atomic.h \
+  drm/drm_atomic_helper.h \
+  drm/drm_atomic_state_helper.h \
+  drm/drm_encoder.h \
+  drm/drm_atomic_uapi.h \
+  drm/drm_drv.h \
+  drm/drm_fbdev_generic.h \
+  drm/drm_fbdev_ttm.h \
+  drm/drm_framebuffer.h \
+  drm/drm_connector.h \
+  drm/drm_probe_helper.h \
+  drm/drm_blend.h \
+  drm/drm_fourcc.h \
+  drm/drm_prime.h \
+  drm/drm_plane.h \
+  drm/drm_vblank.h \
+  drm/drm_file.h \
+  drm/drm_ioctl.h \
+  drm/drm_device.h \
+  drm/drm_mode_config.h \
+  drm/drm_modeset_lock.h \
+  drm/drm_property.h \
+  dt-bindings/interconnect/tegra_icc_id.h \
+  generated/autoconf.h \
+  generated/compile.h \
+  generated/utsrelease.h \
+  linux/efi.h \
+  linux/kconfig.h \
+  linux/platform/tegra/mc_utils.h \
+  linux/printk.h \
+  linux/ratelimit.h \
+  linux/prio_tree.h \
+  linux/log2.h \
+  linux/of.h \
+  linux/bug.h \
+  linux/sched.h \
+  linux/sched/mm.h \
+  linux/sched/signal.h \
+  linux/sched/task.h \
+  linux/sched/task_stack.h \
+  xen/ioemu.h \
+  linux/fence.h \
+  linux/dma-fence.h \
+  linux/dma-resv.h \
+  soc/tegra/chip-id.h \
+  soc/tegra/fuse.h \
+  soc/tegra/fuse-helper.h \
+  soc/tegra/tegra_bpmp.h \
+  video/nv_internal.h \
+  linux/platform/tegra/dce/dce-client-ipc.h \
+  linux/nvhost.h \
+  linux/nvhost_t194.h \
+  linux/host1x-next.h \
+  asm/book3s/64/hash-64k.h \
+  asm/set_memory.h \
+  asm/prom.h \
+  asm/powernv.h \
+  linux/atomic.h \
+  asm/barrier.h \
+  asm/opal-api.h \
+  sound/hdaudio.h \
+  asm/pgtable_types.h \
+  asm/page.h \
+  linux/stringhash.h \
+  linux/dma-map-ops.h \
+  rdma/peer_mem.h \
+  sound/hda_codec.h \
+  linux/dma-buf.h \
+  linux/time.h \
+  linux/platform_device.h \
+  linux/mutex.h \
+  linux/reset.h \
+  linux/of_platform.h \
+  linux/of_device.h \
+  linux/of_gpio.h \
+  linux/gpio.h \
+  linux/gpio/consumer.h \
+  linux/interconnect.h \
+  linux/pm_runtime.h \
+  linux/clk.h \
+  linux/clk-provider.h \
+  linux/ioasid.h \
+  linux/stdarg.h \
+  linux/iosys-map.h \
+  asm/coco.h \
+  linux/vfio_pci_core.h \
+  linux/mdev.h \
+  soc/tegra/bpmp-abi.h \
+  soc/tegra/bpmp.h \
+  linux/sync_file.h \
+  linux/cc_platform.h \
+  asm/cpufeature.h \
+  linux/mpi.h \
+  asm/mshyperv.h \
+  linux/pfn_t.h
+
--- a/kernel-open/nvidia-drm/nv-kthread-q.c
+++ b/kernel-open/nvidia-drm/nv-kthread-q.c
@@ -0,0 +1,334 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2016-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#include "nv-kthread-q.h"
+#include "nv-list-helpers.h"
+
+#include <linux/kthread.h>
+#include <linux/interrupt.h>
+#include <linux/completion.h>
+#include <linux/module.h>
+#include <linux/mm.h>
+
+#if defined(NV_LINUX_BUG_H_PRESENT)
+    #include <linux/bug.h>
+#else
+    #include <asm/bug.h>
+#endif
+
+// Today's implementation is a little simpler and more limited than the
+// API description allows for in nv-kthread-q.h. Details include:
+//
+// 1. Each nv_kthread_q instance is a first-in, first-out queue.
+//
+// 2. Each nv_kthread_q instance is serviced by exactly one kthread.
+//
+// You can create any number of queues, each of which gets its own
+// named kernel thread (kthread). You can then insert arbitrary functions
+// into the queue, and those functions will be run in the context of the
+// queue's kthread.
+
+#ifndef WARN
+    // Only *really* old kernels (2.6.9) end up here. Just use a simple printk
+    // to implement this, because such kernels won't be supported much longer.
+    #define WARN(condition, format...) ({                    \
+        int __ret_warn_on = !!(condition);                   \
+        if (unlikely(__ret_warn_on))                         \
+            printk(KERN_ERR format);                         \
+        unlikely(__ret_warn_on);                             \
+    })
+#endif
+
+#define NVQ_WARN(fmt, ...)                                   \
+    do {                                                     \
+        if (in_interrupt()) {                                \
+            WARN(1, "nv_kthread_q: [in interrupt]: " fmt,    \
+            ##__VA_ARGS__);                                  \
+        }                                                    \
+        else {                                               \
+            WARN(1, "nv_kthread_q: task: %s: " fmt,          \
+                 current->comm,                              \
+                 ##__VA_ARGS__);                             \
+        }                                                    \
+    } while (0)
+
+static int _main_loop(void *args)
+{
+    nv_kthread_q_t *q = (nv_kthread_q_t *)args;
+    nv_kthread_q_item_t *q_item = NULL;
+    unsigned long flags;
+
+    while (1) {
+        // Normally this thread is never interrupted. However,
+        // down_interruptible (instead of down) is called here,
+        // in order to avoid being classified as a potentially
+        // hung task, by the kernel watchdog.
+        while (down_interruptible(&q->q_sem))
+            NVQ_WARN("Interrupted during semaphore wait\n");
+
+        if (atomic_read(&q->main_loop_should_exit))
+            break;
+
+        spin_lock_irqsave(&q->q_lock, flags);
+
+        // The q_sem semaphore prevents us from getting here unless there is
+        // at least one item in the list, so an empty list indicates a bug.
+        if (unlikely(list_empty(&q->q_list_head))) {
+            spin_unlock_irqrestore(&q->q_lock, flags);
+            NVQ_WARN("_main_loop: Empty queue: q: 0x%p\n", q);
+            continue;
+        }
+
+        // Consume one item from the queue
+        q_item = list_first_entry(&q->q_list_head,
+                                   nv_kthread_q_item_t,
+                                   q_list_node);
+
+        list_del_init(&q_item->q_list_node);
+
+        spin_unlock_irqrestore(&q->q_lock, flags);
+
+        // Run the item
+        q_item->function_to_run(q_item->function_args);
+
+        // Make debugging a little simpler by clearing this between runs:
+        q_item = NULL;
+    }
+
+    while (!kthread_should_stop())
+        schedule();
+
+    return 0;
+}
+
+void nv_kthread_q_stop(nv_kthread_q_t *q)
+{
+    // check if queue has been properly initialized
+    if (unlikely(!q->q_kthread))
+        return;
+
+    nv_kthread_q_flush(q);
+
+    // If this assertion fires, then a caller likely either broke the API rules,
+    // by adding items after calling nv_kthread_q_stop, or possibly messed up
+    // with inadequate flushing of self-rescheduling q_items.
+    if (unlikely(!list_empty(&q->q_list_head)))
+        NVQ_WARN("list not empty after flushing\n");
+
+    if (likely(!atomic_read(&q->main_loop_should_exit))) {
+
+        atomic_set(&q->main_loop_should_exit, 1);
+
+        // Wake up the kthread so that it can see that it needs to stop:
+        up(&q->q_sem);
+
+        kthread_stop(q->q_kthread);
+        q->q_kthread = NULL;
+    }
+}
+
+// When CONFIG_VMAP_STACK is defined, the kernel thread stack allocator used by
+// kthread_create_on_node relies on a 2 entry, per-core cache to minimize
+// vmalloc invocations. The cache is NUMA-unaware, so when there is a hit, the
+// stack location ends up being a function of the core assigned to the current
+// thread, instead of being a function of the specified NUMA node. The cache was
+// added to the kernel in commit ac496bf48d97f2503eaa353996a4dd5e4383eaf0
+// ("fork: Optimize task creation by caching two thread stacks per CPU if
+// CONFIG_VMAP_STACK=y")
+//
+// To work around the problematic cache, we create up to three kernel threads
+//   -If the first thread's stack is resident on the preferred node, return this
+//    thread.
+//   -Otherwise, create a second thread. If its stack is resident on the
+//    preferred node, stop the first thread and return this one.
+//   -Otherwise, create a third thread. The stack allocator does not find a
+//    cached stack, and so falls back to vmalloc, which takes the NUMA hint into
+//    consideration. The first two threads are then stopped.
+//
+// When CONFIG_VMAP_STACK is not defined, the first kernel thread is returned.
+//
+// This function is never invoked when there is no NUMA preference (preferred
+// node is NUMA_NO_NODE).
+static struct task_struct *thread_create_on_node(int (*threadfn)(void *data),
+                                                 nv_kthread_q_t *q,
+                                                 int preferred_node,
+                                                 const char *q_name)
+{
+
+    unsigned i, j;
+    static const unsigned attempts = 3;
+    struct task_struct *thread[3];
+
+    for (i = 0;; i++) {
+        struct page *stack;
+
+        thread[i] = kthread_create_on_node(threadfn, q, preferred_node, q_name);
+
+        if (unlikely(IS_ERR(thread[i]))) {
+
+            // Instead of failing, pick the previous thread, even if its
+            // stack is not allocated on the preferred node.
+            if (i > 0)
+                i--;
+
+            break;
+        }
+
+        // vmalloc is not used to allocate the stack, so simply return the
+        // thread, even if its stack may not be allocated on the preferred node
+        if (!is_vmalloc_addr(thread[i]->stack))
+            break;
+
+        // Ran out of attempts - return thread even if its stack may not be
+        // allocated on the preferred node
+        if (i == (attempts - 1))
+            break;
+
+        // Get the NUMA node where the first page of the stack is resident. If
+        // it is the preferred node, select this thread.
+        stack = vmalloc_to_page(thread[i]->stack);
+        if (page_to_nid(stack) == preferred_node)
+            break;
+    }
+
+    for (j = i; j > 0; j--)
+        kthread_stop(thread[j - 1]);
+
+    return thread[i];
+}
+
+int nv_kthread_q_init_on_node(nv_kthread_q_t *q, const char *q_name, int preferred_node)
+{
+    memset(q, 0, sizeof(*q));
+
+    INIT_LIST_HEAD(&q->q_list_head);
+    spin_lock_init(&q->q_lock);
+    sema_init(&q->q_sem, 0);
+
+    if (preferred_node == NV_KTHREAD_NO_NODE) {
+        q->q_kthread = kthread_create(_main_loop, q, q_name);
+    }
+    else {
+        q->q_kthread = thread_create_on_node(_main_loop, q, preferred_node, q_name);
+    }
+
+    if (IS_ERR(q->q_kthread)) {
+        int err = PTR_ERR(q->q_kthread);
+
+        // Clear q_kthread before returning so that nv_kthread_q_stop() can be
+        // safely called on it making error handling easier.
+        q->q_kthread = NULL;
+
+        return err;
+    }
+
+    wake_up_process(q->q_kthread);
+
+    return 0;
+}
+
+int nv_kthread_q_init(nv_kthread_q_t *q, const char *qname)
+{
+    return nv_kthread_q_init_on_node(q, qname, NV_KTHREAD_NO_NODE);
+}
+
+// Returns true (non-zero) if the item was actually scheduled, and false if the
+// item was already pending in a queue.
+static int _raw_q_schedule(nv_kthread_q_t *q, nv_kthread_q_item_t *q_item)
+{
+    unsigned long flags;
+    int ret = 1;
+
+    spin_lock_irqsave(&q->q_lock, flags);
+
+    if (likely(list_empty(&q_item->q_list_node)))
+        list_add_tail(&q_item->q_list_node, &q->q_list_head);
+    else
+        ret = 0;
+
+    spin_unlock_irqrestore(&q->q_lock, flags);
+
+    if (likely(ret))
+        up(&q->q_sem);
+
+    return ret;
+}
+
+void nv_kthread_q_item_init(nv_kthread_q_item_t *q_item,
+                            nv_q_func_t function_to_run,
+                            void *function_args)
+{
+    INIT_LIST_HEAD(&q_item->q_list_node);
+    q_item->function_to_run = function_to_run;
+    q_item->function_args   = function_args;
+}
+
+// Returns true (non-zero) if the q_item got scheduled, false otherwise.
+int nv_kthread_q_schedule_q_item(nv_kthread_q_t *q,
+                                 nv_kthread_q_item_t *q_item)
+{
+    if (unlikely(atomic_read(&q->main_loop_should_exit))) {
+        NVQ_WARN("Not allowed: nv_kthread_q_schedule_q_item was "
+                   "called with a non-alive q: 0x%p\n", q);
+        return 0;
+    }
+
+    return _raw_q_schedule(q, q_item);
+}
+
+static void _q_flush_function(void *args)
+{
+    struct completion *completion = (struct completion *)args;
+    complete(completion);
+}
+
+
+static void _raw_q_flush(nv_kthread_q_t *q)
+{
+    nv_kthread_q_item_t q_item;
+    DECLARE_COMPLETION_ONSTACK(completion);
+
+    nv_kthread_q_item_init(&q_item, _q_flush_function, &completion);
+
+    _raw_q_schedule(q, &q_item);
+
+    // Wait for the flush item to run. Once it has run, then all of the
+    // previously queued items in front of it will have run, so that means
+    // the flush is complete.
+    wait_for_completion(&completion);
+}
+
+void nv_kthread_q_flush(nv_kthread_q_t *q)
+{
+    if (unlikely(atomic_read(&q->main_loop_should_exit))) {
+        NVQ_WARN("Not allowed: nv_kthread_q_flush was called after "
+                   "nv_kthread_q_stop. q: 0x%p\n", q);
+        return;
+    }
+
+    // This 2x flush is not a typing mistake. The queue really does have to be
+    // flushed twice, in order to take care of the case of a q_item that
+    // reschedules itself.
+    _raw_q_flush(q);
+    _raw_q_flush(q);
+}
--- a/kernel-open/nvidia-drm/nv-pci-table.c
+++ b/kernel-open/nvidia-drm/nv-pci-table.c
@@ -25,6 +25,15 @@
 #include <linux/module.h>

 #include "nv-pci-table.h"
+#include "cpuopsys.h"
+
+#if defined(NV_BSD)
+/* Define PCI classes that FreeBSD's linuxkpi is missing */
+#define PCI_VENDOR_ID_NVIDIA 0x10de
+#define PCI_CLASS_DISPLAY_VGA 0x0300
+#define PCI_CLASS_DISPLAY_3D 0x0302
+#define PCI_CLASS_BRIDGE_OTHER 0x0680
+#endif

 /* Devices supported by RM */
 struct pci_device_id nv_pci_table[] = {
@@ -48,7 +57,7 @@ struct pci_device_id nv_pci_table[] = {
 };

 /* Devices supported by all drivers in nvidia.ko */
-struct pci_device_id nv_module_device_table[] = {
+struct pci_device_id nv_module_device_table[4] = {
    {
        .vendor      = PCI_VENDOR_ID_NVIDIA,
        .device      = PCI_ANY_ID,
@@ -76,4 +85,6 @@ struct pci_device_id nv_module_device_table[] = {
    { }
 };

+#if defined(NV_LINUX)
 MODULE_DEVICE_TABLE(pci, nv_module_device_table);
+#endif
--- a/kernel-open/nvidia-drm/nv-pci-table.h
+++ b/kernel-open/nvidia-drm/nv-pci-table.h
@@ -27,5 +27,6 @@
 #include <linux/pci.h>

 extern struct pci_device_id nv_pci_table[];
+extern struct pci_device_id nv_module_device_table[4];

 #endif /* _NV_PCI_TABLE_H_ */
--- a/kernel-open/nvidia-drm/nv_common_utils.h
+++ b/kernel-open/nvidia-drm/nv_common_utils.h
@@ -0,0 +1,120 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2015 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __NV_COMMON_UTILS_H__
+#define __NV_COMMON_UTILS_H__
+
+#include "nvtypes.h"
+#include "nvmisc.h"
+
+#if !defined(TRUE)
+#define TRUE NV_TRUE
+#endif
+
+#if !defined(FALSE)
+#define FALSE NV_FALSE
+#endif
+
+#define NV_IS_UNSIGNED(x) ((__typeof__(x))-1 > 0)
+
+/* Get the length of a statically-sized array. */
+#define ARRAY_LEN(_arr) (sizeof(_arr) / sizeof(_arr[0]))
+
+#define NV_INVALID_HEAD         0xFFFFFFFF
+
+#define NV_INVALID_CONNECTOR_PHYSICAL_INFORMATION (~0)
+
+#if !defined(NV_MIN)
+# define NV_MIN(a,b) (((a)<(b))?(a):(b))
+#endif
+
+#define NV_MIN3(a,b,c) NV_MIN(NV_MIN(a, b), c)
+#define NV_MIN4(a,b,c,d) NV_MIN3(NV_MIN(a,b),c,d)
+
+#if !defined(NV_MAX)
+# define NV_MAX(a,b) (((a)>(b))?(a):(b))
+#endif
+
+#define NV_MAX3(a,b,c) NV_MAX(NV_MAX(a, b), c)
+#define NV_MAX4(a,b,c,d) NV_MAX3(NV_MAX(a,b),c,d)
+
+static inline int NV_LIMIT_VAL_TO_MIN_MAX(int val, int min, int max)
+{
+    if (val < min) {
+        return min;
+    }
+    if (val > max) {
+        return max;
+    }
+    return val;
+}
+
+#define NV_ROUNDUP_DIV(x,y) ((x) / (y) + (((x) % (y)) ? 1 : 0))
+
+/*
+ * Macros used for computing palette entries:
+ *
+ * NV_UNDER_REPLICATE(val, source_size, result_size) expands a value
+ * of source_size bits into a value of target_size bits by shifting
+ * the source value into the high bits and replicating the high bits
+ * of the value into the low bits of the result.
+ *
+ * PALETTE_DEPTH_SHIFT(val, w) maps a colormap entry for a component
+ * that has w bits to an appropriate entry in a LUT of 256 entries.
+ */
+static inline unsigned int NV_UNDER_REPLICATE(unsigned short val,
+                                              int source_size,
+                                              int result_size)
+{
+    return (val << (result_size - source_size)) |
+        (val >> ((source_size << 1) - result_size));
+}
+
+
+static inline unsigned short PALETTE_DEPTH_SHIFT(unsigned short val, int depth)
+{
+    return NV_UNDER_REPLICATE(val, depth, 8);
+}
+
+/*
+ *  Use __builtin_ffs where it is supported, or provide an equivalent
+ *  implementation for platforms like riscv where it is not.
+ */
+#if defined(__GNUC__) && !NVCPU_IS_RISCV64
+static inline int nv_ffs(int x)
+{
+    return __builtin_ffs(x);
+}
+#else
+static inline int nv_ffs(int x)
+{
+    if (x == 0)
+        return 0;
+
+    LOWESTBITIDX_32(x);
+
+    return 1 + x;
+}
+#endif
+
+#endif /* __NV_COMMON_UTILS_H__ */
--- a/kernel-open/nvidia-drm/nvidia-dma-fence-helper.h
+++ b/kernel-open/nvidia-drm/nvidia-dma-fence-helper.h
@@ -43,9 +43,13 @@
 #if defined(NV_LINUX_FENCE_H_PRESENT)
 typedef struct fence nv_dma_fence_t;
 typedef struct fence_ops nv_dma_fence_ops_t;
+typedef struct fence_cb nv_dma_fence_cb_t;
+typedef fence_func_t nv_dma_fence_func_t;
 #else
 typedef struct dma_fence nv_dma_fence_t;
 typedef struct dma_fence_ops nv_dma_fence_ops_t;
+typedef struct dma_fence_cb nv_dma_fence_cb_t;
+typedef dma_fence_func_t nv_dma_fence_func_t;
 #endif

 #if defined(NV_LINUX_FENCE_H_PRESENT)
@@ -97,6 +101,14 @@ static inline int nv_dma_fence_signal(nv_dma_fence_t *fence) {
 #endif
 }

+static inline int nv_dma_fence_signal_locked(nv_dma_fence_t *fence) {
+#if defined(NV_LINUX_FENCE_H_PRESENT)
+    return fence_signal_locked(fence);
+#else
+    return dma_fence_signal_locked(fence);
+#endif
+}
+
 static inline u64 nv_dma_fence_context_alloc(unsigned num) {
 #if defined(NV_LINUX_FENCE_H_PRESENT)
    return fence_context_alloc(num);
@@ -108,7 +120,7 @@ static inline u64 nv_dma_fence_context_alloc(unsigned num) {
 static inline void
 nv_dma_fence_init(nv_dma_fence_t *fence,
                  const nv_dma_fence_ops_t *ops,
-                  spinlock_t *lock, u64 context, unsigned seqno) {
+                  spinlock_t *lock, u64 context, uint64_t seqno) {
 #if defined(NV_LINUX_FENCE_H_PRESENT)
    fence_init(fence, ops, lock, context, seqno);
 #else
@@ -116,6 +128,29 @@ nv_dma_fence_init(nv_dma_fence_t *fence,
 #endif
 }

+static inline void
+nv_dma_fence_set_error(nv_dma_fence_t *fence,
+                       int error) {
+#if defined(NV_DMA_FENCE_SET_ERROR_PRESENT)
+    return dma_fence_set_error(fence, error);
+#elif defined(NV_FENCE_SET_ERROR_PRESENT)
+    return fence_set_error(fence, error);
+#else
+    fence->status = error;
+#endif
+}
+
+static inline int
+nv_dma_fence_add_callback(nv_dma_fence_t *fence,
+                          nv_dma_fence_cb_t *cb,
+                          nv_dma_fence_func_t func) {
+#if defined(NV_LINUX_FENCE_H_PRESENT)
+    return fence_add_callback(fence, cb, func);
+#else
+    return dma_fence_add_callback(fence, cb, func);
+#endif
+}
+
 #endif /* defined(NV_DRM_FENCE_AVAILABLE) */

 #endif /* __NVIDIA_DMA_FENCE_HELPER_H__ */
--- a/kernel-open/nvidia-drm/nvidia-dma-resv-helper.h
+++ b/kernel-open/nvidia-drm/nvidia-dma-resv-helper.h
@@ -121,6 +121,20 @@ static inline void nv_dma_resv_add_excl_fence(nv_dma_resv_t *obj,
 #endif
 }

+static inline void nv_dma_resv_add_shared_fence(nv_dma_resv_t *obj,
+                                                nv_dma_fence_t *fence)
+{
+#if defined(NV_LINUX_DMA_RESV_H_PRESENT)
+#if defined(NV_DMA_RESV_ADD_FENCE_PRESENT)
+    dma_resv_add_fence(obj, fence, DMA_RESV_USAGE_READ);
+#else
+    dma_resv_add_shared_fence(obj, fence);
+#endif
+#else
+    reservation_object_add_shared_fence(obj, fence);
+#endif
+}
+
 #endif /* defined(NV_DRM_FENCE_AVAILABLE) */

 #endif /* __NVIDIA_DMA_RESV_HELPER_H__ */
--- a/kernel-open/nvidia-drm/nvidia-drm-conftest.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-conftest.h
@@ -24,6 +24,7 @@
 #define __NVIDIA_DRM_CONFTEST_H__

 #include "conftest.h"
+#include "nvtypes.h"

 /*
 * NOTE: This file is expected to get included at the top before including any
@@ -61,4 +62,136 @@
 #undef NV_DRM_FENCE_AVAILABLE
 #endif

+/*
+ * We can support color management if either drm_helper_crtc_enable_color_mgmt()
+ * or drm_crtc_enable_color_mgmt() exist.
+ */
+#if defined(NV_DRM_HELPER_CRTC_ENABLE_COLOR_MGMT_PRESENT) || \
+    defined(NV_DRM_CRTC_ENABLE_COLOR_MGMT_PRESENT)
+#define NV_DRM_COLOR_MGMT_AVAILABLE
+#else
+#undef NV_DRM_COLOR_MGMT_AVAILABLE
+#endif
+
+/*
+ * Adapt to quirks in FreeBSD's Linux kernel compatibility layer.
+ */
+#if defined(NV_BSD)
+
+#include <linux/rwsem.h>
+#include <sys/param.h>
+#include <sys/lock.h>
+#include <sys/sx.h>
+
+/* For nv_drm_gem_prime_force_fence_signal */
+#ifndef spin_is_locked
+#if ((__FreeBSD_version >= 1500000) && (__FreeBSD_version < 1500018)) || (__FreeBSD_version < 1401501)
+#define spin_is_locked(lock) mtx_owned(lock.m)
+#else
+#define spin_is_locked(lock) mtx_owned(lock)
+#endif
+#endif
+
+#ifndef rwsem_is_locked
+#define rwsem_is_locked(sem) (((sem)->sx.sx_lock & (SX_LOCK_SHARED)) \
+                              || ((sem)->sx.sx_lock & ~(SX_LOCK_FLAGMASK & ~SX_LOCK_SHARED)))
+#endif
+
+/*
+ * FreeBSD does not define vm_flags_t in its linuxkpi, since there is already
+ * a FreeBSD vm_flags_t (of a different size) and they don't want the names to
+ * collide. Temporarily redefine it when including nv-mm.h
+ */
+#define vm_flags_t unsigned long
+#include "nv-mm.h"
+#undef vm_flags_t
+
+/*
+ * sys/nv.h and nvidia/nv.h have the same header guard
+ * we need to clear it for nvlist_t to get loaded
+ */
+#undef _NV_H_
+#include <sys/nv.h>
+
+/*
+ * For now just use set_page_dirty as the lock variant
+ * is not ported for FreeBSD. (in progress). This calls
+ * vm_page_dirty. Used in nv-mm.h
+ */
+#define set_page_dirty_lock set_page_dirty
+
+/*
+ * FreeBSD does not implement drm_atomic_state_free, simply
+ * default to drm_atomic_state_put
+ */
+#define drm_atomic_state_free drm_atomic_state_put
+
+#if __FreeBSD_version < 1300000
+/* redefine LIST_HEAD_INIT to the linux version */
+#include <linux/list.h>
+#define LIST_HEAD_INIT(name) LINUX_LIST_HEAD_INIT(name)
+#endif
+
+/*
+ * FreeBSD currently has only vmf_insert_pfn_prot defined, and it has a
+ * static assert warning not to use it since all of DRM's usages are in
+ * loops with the vm obj lock(s) held. Instead we should use the lkpi
+ * function itself directly. For us none of this applies so we can just
+ * wrap it in our own definition of vmf_insert_pfn
+ */
+#ifndef NV_VMF_INSERT_PFN_PRESENT
+#define NV_VMF_INSERT_PFN_PRESENT 1
+
+#if __FreeBSD_version < 1300000
+#define VM_SHARED       (1 << 17)
+
+/* Not present in 12.2 */
+static inline vm_fault_t
+lkpi_vmf_insert_pfn_prot_locked(struct vm_area_struct *vma, unsigned long addr,
+    unsigned long pfn, pgprot_t prot)
+{
+       vm_object_t vm_obj = vma->vm_obj;
+       vm_page_t page;
+       vm_pindex_t pindex;
+
+       VM_OBJECT_ASSERT_WLOCKED(vm_obj);
+       pindex = OFF_TO_IDX(addr - vma->vm_start);
+       if (vma->vm_pfn_count == 0)
+               vma->vm_pfn_first = pindex;
+       MPASS(pindex <= OFF_TO_IDX(vma->vm_end));
+
+       page = vm_page_grab(vm_obj, pindex, VM_ALLOC_NORMAL);
+       if (page == NULL) {
+               page = PHYS_TO_VM_PAGE(IDX_TO_OFF(pfn));
+               vm_page_xbusy(page);
+               if (vm_page_insert(page, vm_obj, pindex)) {
+                       vm_page_xunbusy(page);
+                       return (VM_FAULT_OOM);
+               }
+               page->valid = VM_PAGE_BITS_ALL;
+       }
+       pmap_page_set_memattr(page, pgprot2cachemode(prot));
+       vma->vm_pfn_count++;
+
+       return (VM_FAULT_NOPAGE);
+}
+#endif
+
+static inline vm_fault_t
+vmf_insert_pfn(struct vm_area_struct *vma, unsigned long addr,
+    unsigned long pfn)
+{
+       vm_fault_t ret;
+
+       VM_OBJECT_WLOCK(vma->vm_obj);
+       ret = lkpi_vmf_insert_pfn_prot_locked(vma, addr, pfn, vma->vm_page_prot);
+       VM_OBJECT_WUNLOCK(vma->vm_obj);
+
+       return (ret);
+}
+
+#endif
+
+#endif /* defined(NV_BSD) */
+
 #endif /* defined(__NVIDIA_DRM_CONFTEST_H__) */
--- a/kernel-open/nvidia-drm/nvidia-drm-connector.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-connector.c
@@ -27,6 +27,7 @@
 #include "nvidia-drm-helper.h"
 #include "nvidia-drm-priv.h"
 #include "nvidia-drm-connector.h"
+#include "nvidia-drm-crtc.h"
 #include "nvidia-drm-utils.h"
 #include "nvidia-drm-encoder.h"

@@ -42,6 +43,7 @@

 #include <drm/drm_atomic.h>
 #include <drm/drm_atomic_helper.h>
+#include <drm/drm_edid.h>

 static void nv_drm_connector_destroy(struct drm_connector *connector)
 {
@@ -98,7 +100,11 @@ __nv_drm_detect_encoder(struct NvKmsKapiDynamicDisplayParams *pDetectParams,
            break;
    }

+#if defined(NV_DRM_CONNECTOR_HAS_OVERRIDE_EDID)
    if (connector->override_edid) {
+#else
+    if (drm_edid_override_connector_update(connector) > 0) {
+#endif
        const struct drm_property_blob *edid = connector->edid_blob_ptr;

        if (edid->length <= sizeof(pDetectParams->edid.buffer)) {
@@ -118,6 +124,11 @@ __nv_drm_detect_encoder(struct NvKmsKapiDynamicDisplayParams *pDetectParams,
        return false;
    }

+#if defined(NV_DRM_CONNECTOR_HAS_VRR_CAPABLE_PROPERTY)
+    drm_connector_attach_vrr_capable_property(&nv_connector->base);
+    drm_connector_set_vrr_capable_property(&nv_connector->base, pDetectParams->vrrSupported ? true : false);
+#endif
+
    if (pDetectParams->connected) {
        if (!pDetectParams->overrideEdid && pDetectParams->edid.bufferSize) {

@@ -197,6 +208,11 @@ done:

    nv_drm_free(pDetectParams);

+    if (status == connector_status_disconnected &&
+        nv_connector->modeset_permission_filep) {
+        nv_drm_connector_revoke_permissions(dev, nv_connector);
+    }
+
    return status;
 }

@@ -333,10 +349,125 @@ nv_drm_connector_best_encoder(struct drm_connector *connector)
    return NULL;
 }

+#if defined(NV_DRM_MODE_CREATE_DP_COLORSPACE_PROPERTY_HAS_SUPPORTED_COLORSPACES_ARG)
+static const NvU32 __nv_drm_connector_supported_colorspaces =
+    BIT(DRM_MODE_COLORIMETRY_BT2020_RGB) |
+    BIT(DRM_MODE_COLORIMETRY_BT2020_YCC);
+#endif
+
+#if defined(NV_DRM_CONNECTOR_ATTACH_HDR_OUTPUT_METADATA_PROPERTY_PRESENT)
+static int
+__nv_drm_connector_atomic_check(struct drm_connector *connector,
+                                struct drm_atomic_state *state)
+{
+    struct drm_connector_state *new_connector_state =
+        drm_atomic_get_new_connector_state(state, connector);
+    struct drm_connector_state *old_connector_state =
+        drm_atomic_get_old_connector_state(state, connector);
+    struct nv_drm_device *nv_dev = to_nv_device(connector->dev);
+
+    struct drm_crtc *crtc = new_connector_state->crtc;
+    struct drm_crtc_state *crtc_state;
+    struct nv_drm_crtc_state *nv_crtc_state;
+    struct NvKmsKapiHeadRequestedConfig *req_config;
+
+    if (!crtc) {
+        return 0;
+    }
+
+    crtc_state = drm_atomic_get_new_crtc_state(state, crtc);
+    nv_crtc_state = to_nv_crtc_state(crtc_state);
+    req_config = &nv_crtc_state->req_config;
+
+    /*
+     * Override metadata for the entire head instead of allowing NVKMS to derive
+     * it from the layers' metadata.
+     *
+     * This is the metadata that will sent to the display, and if applicable,
+     * layers will be tone mapped to this metadata rather than that of the
+     * display.
+     */
+    req_config->flags.hdrInfoFrameChanged =
+        !drm_connector_atomic_hdr_metadata_equal(old_connector_state,
+                                                 new_connector_state);
+    if (new_connector_state->hdr_output_metadata &&
+        new_connector_state->hdr_output_metadata->data) {
+
+        /*
+         * Note that HDMI definitions are used here even though we might not
+         * be using HDMI. While that seems odd, it is consistent with
+         * upstream behavior.
+         */
+
+        struct hdr_output_metadata *hdr_metadata =
+            new_connector_state->hdr_output_metadata->data;
+        struct hdr_metadata_infoframe *info_frame =
+            &hdr_metadata->hdmi_metadata_type1;
+        unsigned int i;
+
+        if (hdr_metadata->metadata_type != HDMI_STATIC_METADATA_TYPE1) {
+            return -EINVAL;
+        }
+
+        for (i = 0; i < ARRAY_SIZE(info_frame->display_primaries); i++) {
+            req_config->modeSetConfig.hdrInfoFrame.staticMetadata.displayPrimaries[i].x =
+                info_frame->display_primaries[i].x;
+            req_config->modeSetConfig.hdrInfoFrame.staticMetadata.displayPrimaries[i].y =
+                info_frame->display_primaries[i].y;
+        }
+
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.whitePoint.x =
+            info_frame->white_point.x;
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.whitePoint.y =
+            info_frame->white_point.y;
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.maxDisplayMasteringLuminance =
+            info_frame->max_display_mastering_luminance;
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.minDisplayMasteringLuminance =
+            info_frame->min_display_mastering_luminance;
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.maxCLL =
+            info_frame->max_cll;
+        req_config->modeSetConfig.hdrInfoFrame.staticMetadata.maxFALL =
+            info_frame->max_fall;
+
+        req_config->modeSetConfig.hdrInfoFrame.eotf = info_frame->eotf;
+
+        req_config->modeSetConfig.hdrInfoFrame.enabled = NV_TRUE;
+    } else {
+        req_config->modeSetConfig.hdrInfoFrame.enabled = NV_FALSE;
+    }
+
+    req_config->flags.colorimetryChanged =
+        (old_connector_state->colorspace != new_connector_state->colorspace);
+    // When adding a case here, also add to __nv_drm_connector_supported_colorspaces
+    switch (new_connector_state->colorspace) {
+        case DRM_MODE_COLORIMETRY_DEFAULT:
+            req_config->modeSetConfig.colorimetry =
+                NVKMS_OUTPUT_COLORIMETRY_DEFAULT;
+            break;
+        case DRM_MODE_COLORIMETRY_BT2020_RGB:
+        case DRM_MODE_COLORIMETRY_BT2020_YCC:
+            // Ignore RGB/YCC
+            // See https://patchwork.freedesktop.org/patch/525496/?series=111865&rev=4
+            req_config->modeSetConfig.colorimetry =
+                NVKMS_OUTPUT_COLORIMETRY_BT2100;
+            break;
+        default:
+            // XXX HDR TODO: Add support for more color spaces
+            NV_DRM_DEV_LOG_ERR(nv_dev, "Unsupported color space");
+            return -EINVAL;
+    }
+
+    return 0;
+}
+#endif /* defined(NV_DRM_CONNECTOR_ATTACH_HDR_OUTPUT_METADATA_PROPERTY_PRESENT) */
+
 static const struct drm_connector_helper_funcs nv_connector_helper_funcs = {
    .get_modes    = nv_drm_connector_get_modes,
    .mode_valid   = nv_drm_connector_mode_valid,
    .best_encoder = nv_drm_connector_best_encoder,
+#if defined(NV_DRM_CONNECTOR_ATTACH_HDR_OUTPUT_METADATA_PROPERTY_PRESENT)
+    .atomic_check = __nv_drm_connector_atomic_check,
+#endif
 };

 static struct drm_connector*
@@ -362,6 +493,8 @@ nv_drm_connector_new(struct drm_device *dev,
    nv_connector->physicalIndex = physicalIndex;
    nv_connector->type     = type;
    nv_connector->internal = internal;
+    nv_connector->modeset_permission_filep = NULL;
+    nv_connector->modeset_permission_crtc = NULL;

    strcpy(nv_connector->dpAddress, dpAddress);

@@ -387,6 +520,32 @@ nv_drm_connector_new(struct drm_device *dev,
            DRM_CONNECTOR_POLL_CONNECT | DRM_CONNECTOR_POLL_DISCONNECT;
    }

+#if defined(NV_DRM_CONNECTOR_ATTACH_HDR_OUTPUT_METADATA_PROPERTY_PRESENT)
+    if (nv_connector->type == NVKMS_CONNECTOR_TYPE_HDMI) {
+#if defined(NV_DRM_MODE_CREATE_DP_COLORSPACE_PROPERTY_HAS_SUPPORTED_COLORSPACES_ARG)
+        if (drm_mode_create_hdmi_colorspace_property(
+                &nv_connector->base,
+                __nv_drm_connector_supported_colorspaces) == 0) {
+#else
+        if (drm_mode_create_hdmi_colorspace_property(&nv_connector->base) == 0) {
+#endif
+            drm_connector_attach_colorspace_property(&nv_connector->base);
+        }
+        drm_connector_attach_hdr_output_metadata_property(&nv_connector->base);
+    } else if (nv_connector->type == NVKMS_CONNECTOR_TYPE_DP) {
+#if defined(NV_DRM_MODE_CREATE_DP_COLORSPACE_PROPERTY_HAS_SUPPORTED_COLORSPACES_ARG)
+        if (drm_mode_create_dp_colorspace_property(
+                &nv_connector->base,
+                __nv_drm_connector_supported_colorspaces) == 0) {
+#else
+        if (drm_mode_create_dp_colorspace_property(&nv_connector->base) == 0) {
+#endif
+            drm_connector_attach_colorspace_property(&nv_connector->base);
+        }
+        drm_connector_attach_hdr_output_metadata_property(&nv_connector->base);
+    }
+#endif /* defined(NV_DRM_CONNECTOR_ATTACH_HDR_OUTPUT_METADATA_PROPERTY_PRESENT) */
+
    /* Register connector with DRM subsystem */

    ret = drm_connector_register(&nv_connector->base);
@@ -464,4 +623,26 @@ done:
    return connector;
 }

+/*
+ * Revoke the permissions on this connector.
+ */
+bool nv_drm_connector_revoke_permissions(struct drm_device *dev,
+                                         struct nv_drm_connector* nv_connector)
+{
+    struct nv_drm_device *nv_dev = to_nv_device(dev);
+    bool ret = true;
+
+    if (nv_connector->modeset_permission_crtc) {
+        if (nv_connector->nv_detected_encoder) {
+            ret = nvKms->revokePermissions(
+                nv_dev->pDevice, nv_connector->modeset_permission_crtc->head,
+                nv_connector->nv_detected_encoder->hDisplay);
+        }
+        nv_connector->modeset_permission_crtc->modeset_permission_filep = NULL;
+        nv_connector->modeset_permission_crtc = NULL;
+    }
+    nv_connector->modeset_permission_filep = NULL;
+    return ret;
+}
+
 #endif
--- a/kernel-open/nvidia-drm/nvidia-drm-connector.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-connector.h
@@ -51,6 +51,20 @@ struct nv_drm_connector {

    atomic_t connection_status_dirty;

+    /**
+     * @modeset_permission_filep:
+     *
+     * The filep using this connector with DRM_IOCTL_NVIDIA_GRANT_PERMISSIONS.
+     */
+    struct drm_file *modeset_permission_filep;
+
+    /**
+     * @modeset_permission_crtc:
+     *
+     * The crtc using this connector with DRM_IOCTL_NVIDIA_GRANT_PERMISSIONS.
+     */
+    struct nv_drm_crtc *modeset_permission_crtc;
+
    struct drm_connector base;
 };

@@ -84,6 +98,9 @@ nv_drm_get_connector(struct drm_device *dev,
                     NvBool internal,
                     char dpAddress[NVKMS_DP_ADDRESS_STRING_LENGTH]);

+bool nv_drm_connector_revoke_permissions(struct drm_device *dev,
+                                         struct nv_drm_connector *nv_connector);
+
 #endif /* NV_DRM_ATOMIC_MODESET_AVAILABLE */

 #endif /* __NVIDIA_DRM_CONNECTOR_H__ */
--- a/kernel-open/nvidia-drm/nvidia-drm-crtc.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-crtc.c
--- a/kernel-open/nvidia-drm/nvidia-drm-crtc.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-crtc.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+ * Copyright (c) 2016-2022, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
@@ -35,37 +35,15 @@

 #include <drm/drm_crtc.h>

-#if defined(NV_DRM_ALPHA_BLENDING_AVAILABLE) || defined(NV_DRM_ROTATION_AVAILABLE)
-/* For DRM_ROTATE_* , DRM_REFLECT_* */
-#include <drm/drm_blend.h>
-#endif
-
-#if defined(NV_DRM_ROTATION_AVAILABLE)
-/* For DRM_MODE_ROTATE_* and DRM_MODE_REFLECT_* */
-#include <uapi/drm/drm_mode.h>
-#endif
-
 #include "nvtypes.h"
 #include "nvkms-kapi.h"

-#if defined(NV_DRM_ROTATION_AVAILABLE)
-/*
- * 19-05-2017 c2c446ad29437bb92b157423c632286608ebd3ec has added
- * DRM_MODE_ROTATE_* and DRM_MODE_REFLECT_* to UAPI and removed
- * DRM_ROTATE_* and DRM_MODE_REFLECT_*
- */
-#if !defined(DRM_MODE_ROTATE_0)
-#define DRM_MODE_ROTATE_0       DRM_ROTATE_0
-#define DRM_MODE_ROTATE_90      DRM_ROTATE_90
-#define DRM_MODE_ROTATE_180     DRM_ROTATE_180
-#define DRM_MODE_ROTATE_270     DRM_ROTATE_270
-#define DRM_MODE_REFLECT_X      DRM_REFLECT_X
-#define DRM_MODE_REFLECT_Y      DRM_REFLECT_Y
-#define DRM_MODE_ROTATE_MASK    DRM_ROTATE_MASK
-#define DRM_MODE_REFLECT_MASK   DRM_REFLECT_MASK
-#endif
-
-#endif //NV_DRM_ROTATION_AVAILABLE
+enum nv_drm_transfer_function {
+    NV_DRM_TRANSFER_FUNCTION_DEFAULT,
+    NV_DRM_TRANSFER_FUNCTION_LINEAR,
+    NV_DRM_TRANSFER_FUNCTION_PQ,
+    NV_DRM_TRANSFER_FUNCTION_MAX,
+};

 struct nv_drm_crtc {
    NvU32 head;
@@ -85,6 +63,15 @@ struct nv_drm_crtc {
     */
    spinlock_t flip_list_lock;

+    /**
+     * @modeset_permission_filep:
+     *
+     * The filep using this crtc with DRM_IOCTL_NVIDIA_GRANT_PERMISSIONS.
+     */
+    struct drm_file *modeset_permission_filep;
+
+    struct NvKmsLUTCaps olut_caps;
+
    struct drm_crtc base;
 };

@@ -164,6 +151,12 @@ struct nv_drm_crtc_state {
     * nv_drm_atomic_crtc_destroy_state().
     */
    struct nv_drm_flip *nv_flip;
+
+    enum nv_drm_transfer_function regamma_tf;
+    struct drm_property_blob *regamma_lut;
+    uint64_t regamma_divisor;
+    struct nv_drm_lut_surface *regamma_drm_lut_surface;
+    NvBool regamma_changed;
 };

 static inline struct nv_drm_crtc_state *to_nv_crtc_state(struct drm_crtc_state *state)
@@ -171,6 +164,11 @@ static inline struct nv_drm_crtc_state *to_nv_crtc_state(struct drm_crtc_state *
    return container_of(state, struct nv_drm_crtc_state, base);
 }

+static inline const struct nv_drm_crtc_state *to_nv_crtc_state_const(const struct drm_crtc_state *state)
+{
+    return container_of(state, struct nv_drm_crtc_state, base);
+}
+
 struct nv_drm_plane {
    /**
     * @base:
@@ -192,6 +190,9 @@ struct nv_drm_plane {
     * Index of this plane in the per head array of layers.
     */
    uint32_t layer_idx;
+
+    struct NvKmsLUTCaps ilut_caps;
+    struct NvKmsLUTCaps tmo_caps;
 };

 static inline struct nv_drm_plane *to_nv_plane(struct drm_plane *plane)
@@ -202,9 +203,43 @@ static inline struct nv_drm_plane *to_nv_plane(struct drm_plane *plane)
    return container_of(plane, struct nv_drm_plane, base);
 }

+struct nv_drm_lut_surface {
+    struct NvKmsKapiDevice *pDevice;
+    struct NvKmsKapiMemory *nvkms_memory;
+    struct NvKmsKapiSurface *nvkms_surface;
+    struct {
+        NvU32 vssSegments;
+        enum NvKmsLUTVssType vssType;
+
+        NvU32 lutEntries;
+        enum NvKmsLUTFormat entryFormat;
+
+    } properties;
+    void *buffer;
+    struct kref refcount;
+};
+
 struct nv_drm_plane_state {
    struct drm_plane_state base;
    s32 __user *fd_user_ptr;
+    enum NvKmsInputColorSpace input_colorspace;
+#if defined(NV_DRM_HAS_HDR_OUTPUT_METADATA)
+    struct drm_property_blob *hdr_output_metadata;
+#endif
+    struct drm_property_blob *lms_ctm;
+    struct drm_property_blob *lms_to_itp_ctm;
+    struct drm_property_blob *itp_to_lms_ctm;
+    struct drm_property_blob *blend_ctm;
+
+    enum nv_drm_transfer_function degamma_tf;
+    struct drm_property_blob *degamma_lut;
+    uint64_t degamma_multiplier; /* S31.32 Sign-Magnitude Format */
+    struct nv_drm_lut_surface *degamma_drm_lut_surface;
+    NvBool degamma_changed;
+
+    struct drm_property_blob *tmo_lut;
+    struct nv_drm_lut_surface *tmo_drm_lut_surface;
+    NvBool tmo_changed;
 };

 static inline struct nv_drm_plane_state *to_nv_drm_plane_state(struct drm_plane_state *state)
@@ -212,6 +247,11 @@ static inline struct nv_drm_plane_state *to_nv_drm_plane_state(struct drm_plane_
    return container_of(state, struct nv_drm_plane_state, base);
 }

+static inline const struct nv_drm_plane_state *to_nv_drm_plane_state_const(const struct drm_plane_state *state)
+{
+    return container_of(state, const struct nv_drm_plane_state, base);
+}
+
 static inline struct nv_drm_crtc *to_nv_crtc(struct drm_crtc *crtc)
 {
    if (crtc == NULL) {
--- a/kernel-open/nvidia-drm/nvidia-drm-drv.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-drv.c
--- a/kernel-open/nvidia-drm/nvidia-drm-drv.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-drv.h
@@ -31,6 +31,12 @@ int nv_drm_probe_devices(void);

 void nv_drm_remove_devices(void);

+void nv_drm_suspend_resume(NvBool suspend);
+
+void nv_drm_register_drm_device(const nv_gpu_info_t *);
+
+void nv_drm_update_drm_driver_features(void);
+
 #endif /* defined(NV_DRM_AVAILABLE) */

 #endif /* __NVIDIA_DRM_DRV_H__ */
--- a/kernel-open/nvidia-drm/nvidia-drm-encoder.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-encoder.c
@@ -205,7 +205,7 @@ nv_drm_add_encoder(struct drm_device *dev, NvKmsKapiDisplay hDisplay)
    encoder = nv_drm_encoder_new(dev,
                                 displayInfo->handle,
                                 connectorInfo->signalFormat,
-                                 get_crtc_mask(dev, connectorInfo->headMask));
+                                 get_crtc_mask(dev, displayInfo->headMask));

    if (IS_ERR(encoder)) {
        ret = PTR_ERR(encoder);
@@ -300,7 +300,7 @@ void nv_drm_handle_display_change(struct nv_drm_device *nv_dev,

    nv_drm_connector_mark_connection_status_dirty(nv_encoder->nv_connector);

-    drm_kms_helper_hotplug_event(dev);
+    schedule_delayed_work(&nv_dev->hotplug_event_work, 0);
 }

 void nv_drm_handle_dynamic_display_connected(struct nv_drm_device *nv_dev,
@@ -347,6 +347,6 @@ void nv_drm_handle_dynamic_display_connected(struct nv_drm_device *nv_dev,
    drm_reinit_primary_mode_group(dev);
 #endif

-    drm_kms_helper_hotplug_event(dev);
+    schedule_delayed_work(&nv_dev->hotplug_event_work, 0);
 }
 #endif
--- a/kernel-open/nvidia-drm/nvidia-drm-fb.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-fb.c
@@ -36,12 +36,15 @@

 static void __nv_drm_framebuffer_free(struct nv_drm_framebuffer *nv_fb)
 {
+    struct drm_framebuffer *fb = &nv_fb->base;
    uint32_t i;

    /* Unreference gem object */
-    for (i = 0; i < ARRAY_SIZE(nv_fb->nv_gem); i++) {
-        if (nv_fb->nv_gem[i] != NULL) {
-            nv_drm_gem_object_unreference_unlocked(nv_fb->nv_gem[i]);
+    for (i = 0; i < NVKMS_MAX_PLANES_PER_SURFACE; i++) {
+        struct drm_gem_object *gem = nv_fb_get_gem_obj(fb, i);
+        if (gem != NULL) {
+            struct nv_drm_gem_object *nv_gem = to_nv_gem_object(gem);
+            nv_drm_gem_object_unreference_unlocked(nv_gem);
        }
    }

@@ -69,10 +72,8 @@ static int
 nv_drm_framebuffer_create_handle(struct drm_framebuffer *fb,
                                 struct drm_file *file, unsigned int *handle)
 {
-    struct nv_drm_framebuffer *nv_fb = to_nv_framebuffer(fb);
-
    return nv_drm_gem_handle_create(file,
-                                    nv_fb->nv_gem[0],
+                                    to_nv_gem_object(nv_fb_get_gem_obj(fb, 0)),
                                    handle);
 }

@@ -88,6 +89,7 @@ static struct nv_drm_framebuffer *nv_drm_framebuffer_alloc(
 {
    struct nv_drm_device *nv_dev = to_nv_device(dev);
    struct nv_drm_framebuffer *nv_fb;
+    struct nv_drm_gem_object *nv_gem;
    const int num_planes = nv_drm_format_num_planes(cmd->pixel_format);
    uint32_t i;

@@ -101,21 +103,22 @@ static struct nv_drm_framebuffer *nv_drm_framebuffer_alloc(
        return ERR_PTR(-ENOMEM);
    }

-    if (num_planes > ARRAY_SIZE(nv_fb->nv_gem)) {
+    if (num_planes > NVKMS_MAX_PLANES_PER_SURFACE) {
        NV_DRM_DEV_DEBUG_DRIVER(nv_dev, "Unsupported number of planes");
        goto failed;
    }

    for (i = 0; i < num_planes; i++) {
-        if ((nv_fb->nv_gem[i] = nv_drm_gem_object_lookup(
-                        dev,
-                        file,
-                        cmd->handles[i])) == NULL) {
+        nv_gem = nv_drm_gem_object_lookup(dev, file, cmd->handles[i]);
+
+        if (nv_gem == NULL) {
            NV_DRM_DEV_DEBUG_DRIVER(
                nv_dev,
                "Failed to find gem object of type nvkms memory");
            goto failed;
        }
+
+        nv_fb_set_gem_obj(&nv_fb->base, i, &nv_gem->base);
    }

     return nv_fb;
@@ -135,12 +138,14 @@ static int nv_drm_framebuffer_init(struct drm_device *dev,
 {
    struct nv_drm_device *nv_dev = to_nv_device(dev);
    struct NvKmsKapiCreateSurfaceParams params = { };
+    struct nv_drm_gem_object *nv_gem;
+    struct drm_framebuffer *fb = &nv_fb->base;
    uint32_t i;
    int ret;

    /* Initialize the base framebuffer object and add it to drm subsystem */

-    ret = drm_framebuffer_init(dev, &nv_fb->base, &nv_framebuffer_funcs);
+    ret = drm_framebuffer_init(dev, fb, &nv_framebuffer_funcs);
    if (ret != 0) {
        NV_DRM_DEV_DEBUG_DRIVER(
            nv_dev,
@@ -148,15 +153,18 @@ static int nv_drm_framebuffer_init(struct drm_device *dev,
        return ret;
    }

-    for (i = 0; i < ARRAY_SIZE(nv_fb->nv_gem); i++) {
-        if (nv_fb->nv_gem[i] != NULL) {
-            params.planes[i].memory = nv_fb->nv_gem[i]->pMemory;
-            params.planes[i].offset = nv_fb->base.offsets[i];
-            params.planes[i].pitch = nv_fb->base.pitches[i];
+    for (i = 0; i < NVKMS_MAX_PLANES_PER_SURFACE; i++) {
+        struct drm_gem_object *gem = nv_fb_get_gem_obj(fb, i);
+        if (gem != NULL) {
+            nv_gem = to_nv_gem_object(gem);
+
+            params.planes[i].memory = nv_gem->pMemory;
+            params.planes[i].offset = fb->offsets[i];
+            params.planes[i].pitch = fb->pitches[i];
        }
    }
-    params.height = nv_fb->base.height;
-    params.width = nv_fb->base.width;
+    params.height = fb->height;
+    params.width = fb->width;
    params.format = format;

    if (have_modifier) {
@@ -164,6 +172,17 @@ static int nv_drm_framebuffer_init(struct drm_device *dev,
        params.layout = (modifier & 0x10) ?
            NvKmsSurfaceMemoryLayoutBlockLinear :
            NvKmsSurfaceMemoryLayoutPitch;
+
+        // See definition of DRM_FORMAT_MOD_NVIDIA_BLOCK_LINEAR_2D, we are testing
+        // 'c', the lossless compression field of the modifier
+        if (params.layout == NvKmsSurfaceMemoryLayoutBlockLinear &&
+            (modifier >> 23) & 0x7) {
+            NV_DRM_DEV_LOG_ERR(
+                    nv_dev,
+                    "Cannot create FB from compressible surface allocation");
+            goto fail;
+        }
+
        params.log2GobsPerBlockY = modifier & 0xf;
    } else {
        params.explicit_layout = false;
@@ -174,11 +193,14 @@ static int nv_drm_framebuffer_init(struct drm_device *dev,
    nv_fb->pSurface = nvKms->createSurface(nv_dev->pDevice, &params);
    if (nv_fb->pSurface == NULL) {
        NV_DRM_DEV_DEBUG_DRIVER(nv_dev, "Failed to create NvKmsKapiSurface");
-        drm_framebuffer_cleanup(&nv_fb->base);
-        return -EINVAL;
+        goto fail;
    }

    return 0;
+
+fail:
+    drm_framebuffer_cleanup(fb);
+    return -EINVAL;
 }

 struct drm_framebuffer *nv_drm_internal_framebuffer_create(
@@ -218,7 +240,7 @@ struct drm_framebuffer *nv_drm_internal_framebuffer_create(
        if (nv_dev->modifiers[i] == DRM_FORMAT_MOD_INVALID) {
            NV_DRM_DEV_DEBUG_DRIVER(
                nv_dev,
-                "Invalid format modifier for framebuffer object: 0x%016llx",
+                "Invalid format modifier for framebuffer object: 0x%016" NvU64_fmtx,
                modifier);
            return ERR_PTR(-EINVAL);
        }
--- a/kernel-open/nvidia-drm/nvidia-drm-fb.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-fb.h
@@ -41,8 +41,10 @@
 struct nv_drm_framebuffer {
    struct NvKmsKapiSurface *pSurface;

-    struct nv_drm_gem_object*
-        nv_gem[NVKMS_MAX_PLANES_PER_SURFACE];
+#if !defined(NV_DRM_FRAMEBUFFER_OBJ_PRESENT)
+    struct drm_gem_object*
+        obj[NVKMS_MAX_PLANES_PER_SURFACE];
+#endif

    struct drm_framebuffer base;
 };
@@ -56,6 +58,29 @@ static inline struct nv_drm_framebuffer *to_nv_framebuffer(
    return container_of(fb, struct nv_drm_framebuffer, base);
 }

+static inline struct drm_gem_object *nv_fb_get_gem_obj(
+    struct drm_framebuffer *fb,
+    uint32_t plane)
+{
+#if defined(NV_DRM_FRAMEBUFFER_OBJ_PRESENT)
+    return fb->obj[plane];
+#else
+    return to_nv_framebuffer(fb)->obj[plane];
+#endif
+}
+
+static inline void nv_fb_set_gem_obj(
+    struct drm_framebuffer *fb,
+    uint32_t plane,
+    struct drm_gem_object *obj)
+{
+#if defined(NV_DRM_FRAMEBUFFER_OBJ_PRESENT)
+    fb->obj[plane] = obj;
+#else
+    to_nv_framebuffer(fb)->obj[plane] = obj;
+#endif
+}
+
 struct drm_framebuffer *nv_drm_internal_framebuffer_create(
    struct drm_device *dev,
    struct drm_file *file,
--- a/kernel-open/nvidia-drm/nvidia-drm-fence.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-fence.c
--- a/kernel-open/nvidia-drm/nvidia-drm-prime-fence.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-prime-fence.h
@@ -35,11 +35,27 @@ struct drm_device;
 int nv_drm_fence_supported_ioctl(struct drm_device *dev,
                                 void *data, struct drm_file *filep);

-int nv_drm_fence_context_create_ioctl(struct drm_device *dev,
-                                      void *data, struct drm_file *filep);
+int nv_drm_prime_fence_context_create_ioctl(struct drm_device *dev,
+                                            void *data, struct drm_file *filep);

-int nv_drm_gem_fence_attach_ioctl(struct drm_device *dev,
-                                  void *data, struct drm_file *filep);
+int nv_drm_gem_prime_fence_attach_ioctl(struct drm_device *dev,
+                                        void *data, struct drm_file *filep);
+
+int nv_drm_semsurf_fence_ctx_create_ioctl(struct drm_device *dev,
+                                          void *data,
+                                          struct drm_file *filep);
+
+int nv_drm_semsurf_fence_create_ioctl(struct drm_device *dev,
+                                      void *data,
+                                      struct drm_file *filep);
+
+int nv_drm_semsurf_fence_wait_ioctl(struct drm_device *dev,
+                                    void *data,
+                                    struct drm_file *filep);
+
+int nv_drm_semsurf_fence_attach_ioctl(struct drm_device *dev,
+                                      void *data,
+                                      struct drm_file *filep);

 #endif /* NV_DRM_FENCE_AVAILABLE */

--- a/kernel-open/nvidia-drm/nvidia-drm-format.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-format.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
+ * Copyright (c) 2019-2022, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
@@ -40,9 +40,16 @@ static const u32  nvkms_to_drm_format[] = {
    [NvKmsSurfaceMemoryFormatR5G6B5]      = DRM_FORMAT_RGB565,
    [NvKmsSurfaceMemoryFormatA8R8G8B8]    = DRM_FORMAT_ARGB8888,
    [NvKmsSurfaceMemoryFormatX8R8G8B8]    = DRM_FORMAT_XRGB8888,
+    [NvKmsSurfaceMemoryFormatX8B8G8R8]    = DRM_FORMAT_XBGR8888,
    [NvKmsSurfaceMemoryFormatA2B10G10R10] = DRM_FORMAT_ABGR2101010,
    [NvKmsSurfaceMemoryFormatX2B10G10R10] = DRM_FORMAT_XBGR2101010,
    [NvKmsSurfaceMemoryFormatA8B8G8R8]    = DRM_FORMAT_ABGR8888,
+#if defined(DRM_FORMAT_ABGR16161616F)
+    [NvKmsSurfaceMemoryFormatRF16GF16BF16AF16] = DRM_FORMAT_ABGR16161616F,
+#endif
+#if defined(DRM_FORMAT_XBGR16161616F)
+    [NvKmsSurfaceMemoryFormatRF16GF16BF16XF16] = DRM_FORMAT_XBGR16161616F,
+#endif

    [NvKmsSurfaceMemoryFormatY8_U8__Y8_V8_N422] = DRM_FORMAT_YUYV,
    [NvKmsSurfaceMemoryFormatU8_Y8__V8_Y8_N422] = DRM_FORMAT_UYVY,
--- a/kernel-open/nvidia-drm/nvidia-drm-gem-dma-buf.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem-dma-buf.c
@@ -71,12 +71,42 @@ static int __nv_drm_gem_dma_buf_create_mmap_offset(
 static int __nv_drm_gem_dma_buf_mmap(struct nv_drm_gem_object *nv_gem,
                                     struct vm_area_struct *vma)
 {
+#if defined(NV_LINUX)
    struct dma_buf_attachment *attach = nv_gem->base.import_attach;
    struct dma_buf *dma_buf = attach->dmabuf;
+#endif
    struct file *old_file;
    int ret;

    /* check if buffer supports mmap */
+#if defined(NV_BSD)
+    /*
+     * Most of the FreeBSD DRM code refers to struct file*, which is actually
+     * a struct linux_file*. The dmabuf code in FreeBSD is not actually plumbed
+     * through the same linuxkpi bits it seems (probably so it can be used
+     * elsewhere), so dma_buf->file really is a native FreeBSD struct file...
+     */
+    if (!nv_gem->base.filp->f_op->mmap)
+        return -EINVAL;
+
+    /* readjust the vma */
+    get_file(nv_gem->base.filp);
+    old_file = vma->vm_file;
+    vma->vm_file = nv_gem->base.filp;
+    vma->vm_pgoff -= drm_vma_node_start(&nv_gem->base.vma_node);
+
+    ret = nv_gem->base.filp->f_op->mmap(nv_gem->base.filp, vma);
+
+    if (ret) {
+        /* restore old parameters on failure */
+        vma->vm_file = old_file;
+        vma->vm_pgoff += drm_vma_node_start(&nv_gem->base.vma_node);
+        fput(nv_gem->base.filp);
+    } else {
+        if (old_file)
+            fput(old_file);
+    }
+#else
    if (!dma_buf->file->f_op->mmap)
        return -EINVAL;

@@ -84,18 +114,20 @@ static int __nv_drm_gem_dma_buf_mmap(struct nv_drm_gem_object *nv_gem,
    get_file(dma_buf->file);
    old_file = vma->vm_file;
    vma->vm_file = dma_buf->file;
-    vma->vm_pgoff -= drm_vma_node_start(&nv_gem->base.vma_node);;
+    vma->vm_pgoff -= drm_vma_node_start(&nv_gem->base.vma_node);

    ret = dma_buf->file->f_op->mmap(dma_buf->file, vma);

    if (ret) {
        /* restore old parameters on failure */
        vma->vm_file = old_file;
+        vma->vm_pgoff += drm_vma_node_start(&nv_gem->base.vma_node);
        fput(dma_buf->file);
    } else {
        if (old_file)
            fput(old_file);
    }
+#endif

    return ret;
 }
--- a/kernel-open/nvidia-drm/nvidia-drm-gem-nvkms-memory.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem-nvkms-memory.c
@@ -37,6 +37,9 @@
 #endif

 #include <linux/io.h>
+#if defined(NV_BSD)
+#include <vm/vm_pageout.h>
+#endif

 #include "nv-mm.h"

@@ -68,9 +71,20 @@ static void __nv_drm_gem_nvkms_memory_free(struct nv_drm_gem_object *nv_gem)
    nv_drm_free(nv_nvkms_memory);
 }

+static int __nv_drm_gem_nvkms_map(
+    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory);
+
 static int __nv_drm_gem_nvkms_mmap(struct nv_drm_gem_object *nv_gem,
                                   struct vm_area_struct *vma)
 {
+    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory =
+        to_nv_nvkms_memory(nv_gem);
+
+    int ret = __nv_drm_gem_nvkms_map(nv_nvkms_memory);
+    if (ret) {
+       return ret;
+    }
+
    return drm_gem_mmap_obj(&nv_gem->base,
                drm_vma_node_size(&nv_gem->base.vma_node) << PAGE_SHIFT, vma);
 }
@@ -93,9 +107,19 @@ static vm_fault_t __nv_drm_gem_nvkms_handle_vma_fault(
    if (nv_nvkms_memory->pages_count == 0) {
        pfn = (unsigned long)(uintptr_t)nv_nvkms_memory->pPhysicalAddress;
        pfn >>= PAGE_SHIFT;
+#if defined(NV_LINUX)
+        /*
+         * FreeBSD doesn't set pgoff. We instead have pfn be the base physical
+         * address, and we will calculate the index pidx from the virtual address.
+         *
+         * This only works because linux_cdev_pager_populate passes the pidx as
+         * vmf->virtual_address. Then we turn the virtual address
+         * into a physical page number.
+         */
        pfn += page_offset;
+#endif
    } else {
-        BUG_ON(page_offset > nv_nvkms_memory->pages_count);
+        BUG_ON(page_offset >= nv_nvkms_memory->pages_count);
        pfn = page_to_pfn(nv_nvkms_memory->pages[page_offset]);
    }

@@ -131,13 +155,20 @@ static struct drm_gem_object *__nv_drm_gem_nvkms_prime_dup(
    const struct nv_drm_gem_object *nv_gem_src);

 static int __nv_drm_gem_nvkms_map(
-    struct nv_drm_device *nv_dev,
-    struct NvKmsKapiMemory *pMemory,
-    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory,
-    uint64_t size)
+    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory)
 {
+    int ret = 0;
+    struct nv_drm_device *nv_dev = nv_nvkms_memory->base.nv_dev;
+    struct NvKmsKapiMemory *pMemory = nv_nvkms_memory->base.pMemory;
+
+    mutex_lock(&nv_nvkms_memory->map_lock);
+
+    if (nv_nvkms_memory->physically_mapped) {
+        goto done;
+    }
+
    if (!nv_dev->hasVideoMemory) {
-        return 0;
+        goto done;
    }

    if (!nvKms->mapMemory(nv_dev->pDevice,
@@ -148,12 +179,13 @@ static int __nv_drm_gem_nvkms_map(
            nv_dev,
            "Failed to map NvKmsKapiMemory 0x%p",
            pMemory);
-        return -ENOMEM;
+        ret = -ENOMEM;
+        goto done;
    }

    nv_nvkms_memory->pWriteCombinedIORemapAddress = ioremap_wc(
            (uintptr_t)nv_nvkms_memory->pPhysicalAddress,
-            size);
+            nv_nvkms_memory->base.base.size);

    if (!nv_nvkms_memory->pWriteCombinedIORemapAddress) {
        NV_DRM_DEV_LOG_INFO(
@@ -164,7 +196,49 @@ static int __nv_drm_gem_nvkms_map(

    nv_nvkms_memory->physically_mapped = true;

-    return 0;
+done:
+    mutex_unlock(&nv_nvkms_memory->map_lock);
+    return ret;
+}
+
+static void *__nv_drm_gem_nvkms_prime_vmap(
+    struct nv_drm_gem_object *nv_gem)
+{
+    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory =
+        to_nv_nvkms_memory(nv_gem);
+
+    int ret = __nv_drm_gem_nvkms_map(nv_nvkms_memory);
+    if (ret) {
+       return ERR_PTR(ret);
+    }
+
+    if (nv_nvkms_memory->physically_mapped) {
+        return nv_nvkms_memory->pWriteCombinedIORemapAddress;
+    }
+
+    /*
+     * If this buffer isn't physically mapped, it might be backed by struct
+     * pages. Use vmap in that case.
+     */
+    if (nv_nvkms_memory->pages_count > 0) {
+         return nv_drm_vmap(nv_nvkms_memory->pages,
+                            nv_nvkms_memory->pages_count);
+    }
+
+    return ERR_PTR(-ENOMEM);
+}
+
+static void __nv_drm_gem_nvkms_prime_vunmap(
+    struct nv_drm_gem_object *nv_gem,
+    void *address)
+{
+    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory =
+        to_nv_nvkms_memory(nv_gem);
+
+    if (!nv_nvkms_memory->physically_mapped &&
+        nv_nvkms_memory->pages_count > 0) {
+        nv_drm_vunmap(address);
+    }
 }

 static int __nv_drm_gem_map_nvkms_memory_offset(
@@ -172,20 +246,7 @@ static int __nv_drm_gem_map_nvkms_memory_offset(
    struct nv_drm_gem_object *nv_gem,
    uint64_t *offset)
 {
-    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory =
-        to_nv_nvkms_memory(nv_gem);
-
-    if (!nv_nvkms_memory->physically_mapped) {
-        int ret = __nv_drm_gem_nvkms_map(nv_dev,
-                                         nv_nvkms_memory->base.pMemory,
-                                         nv_nvkms_memory,
-                                         nv_nvkms_memory->base.base.size);
-        if (ret) {
-           return ret;
-        }
-    }
-
-    return nv_drm_gem_create_mmap_offset(&nv_nvkms_memory->base, offset);
+    return nv_drm_gem_create_mmap_offset(nv_gem, offset);
 }

 static struct sg_table *__nv_drm_gem_nvkms_memory_prime_get_sg_table(
@@ -197,11 +258,11 @@ static struct sg_table *__nv_drm_gem_nvkms_memory_prime_get_sg_table(
    struct sg_table *sg_table;

    if (nv_nvkms_memory->pages_count == 0) {
-        NV_DRM_DEV_LOG_ERR(
+        NV_DRM_DEV_DEBUG_DRIVER(
                nv_dev,
                "Cannot create sg_table for NvKmsKapiMemory 0x%p",
                nv_gem->pMemory);
-        return NULL;
+        return ERR_PTR(-ENOMEM);
    }

    sg_table = nv_drm_prime_pages_to_sg(nv_dev->dev,
@@ -214,6 +275,8 @@ static struct sg_table *__nv_drm_gem_nvkms_memory_prime_get_sg_table(
 const struct nv_drm_gem_object_funcs nv_gem_nvkms_memory_ops = {
    .free = __nv_drm_gem_nvkms_memory_free,
    .prime_dup = __nv_drm_gem_nvkms_prime_dup,
+    .prime_vmap = __nv_drm_gem_nvkms_prime_vmap,
+    .prime_vunmap = __nv_drm_gem_nvkms_prime_vunmap,
    .mmap = __nv_drm_gem_nvkms_mmap,
    .handle_vma_fault = __nv_drm_gem_nvkms_handle_vma_fault,
    .create_mmap_offset = __nv_drm_gem_map_nvkms_memory_offset,
@@ -229,6 +292,16 @@ static int __nv_drm_nvkms_gem_obj_init(
    NvU64 *pages = NULL;
    NvU32 numPages = 0;

+    if ((size % PAGE_SIZE) != 0) {
+        NV_DRM_DEV_LOG_ERR(
+            nv_dev,
+            "NvKmsKapiMemory 0x%p size should be in a multiple of page size to "
+            "create a gem object",
+            pMemory);
+        return -EINVAL;
+    }
+
+    mutex_init(&nv_nvkms_memory->map_lock);
    nv_nvkms_memory->pPhysicalAddress = NULL;
    nv_nvkms_memory->pWriteCombinedIORemapAddress = NULL;
    nv_nvkms_memory->physically_mapped = false;
@@ -285,11 +358,13 @@ int nv_drm_dumb_create(
    if (nv_dev->hasVideoMemory) {
        pMemory = nvKms->allocateVideoMemory(nv_dev->pDevice,
                                             NvKmsSurfaceMemoryLayoutPitch,
+                                             NVKMS_KAPI_ALLOCATION_TYPE_SCANOUT,
                                             args->size,
                                             &compressible);
    } else {
        pMemory = nvKms->allocateSystemMemory(nv_dev->pDevice,
                                              NvKmsSurfaceMemoryLayoutPitch,
+                                              NVKMS_KAPI_ALLOCATION_TYPE_SCANOUT,
                                              args->size,
                                              &compressible);
    }
@@ -298,7 +373,7 @@ int nv_drm_dumb_create(
        ret = -ENOMEM;
        NV_DRM_DEV_LOG_ERR(
            nv_dev,
-            "Failed to allocate NvKmsKapiMemory for dumb object of size %llu",
+            "Failed to allocate NvKmsKapiMemory for dumb object of size %" NvU64_fmtu,
            args->size);
        goto nvkms_alloc_memory_failed;
    }
@@ -312,7 +387,7 @@ int nv_drm_dumb_create(
     * to use dumb buffers for software rendering, so they're not much use
     * without a CPU mapping.
     */
-    ret = __nv_drm_gem_nvkms_map(nv_dev, pMemory, nv_nvkms_memory, args->size);
+    ret = __nv_drm_gem_nvkms_map(nv_nvkms_memory);
    if (ret) {
        nv_drm_gem_object_unreference_unlocked(&nv_nvkms_memory->base);
        goto fail;
@@ -342,7 +417,7 @@ int nv_drm_gem_import_nvkms_memory_ioctl(struct drm_device *dev,
    int ret;

    if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
-        ret = -EINVAL;
+        ret = -EOPNOTSUPP;
        goto failed;
    }

@@ -392,7 +467,7 @@ int nv_drm_gem_export_nvkms_memory_ioctl(struct drm_device *dev,
    int ret = 0;

    if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
-        ret = -EINVAL;
+        ret = -EOPNOTSUPP;
        goto done;
    }

@@ -441,14 +516,16 @@ int nv_drm_gem_alloc_nvkms_memory_ioctl(struct drm_device *dev,
    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory = NULL;
    struct NvKmsKapiMemory *pMemory;
    enum NvKmsSurfaceMemoryLayout layout;
+    enum NvKmsKapiAllocationType type;
    int ret = 0;

    if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
-        ret = -EINVAL;
+        ret = -EOPNOTSUPP;
        goto failed;
    }

-    if (p->__pad != 0) {
+    if ((p->__pad0 != 0) || (p->__pad1 != 0)) {
+        ret = -EINVAL;
        NV_DRM_DEV_LOG_ERR(nv_dev, "non-zero value in padding field");
        goto failed;
    }
@@ -461,15 +538,19 @@ int nv_drm_gem_alloc_nvkms_memory_ioctl(struct drm_device *dev,

    layout = p->block_linear ?
        NvKmsSurfaceMemoryLayoutBlockLinear : NvKmsSurfaceMemoryLayoutPitch;
+    type = (p->flags & NV_GEM_ALLOC_NO_SCANOUT) ?
+        NVKMS_KAPI_ALLOCATION_TYPE_OFFSCREEN : NVKMS_KAPI_ALLOCATION_TYPE_SCANOUT;

    if (nv_dev->hasVideoMemory) {
        pMemory = nvKms->allocateVideoMemory(nv_dev->pDevice,
                                             layout,
+                                             type,
                                             p->memory_size,
                                             &p->compressible);
    } else {
        pMemory = nvKms->allocateSystemMemory(nv_dev->pDevice,
                                              layout,
+                                              type,
                                              p->memory_size,
                                              &p->compressible);
    }
@@ -507,14 +588,12 @@ static struct drm_gem_object *__nv_drm_gem_nvkms_prime_dup(
 {
    struct nv_drm_device *nv_dev = to_nv_device(dev);
    const struct nv_drm_device *nv_dev_src;
-    const struct nv_drm_gem_nvkms_memory *nv_nvkms_memory_src;
    struct nv_drm_gem_nvkms_memory *nv_nvkms_memory;
    struct NvKmsKapiMemory *pMemory;

    BUG_ON(nv_gem_src == NULL || nv_gem_src->ops != &nv_gem_nvkms_memory_ops);

    nv_dev_src = to_nv_device(nv_gem_src->base.dev);
-    nv_nvkms_memory_src = to_nv_nvkms_memory_const(nv_gem_src);

    if ((nv_nvkms_memory =
            nv_drm_calloc(1, sizeof(*nv_nvkms_memory))) == NULL) {
@@ -575,11 +654,13 @@ int nv_drm_dumb_map_offset(struct drm_file *file,
    return ret;
 }

+#if defined(NV_DRM_DRIVER_HAS_DUMB_DESTROY)
 int nv_drm_dumb_destroy(struct drm_file *file,
                        struct drm_device *dev,
                        uint32_t handle)
 {
    return drm_gem_handle_delete(file, handle);
 }
+#endif /* NV_DRM_DRIVER_HAS_DUMB_DESTROY */

 #endif
--- a/kernel-open/nvidia-drm/nvidia-drm-gem-nvkms-memory.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem-nvkms-memory.h
@@ -32,8 +32,15 @@
 struct nv_drm_gem_nvkms_memory {
    struct nv_drm_gem_object base;

+    /*
+     * Lock to protect concurrent writes to physically_mapped, pPhysicalAddress,
+     * and pWriteCombinedIORemapAddress.
+     *
+     * __nv_drm_gem_nvkms_map(), the sole writer, is structured such that
+     * readers are not required to hold the lock.
+     */
+    struct mutex map_lock;
    bool physically_mapped;
-
    void *pPhysicalAddress;
    void *pWriteCombinedIORemapAddress;

@@ -97,9 +104,11 @@ int nv_drm_dumb_map_offset(struct drm_file *file,
                           struct drm_device *dev, uint32_t handle,
                           uint64_t *offset);

+#if defined(NV_DRM_DRIVER_HAS_DUMB_DESTROY)
 int nv_drm_dumb_destroy(struct drm_file *file,
                        struct drm_device *dev,
                        uint32_t handle);
+#endif /* NV_DRM_DRIVER_HAS_DUMB_DESTROY */

 struct drm_gem_object *nv_drm_gem_nvkms_prime_import(
    struct drm_device *dev,
--- a/kernel-open/nvidia-drm/nvidia-drm-gem-user-memory.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem-user-memory.c
@@ -36,6 +36,14 @@
 #include "linux/mm.h"
 #include "nv-mm.h"

+#if defined(NV_LINUX_PFN_T_H_PRESENT)
+#include "linux/pfn_t.h"
+#endif
+
+#if defined(NV_BSD)
+#include <vm/vm_pageout.h>
+#endif
+
 static inline
 void __nv_drm_gem_user_memory_free(struct nv_drm_gem_object *nv_gem)
 {
@@ -92,13 +100,44 @@ static int __nv_drm_gem_user_memory_mmap(struct nv_drm_gem_object *nv_gem,
        return -EINVAL;
    }

-    vma->vm_flags &= ~VM_PFNMAP;
-    vma->vm_flags &= ~VM_IO;
-    vma->vm_flags |= VM_MIXEDMAP;
+    nv_vm_flags_clear(vma, VM_PFNMAP);
+    nv_vm_flags_clear(vma, VM_IO);
+    nv_vm_flags_set(vma, VM_MIXEDMAP);

    return 0;
 }

+#if defined(NV_LINUX) && !defined(NV_VMF_INSERT_MIXED_PRESENT)
+static vm_fault_t __nv_vm_insert_mixed_helper(
+    struct vm_area_struct *vma,
+    unsigned long address,
+    unsigned long pfn)
+{
+    int ret;
+
+#if defined(NV_PFN_TO_PFN_T_PRESENT)
+    ret = vm_insert_mixed(vma, address, pfn_to_pfn_t(pfn));
+#else
+    ret = vm_insert_mixed(vma, address, pfn);
+#endif
+
+    switch (ret) {
+        case 0:
+        case -EBUSY:
+            /*
+             * EBUSY indicates that another thread already handled
+             * the faulted range.
+             */
+            return VM_FAULT_NOPAGE;
+        case -ENOMEM:
+            return VM_FAULT_OOM;
+        default:
+            WARN_ONCE(1, "Unhandled error in %s: %d\n", __FUNCTION__, ret);
+            return VM_FAULT_SIGBUS;
+    }
+}
+#endif
+
 static vm_fault_t __nv_drm_gem_user_memory_handle_vma_fault(
    struct nv_drm_gem_object *nv_gem,
    struct vm_area_struct *vma,
@@ -108,32 +147,19 @@ static vm_fault_t __nv_drm_gem_user_memory_handle_vma_fault(
    unsigned long address = nv_page_fault_va(vmf);
    struct drm_gem_object *gem = vma->vm_private_data;
    unsigned long page_offset;
-    vm_fault_t ret;
+    unsigned long pfn;

    page_offset = vmf->pgoff - drm_vma_node_start(&gem->vma_node);
+    BUG_ON(page_offset >= nv_user_memory->pages_count);
+    pfn = page_to_pfn(nv_user_memory->pages[page_offset]);

-    BUG_ON(page_offset > nv_user_memory->pages_count);
-
-    ret = vm_insert_page(vma, address, nv_user_memory->pages[page_offset]);
-    switch (ret) {
-        case 0:
-        case -EBUSY:
-            /*
-             * EBUSY indicates that another thread already handled
-             * the faulted range.
-             */
-            ret = VM_FAULT_NOPAGE;
-            break;
-        case -ENOMEM:
-            ret = VM_FAULT_OOM;
-            break;
-        default:
-            WARN_ONCE(1, "Unhandled error in %s: %d\n", __FUNCTION__, ret);
-            ret = VM_FAULT_SIGBUS;
-            break;
-    }
-
-    return ret;
+#if !defined(NV_LINUX)
+    return vmf_insert_pfn(vma, address, pfn);
+#elif defined(NV_VMF_INSERT_MIXED_PRESENT)
+    return vmf_insert_mixed(vma, address, pfn_to_pfn_t(pfn));
+#else
+    return __nv_vm_insert_mixed_helper(vma, address, pfn);
+#endif
 }

 static int __nv_drm_gem_user_create_mmap_offset(
@@ -171,7 +197,7 @@ int nv_drm_gem_import_userspace_memory_ioctl(struct drm_device *dev,
    if ((params->size % PAGE_SIZE) != 0) {
        NV_DRM_DEV_LOG_ERR(
            nv_dev,
-            "Userspace memory 0x%llx size should be in a multiple of page "
+            "Userspace memory 0x%" NvU64_fmtx " size should be in a multiple of page "
            "size to create a gem object",
            params->address);
        return -EINVAL;
@@ -184,7 +210,7 @@ int nv_drm_gem_import_userspace_memory_ioctl(struct drm_device *dev,
    if (ret != 0) {
        NV_DRM_DEV_LOG_ERR(
            nv_dev,
-            "Failed to lock user pages for address 0x%llx: %d",
+            "Failed to lock user pages for address 0x%" NvU64_fmtx ": %d",
            params->address, ret);
        return ret;
    }
--- a/kernel-open/nvidia-drm/nvidia-drm-gem.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem.c
@@ -26,7 +26,7 @@

 #include "nvidia-drm-priv.h"
 #include "nvidia-drm-ioctl.h"
-#include "nvidia-drm-prime-fence.h"
+#include "nvidia-drm-fence.h"
 #include "nvidia-drm-gem.h"
 #include "nvidia-drm-gem-nvkms-memory.h"
 #include "nvidia-drm-gem-user-memory.h"
@@ -81,10 +81,13 @@ typedef struct dma_buf_map nv_sysio_map_t;
 static int nv_drm_gem_vmap(struct drm_gem_object *gem,
                           nv_sysio_map_t *map)
 {
-    map->vaddr = nv_drm_gem_prime_vmap(gem);
-    if (map->vaddr == NULL) {
+    void *vaddr = nv_drm_gem_prime_vmap(gem);
+    if (vaddr == NULL) {
        return -ENOMEM;
+    } else if (IS_ERR(vaddr)) {
+        return PTR_ERR(vaddr);
    }
+    map->vaddr = vaddr;
    map->is_iomem = true;
    return 0;
 }
@@ -132,13 +135,8 @@ void nv_drm_gem_object_init(struct nv_drm_device *nv_dev,

    /* Initialize the gem object */

-#if defined(NV_DRM_FENCE_AVAILABLE)
+#if defined(NV_DRM_FENCE_AVAILABLE) && !defined(NV_DRM_GEM_OBJECT_HAS_RESV)
    nv_dma_resv_init(&nv_gem->resv);
-
-#if defined(NV_DRM_GEM_OBJECT_HAS_RESV)
-    nv_gem->base.resv = &nv_gem->resv;
-#endif
-
 #endif

 #if !defined(NV_DRM_DRIVER_HAS_GEM_FREE_OBJECT)
@@ -146,6 +144,12 @@ void nv_drm_gem_object_init(struct nv_drm_device *nv_dev,
 #endif

    drm_gem_private_object_init(dev, &nv_gem->base, size);
+
+    /* Create mmap offset early for drm_gem_prime_mmap(), if possible. */
+    if (nv_gem->ops->create_mmap_offset) {
+        uint64_t offset;
+        nv_gem->ops->create_mmap_offset(nv_dev, nv_gem, &offset);
+    }
 }

 struct drm_gem_object *nv_drm_gem_prime_import(struct drm_device *dev,
@@ -212,8 +216,7 @@ void nv_drm_gem_prime_vunmap(struct drm_gem_object *gem, void *address)
 nv_dma_resv_t* nv_drm_gem_prime_res_obj(struct drm_gem_object *obj)
 {
    struct nv_drm_gem_object *nv_gem = to_nv_gem_object(obj);
-
-    return &nv_gem->resv;
+    return nv_drm_gem_res_obj(nv_gem);
 }
 #endif

@@ -235,6 +238,7 @@ int nv_drm_gem_map_offset_ioctl(struct drm_device *dev,
        return -EINVAL;
    }

+    /* mmap offset creation is idempotent, fetch it by creating it again. */
    if (nv_gem->ops->create_mmap_offset) {
        ret = nv_gem->ops->create_mmap_offset(nv_dev, nv_gem, &params->offset);
    } else {
@@ -299,7 +303,7 @@ int nv_drm_mmap(struct file *file, struct vm_area_struct *vma)
            ret = -EINVAL;
            goto done;
        }
-        vma->vm_flags &= ~VM_MAYWRITE;
+        nv_vm_flags_clear(vma, VM_MAYWRITE);
    }
 #endif

@@ -322,7 +326,7 @@ int nv_drm_gem_identify_object_ioctl(struct drm_device *dev,
    struct nv_drm_gem_object *nv_gem = NULL;

    if (!drm_core_check_feature(dev, DRIVER_MODESET)) {
-        return -EINVAL;
+        return -EOPNOTSUPP;
    }

    nv_dma_buf = nv_drm_gem_object_dma_buf_lookup(dev, filep, p->handle);
--- a/kernel-open/nvidia-drm/nvidia-drm-gem.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-gem.h
@@ -45,6 +45,8 @@
 #include "nvidia-dma-resv-helper.h"
 #endif

+#include "linux/dma-buf.h"
+
 struct nv_drm_gem_object;

 struct nv_drm_gem_object_funcs {
@@ -71,7 +73,7 @@ struct nv_drm_gem_object {

    struct NvKmsKapiMemory *pMemory;

-#if defined(NV_DRM_FENCE_AVAILABLE)
+#if defined(NV_DRM_FENCE_AVAILABLE) && !defined(NV_DRM_GEM_OBJECT_HAS_RESV)
    nv_dma_resv_t  resv;
 #endif
 };
@@ -93,6 +95,16 @@ static inline struct nv_drm_gem_object *to_nv_gem_object(
 * 3e70fd160cf0b1945225eaa08dd2cb8544f21cb8 (2018-11-15).
 */

+static inline void
+nv_drm_gem_object_reference(struct nv_drm_gem_object *nv_gem)
+{
+#if defined(NV_DRM_GEM_OBJECT_GET_PRESENT)
+    drm_gem_object_get(&nv_gem->base);
+#else
+    drm_gem_object_reference(&nv_gem->base);
+#endif
+}
+
 static inline void
 nv_drm_gem_object_unreference_unlocked(struct nv_drm_gem_object *nv_gem)
 {
@@ -177,6 +189,17 @@ static inline int nv_drm_gem_handle_create(struct drm_file *filp,
    return drm_gem_handle_create(filp, &nv_gem->base, handle);
 }

+#if defined(NV_DRM_FENCE_AVAILABLE)
+static inline nv_dma_resv_t *nv_drm_gem_res_obj(struct nv_drm_gem_object *nv_gem)
+{
+#if defined(NV_DRM_GEM_OBJECT_HAS_RESV)
+    return nv_gem->base.resv;
+#else
+    return nv_gem->base.dma_buf ? nv_gem->base.dma_buf->resv : &nv_gem->resv;
+#endif
+}
+#endif
+
 void nv_drm_gem_object_init(struct nv_drm_device *nv_dev,
                            struct nv_drm_gem_object *nv_gem,
                            const struct nv_drm_gem_object_funcs * const ops,
--- a/kernel-open/nvidia-drm/nvidia-drm-helper.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-helper.c
@@ -28,6 +28,8 @@
 */

 #include "nvidia-drm-helper.h"
+#include "nvidia-drm-priv.h"
+#include "nvidia-drm-crtc.h"

 #include "nvmisc.h"

@@ -41,6 +43,18 @@
 #include <drm/drm_atomic_uapi.h>
 #endif

+/*
+ * The inclusion of drm_framebuffer.h was removed from drm_crtc.h by commit
+ * 720cf96d8fec ("drm: Drop drm_framebuffer.h from drm_crtc.h") in v6.0.
+ *
+ * We only need drm_framebuffer.h for drm_framebuffer_put(), and it is always
+ * present (v4.9+) when drm_framebuffer_{put,get}() is present (v4.12+), so it
+ * is safe to unconditionally include it when drm_framebuffer_get() is present.
+ */
+#if defined(NV_DRM_FRAMEBUFFER_GET_PRESENT)
+#include <drm/drm_framebuffer.h>
+#endif
+
 static void __nv_drm_framebuffer_put(struct drm_framebuffer *fb)
 {
 #if defined(NV_DRM_FRAMEBUFFER_GET_PRESENT)
@@ -135,6 +149,18 @@ int nv_drm_atomic_helper_disable_all(struct drm_device *dev,
            goto free;
    }

+#if defined(NV_DRM_ROTATION_AVAILABLE)
+    nv_drm_for_each_plane(plane, dev) {
+        plane_state = drm_atomic_get_plane_state(state, plane);
+        if (IS_ERR(plane_state)) {
+            ret = PTR_ERR(plane_state);
+            goto free;
+        }
+
+        plane_state->rotation = DRM_MODE_ROTATE_0;
+    }
+#endif
+
    nv_drm_for_each_connector_in_state(state, conn, conn_state, i) {
        ret = drm_atomic_set_crtc_for_connector(conn_state, NULL);
        if (ret < 0)
--- a/kernel-open/nvidia-drm/nvidia-drm-helper.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-helper.h
@@ -35,6 +35,40 @@
 #include <drm/drm_drv.h>
 #endif

+#if defined(NV_DRM_ALPHA_BLENDING_AVAILABLE) || defined(NV_DRM_ROTATION_AVAILABLE)
+/* For DRM_ROTATE_* , DRM_REFLECT_* */
+#include <drm/drm_blend.h>
+#endif
+
+#if defined(NV_DRM_ROTATION_AVAILABLE) || \
+    defined(NV_DRM_COLOR_CTM_3X4_PRESENT) || \
+    defined(NV_DRM_COLOR_LUT_PRESENT)
+/*
+ * For DRM_MODE_ROTATE_*, DRM_MODE_REFLECT_*, struct drm_color_ctm_3x4, and
+ * struct drm_color_lut.
+ */
+#include <uapi/drm/drm_mode.h>
+#endif
+
+#if defined(NV_DRM_ROTATION_AVAILABLE)
+/*
+ * 19-05-2017 c2c446ad29437bb92b157423c632286608ebd3ec has added
+ * DRM_MODE_ROTATE_* and DRM_MODE_REFLECT_* to UAPI and removed
+ * DRM_ROTATE_* and DRM_REFLECT_*
+ */
+#if !defined(DRM_MODE_ROTATE_0)
+#define DRM_MODE_ROTATE_0       DRM_ROTATE_0
+#define DRM_MODE_ROTATE_90      DRM_ROTATE_90
+#define DRM_MODE_ROTATE_180     DRM_ROTATE_180
+#define DRM_MODE_ROTATE_270     DRM_ROTATE_270
+#define DRM_MODE_REFLECT_X      DRM_REFLECT_X
+#define DRM_MODE_REFLECT_Y      DRM_REFLECT_Y
+#define DRM_MODE_ROTATE_MASK    DRM_ROTATE_MASK
+#define DRM_MODE_REFLECT_MASK   DRM_REFLECT_MASK
+#endif
+
+#endif //NV_DRM_ROTATION_AVAILABLE
+
 /*
 * drm_dev_put() is added by commit 9a96f55034e41b4e002b767e9218d55f03bdff7d
 * (2017-09-26) and drm_dev_unref() is removed by
@@ -277,11 +311,81 @@ int nv_drm_atomic_helper_disable_all(struct drm_device *dev,
    for_each_plane_in_state(__state, plane, plane_state, __i)
 #endif

-static inline struct drm_crtc *nv_drm_crtc_find(struct drm_device *dev,
-    uint32_t id)
+/*
+ * for_each_new_plane_in_state() was added by kernel commit
+ * 581e49fe6b411f407102a7f2377648849e0fa37f which was Signed-off-by:
+ *      Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
+ *      Daniel Vetter <daniel.vetter@ffwll.ch>
+ *
+ * This commit also added the old_state and new_state pointers to
+ * __drm_planes_state. Because of this, the best that can be done on kernel
+ * versions without this macro is for_each_plane_in_state.
+ */
+
+/**
+ * nv_drm_for_each_new_plane_in_state - iterate over all planes in an atomic update
+ * @__state: &struct drm_atomic_state pointer
+ * @plane: &struct drm_plane iteration cursor
+ * @new_plane_state: &struct drm_plane_state iteration cursor for the new state
+ * @__i: int iteration cursor, for macro-internal use
+ *
+ * This iterates over all planes in an atomic update, tracking only the new
+ * state. This is useful in enable functions, where we need the new state the
+ * hardware should be in when the atomic commit operation has completed.
+ */
+#if !defined(for_each_new_plane_in_state)
+#define nv_drm_for_each_new_plane_in_state(__state, plane, new_plane_state, __i) \
+    nv_drm_for_each_plane_in_state(__state, plane, new_plane_state, __i)
+#else
+#define nv_drm_for_each_new_plane_in_state(__state, plane, new_plane_state, __i) \
+    for_each_new_plane_in_state(__state, plane, new_plane_state, __i)
+#endif
+
+static inline struct drm_connector *
+nv_drm_connector_lookup(struct drm_device *dev, struct drm_file *filep,
+                        uint32_t id)
+{
+#if !defined(NV_DRM_CONNECTOR_LOOKUP_PRESENT)
+    return drm_connector_find(dev, id);
+#elif defined(NV_DRM_MODE_OBJECT_FIND_HAS_FILE_PRIV_ARG)
+    return drm_connector_lookup(dev, filep, id);
+#else
+    return drm_connector_lookup(dev, id);
+#endif
+}
+
+static inline void nv_drm_connector_put(struct drm_connector *connector)
+{
+#if defined(NV_DRM_CONNECTOR_PUT_PRESENT)
+    drm_connector_put(connector);
+#elif defined(NV_DRM_CONNECTOR_LOOKUP_PRESENT)
+    drm_connector_unreference(connector);
+#endif
+}
+
+static inline void nv_drm_property_blob_put(struct drm_property_blob *blob)
+{
+#if defined(NV_DRM_PROPERTY_BLOB_PUT_PRESENT)
+    drm_property_blob_put(blob);
+#else
+    drm_property_unreference_blob(blob);
+#endif
+}
+
+static inline void nv_drm_property_blob_get(struct drm_property_blob *blob)
+{
+#if defined(NV_DRM_PROPERTY_BLOB_PUT_PRESENT)
+    drm_property_blob_get(blob);
+#else
+    drm_property_reference_blob(blob);
+#endif
+}
+
+static inline struct drm_crtc *
+nv_drm_crtc_find(struct drm_device *dev, struct drm_file *filep, uint32_t id)
 {
 #if defined(NV_DRM_MODE_OBJECT_FIND_HAS_FILE_PRIV_ARG)
-    return drm_crtc_find(dev, NULL /* file_priv */, id);
+    return drm_crtc_find(dev, filep, id);
 #else
    return drm_crtc_find(dev, id);
 #endif
@@ -297,6 +401,30 @@ static inline struct drm_encoder *nv_drm_encoder_find(struct drm_device *dev,
 #endif
 }

+#if defined(NV_DRM_DRM_AUTH_H_PRESENT)
+#include <drm/drm_auth.h>
+#endif
+#if defined(NV_DRM_DRM_FILE_H_PRESENT)
+#include <drm/drm_file.h>
+#endif
+
+/*
+ * drm_file_get_master() added by commit 56f0729a510f ("drm: protect drm_master
+ * pointers in drm_lease.c") in v5.15 (2021-07-20)
+ */
+static inline struct drm_master *nv_drm_file_get_master(struct drm_file *filep)
+{
+#if defined(NV_DRM_FILE_GET_MASTER_PRESENT)
+    return drm_file_get_master(filep);
+#else
+    if (filep->master) {
+        return drm_master_get(filep->master);
+    } else {
+        return NULL;
+    }
+#endif
+}
+
 /*
 * drm_connector_for_each_possible_encoder() is added by commit
 * 83aefbb887b59df0b3520965c3701e01deacfc52 which was Signed-off-by:
@@ -507,6 +635,44 @@ static inline int nv_drm_format_num_planes(uint32_t format)

 #endif /* defined(NV_DRM_FORMAT_MODIFIERS_PRESENT) */

+/*
+ * DRM_UNLOCKED was removed with commit 2798ffcc1d6a ("drm: Remove locking for
+ * legacy ioctls and DRM_UNLOCKED") in v6.8, but it was previously made
+ * implicit for all non-legacy DRM driver IOCTLs since Linux v4.10 commit
+ * fa5386459f06 "drm: Used DRM_LEGACY for all legacy functions" (Linux v4.4
+ * commit ea487835e887 "drm: Enforce unlocked ioctl operation for kms driver
+ * ioctls" previously did it only for drivers that set the DRM_MODESET flag), so
+ * it was effectively a no-op anyway.
+ */
+#if !defined(NV_DRM_UNLOCKED_IOCTL_FLAG_PRESENT)
+#define DRM_UNLOCKED 0
+#endif
+
+/*
+ * struct drm_color_ctm_3x4 was added by commit 6872a189be50 ("drm/amd/display:
+ * Add 3x4 CTM support for plane CTM") in v6.8. For backwards compatibility,
+ * define it when not present.
+ */
+#if !defined(NV_DRM_COLOR_CTM_3X4_PRESENT)
+struct drm_color_ctm_3x4 {
+    __u64 matrix[12];
+};
+#endif
+
+/*
+ * struct drm_color_lut was added by commit 5488dc16fde7 ("drm: introduce pipe
+ * color correction properties") in v4.6. For backwards compatibility, define it
+ * when not present.
+ */
+#if !defined(NV_DRM_COLOR_LUT_PRESENT)
+struct drm_color_lut {
+    __u16 red;
+    __u16 green;
+    __u16 blue;
+    __u16 reserved;
+};
+#endif
+
 /*
 * drm_vma_offset_exact_lookup_locked() were added
 * by kernel commit 2225cfe46bcc which was Signed-off-by:
--- a/kernel-open/nvidia-drm/nvidia-drm-ioctl.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-ioctl.h
@@ -34,8 +34,8 @@
 #define DRM_NVIDIA_GEM_IMPORT_USERSPACE_MEMORY      0x02
 #define DRM_NVIDIA_GET_DEV_INFO                     0x03
 #define DRM_NVIDIA_FENCE_SUPPORTED                  0x04
-#define DRM_NVIDIA_FENCE_CONTEXT_CREATE             0x05
-#define DRM_NVIDIA_GEM_FENCE_ATTACH                 0x06
+#define DRM_NVIDIA_PRIME_FENCE_CONTEXT_CREATE       0x05
+#define DRM_NVIDIA_GEM_PRIME_FENCE_ATTACH           0x06
 #define DRM_NVIDIA_GET_CLIENT_CAPABILITY            0x08
 #define DRM_NVIDIA_GEM_EXPORT_NVKMS_MEMORY          0x09
 #define DRM_NVIDIA_GEM_MAP_OFFSET                   0x0a
@@ -43,6 +43,16 @@
 #define DRM_NVIDIA_GET_CRTC_CRC32_V2                0x0c
 #define DRM_NVIDIA_GEM_EXPORT_DMABUF_MEMORY         0x0d
 #define DRM_NVIDIA_GEM_IDENTIFY_OBJECT              0x0e
+#define DRM_NVIDIA_DMABUF_SUPPORTED                 0x0f
+#define DRM_NVIDIA_GET_DPY_ID_FOR_CONNECTOR_ID      0x10
+#define DRM_NVIDIA_GET_CONNECTOR_ID_FOR_DPY_ID      0x11
+#define DRM_NVIDIA_GRANT_PERMISSIONS                0x12
+#define DRM_NVIDIA_REVOKE_PERMISSIONS               0x13
+#define DRM_NVIDIA_SEMSURF_FENCE_CTX_CREATE         0x14
+#define DRM_NVIDIA_SEMSURF_FENCE_CREATE             0x15
+#define DRM_NVIDIA_SEMSURF_FENCE_WAIT               0x16
+#define DRM_NVIDIA_SEMSURF_FENCE_ATTACH             0x17
+#define DRM_NVIDIA_GET_DRM_FILE_UNIQUE_ID           0x18

 #define DRM_IOCTL_NVIDIA_GEM_IMPORT_NVKMS_MEMORY                           \
    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_IMPORT_NVKMS_MEMORY),      \
@@ -62,53 +72,97 @@
 *
 * 'warning: suggest parentheses around arithmetic in operand of |'
 */
-#if defined(NV_LINUX)
+#if defined(NV_LINUX) || defined(NV_BSD)
 #define DRM_IOCTL_NVIDIA_FENCE_SUPPORTED                         \
    DRM_IO(DRM_COMMAND_BASE + DRM_NVIDIA_FENCE_SUPPORTED)
+#define DRM_IOCTL_NVIDIA_DMABUF_SUPPORTED                        \
+    DRM_IO(DRM_COMMAND_BASE + DRM_NVIDIA_DMABUF_SUPPORTED)
 #else
 #define DRM_IOCTL_NVIDIA_FENCE_SUPPORTED 0
+#define DRM_IOCTL_NVIDIA_DMABUF_SUPPORTED 0
 #endif

-#define DRM_IOCTL_NVIDIA_FENCE_CONTEXT_CREATE                        \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_FENCE_CONTEXT_CREATE),   \
-             struct drm_nvidia_fence_context_create_params)
+#define DRM_IOCTL_NVIDIA_PRIME_FENCE_CONTEXT_CREATE                     \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_PRIME_FENCE_CONTEXT_CREATE),\
+             struct drm_nvidia_prime_fence_context_create_params)

-#define DRM_IOCTL_NVIDIA_GEM_FENCE_ATTACH                            \
-    DRM_IOW((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_FENCE_ATTACH),        \
-            struct drm_nvidia_gem_fence_attach_params)
+#define DRM_IOCTL_NVIDIA_GEM_PRIME_FENCE_ATTACH                         \
+    DRM_IOW((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_PRIME_FENCE_ATTACH),     \
+            struct drm_nvidia_gem_prime_fence_attach_params)

-#define DRM_IOCTL_NVIDIA_GET_CLIENT_CAPABILITY                       \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CLIENT_CAPABILITY),  \
+#define DRM_IOCTL_NVIDIA_GET_CLIENT_CAPABILITY                          \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CLIENT_CAPABILITY),     \
             struct drm_nvidia_get_client_capability_params)

-#define DRM_IOCTL_NVIDIA_GET_CRTC_CRC32                              \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CRTC_CRC32),         \
+#define DRM_IOCTL_NVIDIA_GET_CRTC_CRC32                                 \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CRTC_CRC32),            \
             struct drm_nvidia_get_crtc_crc32_params)

-#define DRM_IOCTL_NVIDIA_GET_CRTC_CRC32_V2                           \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CRTC_CRC32_V2),      \
+#define DRM_IOCTL_NVIDIA_GET_CRTC_CRC32_V2                              \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CRTC_CRC32_V2),         \
              struct drm_nvidia_get_crtc_crc32_v2_params)

-#define DRM_IOCTL_NVIDIA_GEM_EXPORT_NVKMS_MEMORY                           \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_EXPORT_NVKMS_MEMORY),      \
+#define DRM_IOCTL_NVIDIA_GEM_EXPORT_NVKMS_MEMORY                        \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_EXPORT_NVKMS_MEMORY),   \
              struct drm_nvidia_gem_export_nvkms_memory_params)

-#define DRM_IOCTL_NVIDIA_GEM_MAP_OFFSET                              \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_MAP_OFFSET),         \
+#define DRM_IOCTL_NVIDIA_GEM_MAP_OFFSET                                 \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_MAP_OFFSET),            \
             struct drm_nvidia_gem_map_offset_params)

-#define DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY                      \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_ALLOC_NVKMS_MEMORY), \
+#define DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY                         \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_ALLOC_NVKMS_MEMORY),    \
              struct drm_nvidia_gem_alloc_nvkms_memory_params)

-#define DRM_IOCTL_NVIDIA_GEM_EXPORT_DMABUF_MEMORY                      \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_EXPORT_DMABUF_MEMORY), \
+#define DRM_IOCTL_NVIDIA_GEM_EXPORT_DMABUF_MEMORY                       \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_EXPORT_DMABUF_MEMORY),  \
              struct drm_nvidia_gem_export_dmabuf_memory_params)

-#define DRM_IOCTL_NVIDIA_GEM_IDENTIFY_OBJECT                      \
-    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_IDENTIFY_OBJECT), \
+#define DRM_IOCTL_NVIDIA_GEM_IDENTIFY_OBJECT                            \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GEM_IDENTIFY_OBJECT),       \
              struct drm_nvidia_gem_identify_object_params)

+#define DRM_IOCTL_NVIDIA_GET_DPY_ID_FOR_CONNECTOR_ID                     \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_DPY_ID_FOR_CONNECTOR_ID),\
+             struct drm_nvidia_get_dpy_id_for_connector_id_params)
+
+#define DRM_IOCTL_NVIDIA_GET_CONNECTOR_ID_FOR_DPY_ID                     \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GET_CONNECTOR_ID_FOR_DPY_ID),\
+             struct drm_nvidia_get_connector_id_for_dpy_id_params)
+
+#define DRM_IOCTL_NVIDIA_GRANT_PERMISSIONS                              \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_GRANT_PERMISSIONS),         \
+             struct drm_nvidia_grant_permissions_params)
+
+#define DRM_IOCTL_NVIDIA_REVOKE_PERMISSIONS                             \
+    DRM_IOWR((DRM_COMMAND_BASE + DRM_NVIDIA_REVOKE_PERMISSIONS),        \
+             struct drm_nvidia_revoke_permissions_params)
+
+#define DRM_IOCTL_NVIDIA_SEMSURF_FENCE_CTX_CREATE                       \
+    DRM_IOWR((DRM_COMMAND_BASE +                                        \
+              DRM_NVIDIA_SEMSURF_FENCE_CTX_CREATE),                     \
+              struct drm_nvidia_semsurf_fence_ctx_create_params)
+
+#define DRM_IOCTL_NVIDIA_SEMSURF_FENCE_CREATE                           \
+    DRM_IOWR((DRM_COMMAND_BASE +                                        \
+              DRM_NVIDIA_SEMSURF_FENCE_CREATE),                         \
+              struct drm_nvidia_semsurf_fence_create_params)
+
+#define DRM_IOCTL_NVIDIA_SEMSURF_FENCE_WAIT                             \
+    DRM_IOW((DRM_COMMAND_BASE +                                         \
+              DRM_NVIDIA_SEMSURF_FENCE_WAIT),                           \
+              struct drm_nvidia_semsurf_fence_wait_params)
+
+#define DRM_IOCTL_NVIDIA_SEMSURF_FENCE_ATTACH                           \
+    DRM_IOW((DRM_COMMAND_BASE +                                         \
+              DRM_NVIDIA_SEMSURF_FENCE_ATTACH),                         \
+              struct drm_nvidia_semsurf_fence_attach_params)
+
+#define DRM_IOCTL_NVIDIA_GET_DRM_FILE_UNIQUE_ID                         \
+    DRM_IOWR((DRM_COMMAND_BASE +                                        \
+              DRM_NVIDIA_GET_DRM_FILE_UNIQUE_ID),                       \
+              struct drm_nvidia_get_drm_file_unique_id_params)
+
 struct drm_nvidia_gem_import_nvkms_memory_params {
    uint64_t mem_size;           /* IN */

@@ -130,13 +184,18 @@ struct drm_nvidia_get_dev_info_params {
    uint32_t gpu_id;             /* OUT */
    uint32_t primary_index;      /* OUT; the "card%d" value */

-    /* See DRM_FORMAT_MOD_NVIDIA_BLOCK_LINEAR_2D definitions of these */
+    uint32_t supports_alloc;     /* OUT */
+    /* The generic_page_kind, page_kind_generation, and sector_layout
+     * fields are only valid if supports_alloc is true.
+     * See DRM_FORMAT_MOD_NVIDIA_BLOCK_LINEAR_2D definitions of these. */
    uint32_t generic_page_kind;    /* OUT */
    uint32_t page_kind_generation; /* OUT */
    uint32_t sector_layout;        /* OUT */
+    uint32_t supports_sync_fd;     /* OUT */
+    uint32_t supports_semsurf;     /* OUT */
 };

-struct drm_nvidia_fence_context_create_params {
+struct drm_nvidia_prime_fence_context_create_params {
    uint32_t handle;            /* OUT GEM handle to fence context */

    uint32_t index;             /* IN Index of semaphore to use for fencing */
@@ -151,10 +210,11 @@ struct drm_nvidia_fence_context_create_params {
    uint64_t event_nvkms_params_size; /* IN */
 };

-struct drm_nvidia_gem_fence_attach_params {
+struct drm_nvidia_gem_prime_fence_attach_params {
    uint32_t handle;                /* IN GEM handle to attach fence to */
    uint32_t fence_context_handle;  /* IN GEM handle to fence context on which fence is run on */
    uint32_t sem_thresh;            /* IN Semaphore value to reach before signal */
+    uint32_t __pad;
 };

 struct drm_nvidia_get_client_capability_params {
@@ -166,6 +226,8 @@ struct drm_nvidia_get_client_capability_params {
 struct drm_nvidia_crtc_crc32 {
    uint32_t value; /* Read value, undefined if supported is false */
    uint8_t supported; /* Supported boolean, true if readable by hardware */
+    uint8_t __pad0;
+    uint16_t __pad1;
 };

 struct drm_nvidia_crtc_crc32_v2_out {
@@ -199,13 +261,17 @@ struct drm_nvidia_gem_map_offset_params {
    uint64_t offset;             /* OUT Fake offset */
 };

+#define NV_GEM_ALLOC_NO_SCANOUT                     (1 << 0)
+
 struct drm_nvidia_gem_alloc_nvkms_memory_params {
    uint32_t handle;              /* OUT */
    uint8_t  block_linear;        /* IN */
    uint8_t  compressible;        /* IN/OUT */
-    uint16_t __pad;
+    uint16_t __pad0;

    uint64_t memory_size;         /* IN */
+    uint32_t flags;               /* IN */
+    uint32_t __pad1;
 };

 struct drm_nvidia_gem_export_dmabuf_memory_params {
@@ -229,4 +295,104 @@ struct drm_nvidia_gem_identify_object_params {
    drm_nvidia_gem_object_type  object_type;    /* OUT GEM object type */
 };

+struct drm_nvidia_get_dpy_id_for_connector_id_params {
+    uint32_t connectorId; /* IN */
+    uint32_t dpyId;       /* OUT */
+};
+
+struct drm_nvidia_get_connector_id_for_dpy_id_params {
+    uint32_t dpyId;       /* IN */
+    uint32_t connectorId; /* OUT */
+};
+
+enum drm_nvidia_permissions_type {
+    NV_DRM_PERMISSIONS_TYPE_MODESET = 2,
+    NV_DRM_PERMISSIONS_TYPE_SUB_OWNER = 3
+};
+
+struct drm_nvidia_grant_permissions_params {
+    int32_t fd;           /* IN */
+    uint32_t dpyId;       /* IN */
+    uint32_t type;        /* IN */
+};
+
+struct drm_nvidia_revoke_permissions_params {
+    uint32_t dpyId;       /* IN */
+    uint32_t type;        /* IN */
+};
+
+struct drm_nvidia_semsurf_fence_ctx_create_params {
+    uint64_t index;             /* IN Index of the desired semaphore in the
+                                 * fence context's semaphore surface */
+
+    /* Params for importing userspace semaphore surface */
+    uint64_t nvkms_params_ptr;  /* IN */
+    uint64_t nvkms_params_size; /* IN */
+
+    uint32_t handle;            /* OUT GEM handle to fence context */
+    uint32_t __pad;
+};
+
+struct drm_nvidia_semsurf_fence_create_params {
+    uint32_t fence_context_handle;  /* IN GEM handle to fence context on which
+                                     * fence is run on */
+
+    uint32_t timeout_value_ms;      /* IN Timeout value in ms for the fence
+                                     * after which the fence will be signaled
+                                     * with its error status set to -ETIMEDOUT.
+                                     * Default timeout value is 5000ms */
+
+    uint64_t wait_value;            /* IN Semaphore value to reach before signal */
+
+    int32_t  fd;                    /* OUT sync FD object representing the
+                                     * semaphore at the specified index reaching
+                                     * a value >= wait_value */
+    uint32_t __pad;
+};
+
+/*
+ * Note there is no provision for timeouts in this ioctl. The kernel
+ * documentation asserts timeouts should be handled by fence producers, and
+ * that waiters should not second-guess their logic, as it is producers rather
+ * than consumers that have better information when it comes to determining a
+ * reasonable timeout for a given workload.
+ */
+struct drm_nvidia_semsurf_fence_wait_params {
+    uint32_t fence_context_handle;  /* IN GEM handle to fence context which will
+                                     * be used to wait on the sync FD.  Need not
+                                     * be the fence context used to create the
+                                     * sync FD. */
+
+    int32_t  fd;                    /* IN sync FD object to wait on */
+
+    uint64_t pre_wait_value;        /* IN Wait for the semaphore represented by
+                                     * fence_context to reach this value before
+                                     * waiting for the sync file. */
+
+    uint64_t post_wait_value;       /* IN Signal the semaphore represented by
+                                     * fence_context to this value after waiting
+                                     * for the sync file */
+};
+
+struct drm_nvidia_semsurf_fence_attach_params {
+    uint32_t handle;                /* IN GEM handle of buffer */
+
+    uint32_t fence_context_handle;  /* IN GEM handle of fence context */
+
+    uint32_t timeout_value_ms;      /* IN Timeout value in ms for the fence
+                                     * after which the fence will be signaled
+                                     * with its error status set to -ETIMEDOUT.
+                                     * Default timeout value is 5000ms */
+
+    uint32_t shared;                /* IN If true, fence will reserve shared
+                                     * access to the buffer, otherwise it will
+                                     * reserve exclusive access */
+
+    uint64_t wait_value;            /* IN Semaphore value to reach before signal */
+};
+
+struct drm_nvidia_get_drm_file_unique_id_params {
+    uint64_t id;                    /* OUT Unique ID of the DRM file */
+};
+
 #endif /* _UAPI_NVIDIA_DRM_IOCTL_H_ */
--- a/kernel-open/nvidia-drm/nvidia-drm-linux.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-linux.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
+ * Copyright (c) 2015-2023, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
@@ -21,8 +21,6 @@
 */

 #include <linux/module.h>
-#include <linux/slab.h>
-#include <linux/err.h>

 #include "nvidia-drm-os-interface.h"
 #include "nvidia-drm.h"
@@ -31,130 +29,18 @@

 #if defined(NV_DRM_AVAILABLE)

-#if defined(NV_DRM_DRMP_H_PRESENT)
-#include <drm/drmP.h>
-#endif
-
-#include <linux/vmalloc.h>
-
-#include "nv-mm.h"
-
 MODULE_PARM_DESC(
    modeset,
    "Enable atomic kernel modesetting (1 = enable, 0 = disable (default))");
-bool nv_drm_modeset_module_param = false;
 module_param_named(modeset, nv_drm_modeset_module_param, bool, 0400);

-void *nv_drm_calloc(size_t nmemb, size_t size)
-{
-    return kzalloc(nmemb * size, GFP_KERNEL);
-}
-
-void nv_drm_free(void *ptr)
-{
-    if (IS_ERR(ptr)) {
-        return;
-    }
-
-    kfree(ptr);
-}
-
-char *nv_drm_asprintf(const char *fmt, ...)
-{
-    va_list ap;
-    char *p;
-
-    va_start(ap, fmt);
-    p = kvasprintf(GFP_KERNEL, fmt, ap);
-    va_end(ap);
-
-    return p;
-}
-
-#if defined(NVCPU_X86) || defined(NVCPU_X86_64)
-  #define WRITE_COMBINE_FLUSH()    asm volatile("sfence":::"memory")
-#elif defined(NVCPU_FAMILY_ARM)
-  #if defined(NVCPU_ARM)
-    #define WRITE_COMBINE_FLUSH()  { dsb(); outer_sync(); }
-  #elif defined(NVCPU_AARCH64)
-    #define WRITE_COMBINE_FLUSH()  mb()
-  #endif
-#elif defined(NVCPU_PPC64LE)
-  #define WRITE_COMBINE_FLUSH()    asm volatile("sync":::"memory")
+#if defined(NV_DRM_FBDEV_AVAILABLE)
+MODULE_PARM_DESC(
+    fbdev,
+    "Create a framebuffer device (1 = enable, 0 = disable (default)) (EXPERIMENTAL)");
+module_param_named(fbdev, nv_drm_fbdev_module_param, bool, 0400);
 #endif

-void nv_drm_write_combine_flush(void)
-{
-    WRITE_COMBINE_FLUSH();
-}
-
-int nv_drm_lock_user_pages(unsigned long address,
-                           unsigned long pages_count, struct page ***pages)
-{
-    struct mm_struct *mm = current->mm;
-    struct page **user_pages;
-    const int write = 1;
-    const int force = 0;
-    int pages_pinned;
-
-    user_pages = nv_drm_calloc(pages_count, sizeof(*user_pages));
-
-    if (user_pages == NULL) {
-        return -ENOMEM;
-    }
-
-    nv_mmap_read_lock(mm);
-
-    pages_pinned = NV_GET_USER_PAGES(address, pages_count, write, force,
-                                     user_pages, NULL);
-    nv_mmap_read_unlock(mm);
-
-    if (pages_pinned < 0 || (unsigned)pages_pinned < pages_count) {
-        goto failed;
-    }
-
-    *pages = user_pages;
-
-    return 0;
-
-failed:
-
-    if (pages_pinned > 0) {
-        int i;
-
-        for (i = 0; i < pages_pinned; i++) {
-            put_page(user_pages[i]);
-        }
-    }
-
-    nv_drm_free(user_pages);
-
-    return (pages_pinned < 0) ? pages_pinned : -EINVAL;
-}
-
-void nv_drm_unlock_user_pages(unsigned long  pages_count, struct page **pages)
-{
-    unsigned long i;
-
-    for (i = 0; i < pages_count; i++) {
-        set_page_dirty_lock(pages[i]);
-
-        put_page(pages[i]);
-    }
-
-    nv_drm_free(pages);
-}
-
-void *nv_drm_vmap(struct page **pages, unsigned long pages_count)
-{
-    return vmap(pages, pages_count, VM_USERMAP, PAGE_KERNEL);
-}
-
-void nv_drm_vunmap(void *address)
-{
-    vunmap(address);
-}
-
 #endif /* NV_DRM_AVAILABLE */

 /*************************************************************************
@@ -174,16 +60,7 @@ static void __exit nv_linux_drm_exit(void)
 module_init(nv_linux_drm_init);
 module_exit(nv_linux_drm_exit);

-#if defined(MODULE_LICENSE)
-
  MODULE_LICENSE("Dual MIT/GPL");

-
-
-#endif
-#if defined(MODULE_INFO)
-  MODULE_INFO(supported, "external");
-#endif
-#if defined(MODULE_VERSION)
-  MODULE_VERSION(NV_VERSION_STRING);
-#endif
+MODULE_INFO(supported, "external");
+MODULE_VERSION(NV_VERSION_STRING);
--- a/kernel-open/nvidia-drm/nvidia-drm-modeset.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-modeset.c
@@ -42,6 +42,16 @@
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_crtc.h>

+#if defined(NV_LINUX_NVHOST_H_PRESENT) && defined(CONFIG_TEGRA_GRHOST)
+#include <linux/nvhost.h>
+#elif defined(NV_LINUX_HOST1X_NEXT_H_PRESENT)            
+#include <linux/host1x-next.h>
+#endif
+
+#if defined(NV_DRM_FENCE_AVAILABLE)
+#include "nvidia-dma-fence-helper.h"
+#endif
+
 struct nv_drm_atomic_state {
    struct NvKmsKapiRequestedModeSetConfig config;
    struct drm_atomic_state base;
@@ -93,9 +103,6 @@ static bool __will_generate_flip_event(struct drm_crtc *crtc,
        to_nv_crtc_state(new_crtc_state);
    struct drm_plane_state *old_plane_state = NULL;
    struct drm_plane *plane = NULL;
-    struct drm_plane *primary_plane = crtc->primary;
-    bool primary_event = false;
-    bool overlay_event = false;
    int i;

    if (!old_crtc_state->active  && !new_crtc_state->active) {
@@ -134,17 +141,179 @@ static int __nv_drm_put_back_post_fence_fd(
    const struct NvKmsKapiLayerReplyConfig *layer_reply_config)
 {
    int fd = layer_reply_config->postSyncptFd;
+    int ret = 0;

    if ((fd >= 0) && (plane_state->fd_user_ptr != NULL)) {
-        if (put_user(fd, plane_state->fd_user_ptr)) {
-            return -EFAULT;
+        ret = copy_to_user(plane_state->fd_user_ptr, &fd, sizeof(fd));
+        if (ret != 0) {
+            return ret;
        }

        /*! set back to Null and let set_property specify it again */
        plane_state->fd_user_ptr = NULL;
    }
+
+    return ret;
+}
+
+#if defined(NV_DRM_FENCE_AVAILABLE)
+struct nv_drm_plane_fence_cb_data {
+    nv_dma_fence_cb_t dma_fence_cb;
+    struct nv_drm_device *nv_dev;
+    NvU32 semaphore_index;
+};
+
+static void
+__nv_drm_plane_fence_cb(
+    nv_dma_fence_t *fence,
+    nv_dma_fence_cb_t *cb_data
+)
+{
+    struct nv_drm_plane_fence_cb_data *fence_data =
+        container_of(cb_data, typeof(*fence_data), dma_fence_cb);
+    struct nv_drm_device *nv_dev = fence_data->nv_dev;
+
+    nv_dma_fence_put(fence);
+    nvKms->signalDisplaySemaphore(nv_dev->pDevice, fence_data->semaphore_index);
+    nv_drm_free(fence_data);
+}
+
+static int __nv_drm_convert_in_fences(
+    struct nv_drm_device *nv_dev,
+    struct drm_atomic_state *state,
+    struct drm_crtc *crtc,
+    struct drm_crtc_state *crtc_state)
+{
+    struct drm_plane *plane = NULL;
+    struct drm_plane_state *plane_state = NULL;
+    struct nv_drm_plane *nv_plane = NULL;
+    struct NvKmsKapiLayerRequestedConfig *plane_req_config = NULL;
+    struct NvKmsKapiHeadRequestedConfig *head_req_config =
+        &to_nv_crtc_state(crtc_state)->req_config;
+    struct nv_drm_plane_fence_cb_data *fence_data;
+    uint32_t semaphore_index;
+    uint32_t idx_count;
+    int ret, i;
+
+    if (!crtc_state->active) {
+        return 0;
+    }
+
+    nv_drm_for_each_new_plane_in_state(state, plane, plane_state, i) {
+        if ((plane->type == DRM_PLANE_TYPE_CURSOR) ||
+            (plane_state->crtc != crtc) ||
+            (plane_state->fence == NULL)) {
+            continue;
+        }
+
+        nv_plane = to_nv_plane(plane);
+        plane_req_config =
+            &head_req_config->layerRequestedConfig[nv_plane->layer_idx];
+
+        if (nv_dev->supportsSyncpts) {
+#if defined(NV_LINUX_NVHOST_H_PRESENT) && defined(CONFIG_TEGRA_GRHOST)
+#if defined(NV_NVHOST_DMA_FENCE_UNPACK_PRESENT)
+            int ret =
+                nvhost_dma_fence_unpack(
+                    plane_state->fence,
+                    &plane_req_config->config.syncParams.u.syncpt.preSyncptId,
+                    &plane_req_config->config.syncParams.u.syncpt.preSyncptValue);
+            if (ret == 0) {
+                plane_req_config->config.syncParams.preSyncptSpecified = true;
+                continue;
+            }
+#endif
+#elif defined(NV_LINUX_HOST1X_NEXT_H_PRESENT)
+            int ret =
+                host1x_fence_extract(
+                    plane_state->fence,
+                    &plane_req_config->config.syncParams.u.syncpt.preSyncptId,
+                    &plane_req_config->config.syncParams.u.syncpt.preSyncptValue);
+            if (ret == 0) {
+                plane_req_config->config.syncParams.preSyncptSpecified = true;
+                continue;
+            }
+#endif
+        }
+
+        /*
+         * Syncpt extraction failed, or syncpts are not supported.
+         * Use general DRM fence support with semaphores instead.
+         */
+        if (plane_req_config->config.syncParams.postSyncptRequested) {
+            // Can't mix Syncpts and semaphores in a given request.
+            return -EINVAL;
+        }
+
+        for (idx_count = 0; idx_count < nv_dev->display_semaphores.count; idx_count++) {
+            semaphore_index = nv_drm_next_display_semaphore(nv_dev);
+            if (nvKms->tryInitDisplaySemaphore(nv_dev->pDevice, semaphore_index)) {
+                break;
+            }
+        }
+
+        if (idx_count == nv_dev->display_semaphores.count) {
+            NV_DRM_DEV_LOG_ERR(
+                nv_dev,
+                "Failed to initialize semaphore for plane fence");
+            /*
+             * This should only happen if the semaphore pool was somehow
+             * exhausted. Waiting a bit and retrying may help in that case.
+             */
+            return -EAGAIN;
+        }
+
+        plane_req_config->config.syncParams.semaphoreSpecified = true;
+        plane_req_config->config.syncParams.u.semaphore.index = semaphore_index;
+
+        fence_data = nv_drm_calloc(1, sizeof(*fence_data));
+
+        if (!fence_data) {
+            NV_DRM_DEV_LOG_ERR(
+                nv_dev,
+                "Failed to allocate callback data for plane fence");
+            nvKms->cancelDisplaySemaphore(nv_dev->pDevice, semaphore_index);
+            return -ENOMEM;
+        }
+
+        fence_data->nv_dev = nv_dev;
+        fence_data->semaphore_index = semaphore_index;
+
+        ret = nv_dma_fence_add_callback(plane_state->fence,
+                                        &fence_data->dma_fence_cb,
+                                        __nv_drm_plane_fence_cb);
+
+        switch (ret) {
+        case -ENOENT:
+            /* The fence is already signaled */
+            __nv_drm_plane_fence_cb(plane_state->fence,
+                                    &fence_data->dma_fence_cb);
+#if defined(fallthrough)
+            fallthrough;
+#else
+            /* Fallthrough */
+#endif
+        case 0:
+            /*
+             * The plane state's fence reference has either been consumed or
+             * belongs to the outstanding callback now.
+             */
+            plane_state->fence = NULL;
+            break;
+        default:
+            NV_DRM_DEV_LOG_ERR(
+                nv_dev,
+                "Failed plane fence callback registration");
+            /* Fence callback registration failed */
+            nvKms->cancelDisplaySemaphore(nv_dev->pDevice, semaphore_index);
+            nv_drm_free(fence_data);
+            return ret;
+        }
+    }
+
    return 0;
 }
+#endif /* defined(NV_DRM_FENCE_AVAILABLE) */

 static int __nv_drm_get_syncpt_data(
    struct nv_drm_device *nv_dev,
@@ -237,6 +406,39 @@ nv_drm_atomic_apply_modeset_config(struct drm_device *dev,
    int i;
    int ret;

+    /*
+     * If sub-owner permission was granted to another NVKMS client, disallow
+     * modesets through the DRM interface.
+     */
+    if (nv_dev->subOwnershipGranted) {
+        return -EINVAL;
+    }
+
+#if defined(NV_DRM_FRAMEBUFFER_OBJ_PRESENT)
+    if (commit) {
+        /*
+         * This function does what is necessary to prepare the framebuffers
+         * attached to each new plane in the state for scan out, mostly by
+         * calling back into driver callbacks the NVIDIA driver does not
+         * provide. The end result is that all it does on the NVIDIA driver
+         * is populate the plane state's dma fence pointers with any implicit
+         * sync fences attached to the GEM objects associated with those planes
+         * in the new state, prefering explicit sync fences when appropriate.
+         * This must be done prior to converting the per-plane fences to
+         * semaphore waits below.
+         *
+         * Note this only works when the drm_framebuffer:obj[] field is present
+         * and populated, so skip calling this function on kernels where that
+         * field is not present.
+         */
+        ret = drm_atomic_helper_prepare_planes(dev, state);
+
+        if (ret) {
+            return ret;
+        }
+    }
+#endif /* defined(NV_DRM_FRAMEBUFFER_OBJ_PRESENT) */
+
    memset(requested_config, 0, sizeof(*requested_config));

    /* Loop over affected crtcs and construct NvKmsKapiRequestedModeSetConfig */
@@ -250,11 +452,6 @@ nv_drm_atomic_apply_modeset_config(struct drm_device *dev,
                               commit ? crtc->state : crtc_state;
        struct nv_drm_crtc *nv_crtc = to_nv_crtc(crtc);

-        requested_config->headRequestedConfig[nv_crtc->head] =
-            to_nv_crtc_state(new_crtc_state)->req_config;
-
-        requested_config->headsMask |= 1 << nv_crtc->head;
-
        if (commit) {
            struct drm_crtc_state *old_crtc_state = crtc_state;
            struct nv_drm_crtc_state *nv_new_crtc_state =
@@ -274,7 +471,27 @@ nv_drm_atomic_apply_modeset_config(struct drm_device *dev,

                nv_new_crtc_state->nv_flip = NULL;
            }
+
+#if defined(NV_DRM_FENCE_AVAILABLE)
+            ret = __nv_drm_convert_in_fences(nv_dev,
+                                             state,
+                                             crtc,
+                                             new_crtc_state);
+
+            if (ret != 0) {
+                return ret;
+            }
+#endif /* defined(NV_DRM_FENCE_AVAILABLE) */
        }
+
+        /*
+         * Do this deep copy after calling __nv_drm_convert_in_fences,
+         * which modifies the new CRTC state's req_config member
+         */
+        requested_config->headRequestedConfig[nv_crtc->head] =
+            to_nv_crtc_state(new_crtc_state)->req_config;
+
+        requested_config->headsMask |= 1 << nv_crtc->head;
    }

    if (commit && nvKms->systemInfo.bAllowWriteCombining) {
@@ -289,7 +506,9 @@ nv_drm_atomic_apply_modeset_config(struct drm_device *dev,
                                   requested_config,
                                   &reply_config,
                                   commit)) {
-        return -EINVAL;
+        if (commit || reply_config.flipResult != NV_KMS_FLIP_RESULT_IN_PROGRESS) {
+            return -EINVAL;
+        }
    }

    if (commit && nv_dev->supportsSyncpts) {
@@ -303,6 +522,10 @@ nv_drm_atomic_apply_modeset_config(struct drm_device *dev,
        }
    }

+    if (commit && nv_dev->requiresVrrSemaphores && reply_config.vrrFlip) {
+        nvKms->signalVrrSemaphore(nv_dev->pDevice, reply_config.vrrSemaphoreIndex);
+    }
+
    return 0;
 }

@@ -311,6 +534,24 @@ int nv_drm_atomic_check(struct drm_device *dev,
 {
    int ret = 0;

+#if defined(NV_DRM_COLOR_MGMT_AVAILABLE)
+    struct drm_crtc *crtc;
+    struct drm_crtc_state *crtc_state;
+    int i;
+
+    nv_drm_for_each_crtc_in_state(state, crtc, crtc_state, i) {
+        /*
+         * if the color management changed on the crtc, we need to update the
+         * crtc's plane's CSC matrices, so add the crtc's planes to the commit
+         */
+        if (crtc_state->color_mgmt_changed) {
+            if ((ret = drm_atomic_add_affected_planes(state, crtc)) != 0) {
+                goto done;
+            }
+        }
+    }
+#endif /* NV_DRM_COLOR_MGMT_AVAILABLE */
+
    if ((ret = drm_atomic_helper_check(dev, state)) != 0) {
        goto done;
    }
@@ -385,42 +626,56 @@ int nv_drm_atomic_commit(struct drm_device *dev,
    struct nv_drm_device *nv_dev = to_nv_device(dev);

    /*
-     * drm_mode_config_funcs::atomic_commit() mandates to return -EBUSY
-     * for nonblocking commit if previous updates (commit tasks/flip event) are
-     * pending. In case of blocking commits it mandates to wait for previous
-     * updates to complete.
+     * XXX: drm_mode_config_funcs::atomic_commit() mandates to return -EBUSY
+     * for nonblocking commit if the commit would need to wait for previous
+     * updates (commit tasks/flip event) to complete. In case of blocking
+     * commits it mandates to wait for previous updates to complete. However,
+     * the kernel DRM-KMS documentation does explicitly allow maintaining a
+     * queue of outstanding commits.
+     *
+     * Our system already implements such a queue, but due to
+     * bug 4054608, it is currently not used.
     */
-    if (nonblock) {
-        nv_drm_for_each_crtc_in_state(state, crtc, crtc_state, i) {
-            struct nv_drm_crtc *nv_crtc = to_nv_crtc(crtc);
+    nv_drm_for_each_crtc_in_state(state, crtc, crtc_state, i) {
+        struct nv_drm_crtc *nv_crtc = to_nv_crtc(crtc);

-            /*
-             * Here you aren't required to hold nv_drm_crtc::flip_list_lock
-             * because:
-             *
-             * The core DRM driver acquires lock for all affected crtcs before
-             * calling into ->commit() hook, therefore it is not possible for
-             * other threads to call into ->commit() hook affecting same crtcs
-             * and enqueue flip objects into flip_list -
-             *
-             *   nv_drm_atomic_commit_internal()
-             *     |-> nv_drm_atomic_apply_modeset_config(commit=true)
-             *           |-> nv_drm_crtc_enqueue_flip()
-             *
-             * Only possibility is list_empty check races with code path
-             * dequeuing flip object -
-             *
-             *   __nv_drm_handle_flip_event()
-             *     |-> nv_drm_crtc_dequeue_flip()
-             *
-             * But this race condition can't lead list_empty() to return
-             * incorrect result. nv_drm_crtc_dequeue_flip() in the middle of
-             * updating the list could not trick us into thinking the list is
-             * empty when it isn't.
-             */
+        /*
+         * Here you aren't required to hold nv_drm_crtc::flip_list_lock
+         * because:
+         *
+         * The core DRM driver acquires lock for all affected crtcs before
+         * calling into ->commit() hook, therefore it is not possible for
+         * other threads to call into ->commit() hook affecting same crtcs
+         * and enqueue flip objects into flip_list -
+         *
+         *   nv_drm_atomic_commit_internal()
+         *     |-> nv_drm_atomic_apply_modeset_config(commit=true)
+         *           |-> nv_drm_crtc_enqueue_flip()
+         *
+         * Only possibility is list_empty check races with code path
+         * dequeuing flip object -
+         *
+         *   __nv_drm_handle_flip_event()
+         *     |-> nv_drm_crtc_dequeue_flip()
+         *
+         * But this race condition can't lead list_empty() to return
+         * incorrect result. nv_drm_crtc_dequeue_flip() in the middle of
+         * updating the list could not trick us into thinking the list is
+         * empty when it isn't.
+         */
+        if (nonblock) {
            if (!list_empty(&nv_crtc->flip_list)) {
                return -EBUSY;
            }
+        } else {
+            if (wait_event_timeout(
+                    nv_dev->flip_event_wq,
+                    list_empty(&nv_crtc->flip_list),
+                    3 * HZ /* 3 second */) == 0) {
+                NV_DRM_DEV_LOG_ERR(
+                    nv_dev,
+                    "Flip event timeout on head %u", nv_crtc->head);
+            }
        }
    }

@@ -544,6 +799,9 @@ int nv_drm_atomic_commit(struct drm_device *dev,
                NV_DRM_DEV_LOG_ERR(
                    nv_dev,
                    "Flip event timeout on head %u", nv_crtc->head);
+                while (!list_empty(&nv_crtc->flip_list)) {
+                    __nv_drm_handle_flip_event(nv_crtc);
+                }
            }
        }
    }
--- a/kernel-open/nvidia-drm/nvidia-drm-os-interface.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-os-interface.c
@@ -0,0 +1,285 @@
+/*
+ * Copyright (c) 2015-2023, NVIDIA CORPORATION. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#include <linux/slab.h>
+
+#include "nvidia-drm-os-interface.h"
+
+#if defined(NV_DRM_AVAILABLE)
+
+#if defined(NV_LINUX_SYNC_FILE_H_PRESENT)
+#include <linux/file.h>
+#include <linux/sync_file.h>
+#endif
+
+#include <linux/vmalloc.h>
+#include <linux/sched.h>
+#include <linux/device.h>
+
+#include "nv-mm.h"
+
+#if defined(NV_DRM_DRMP_H_PRESENT)
+#include <drm/drmP.h>
+#endif
+
+bool nv_drm_modeset_module_param = false;
+bool nv_drm_fbdev_module_param = false;
+
+void *nv_drm_calloc(size_t nmemb, size_t size)
+{
+    size_t total_size = nmemb * size;
+    //
+    // Check for overflow.
+    //
+    if ((nmemb != 0) && ((total_size / nmemb) != size))
+    {
+        return NULL;
+    }
+    return kzalloc(nmemb * size, GFP_KERNEL);
+}
+
+void nv_drm_free(void *ptr)
+{
+    if (IS_ERR(ptr)) {
+        return;
+    }
+
+    kfree(ptr);
+}
+
+char *nv_drm_asprintf(const char *fmt, ...)
+{
+    va_list ap;
+    char *p;
+
+    va_start(ap, fmt);
+    p = kvasprintf(GFP_KERNEL, fmt, ap);
+    va_end(ap);
+
+    return p;
+}
+
+#if defined(NVCPU_X86) || defined(NVCPU_X86_64)
+  #define WRITE_COMBINE_FLUSH()    asm volatile("sfence":::"memory")
+#elif defined(NVCPU_PPC64LE)
+  #define WRITE_COMBINE_FLUSH()    asm volatile("sync":::"memory")
+#else
+  #define WRITE_COMBINE_FLUSH()    mb()
+#endif
+
+void nv_drm_write_combine_flush(void)
+{
+    WRITE_COMBINE_FLUSH();
+}
+
+int nv_drm_lock_user_pages(unsigned long address,
+                           unsigned long pages_count, struct page ***pages)
+{
+    struct mm_struct *mm = current->mm;
+    struct page **user_pages;
+    int pages_pinned;
+
+    user_pages = nv_drm_calloc(pages_count, sizeof(*user_pages));
+
+    if (user_pages == NULL) {
+        return -ENOMEM;
+    }
+
+    nv_mmap_read_lock(mm);
+
+    pages_pinned = NV_PIN_USER_PAGES(address, pages_count, FOLL_WRITE,
+                                     user_pages);
+    nv_mmap_read_unlock(mm);
+
+    if (pages_pinned < 0 || (unsigned)pages_pinned < pages_count) {
+        goto failed;
+    }
+
+    *pages = user_pages;
+
+    return 0;
+
+failed:
+
+    if (pages_pinned > 0) {
+        int i;
+
+        for (i = 0; i < pages_pinned; i++) {
+           NV_UNPIN_USER_PAGE(user_pages[i]);
+        }
+    }
+
+    nv_drm_free(user_pages);
+
+    return (pages_pinned < 0) ? pages_pinned : -EINVAL;
+}
+
+void nv_drm_unlock_user_pages(unsigned long  pages_count, struct page **pages)
+{
+    unsigned long i;
+
+    for (i = 0; i < pages_count; i++) {
+        set_page_dirty_lock(pages[i]);
+        NV_UNPIN_USER_PAGE(pages[i]);
+    }
+
+    nv_drm_free(pages);
+}
+
+/*
+ * linuxkpi vmap doesn't use the flags argument as it
+ * doesn't seem to be needed. Define VM_USERMAP to 0
+ * to make errors go away
+ *
+ * vmap: sys/compat/linuxkpi/common/src/linux_compat.c
+ */
+#if defined(NV_BSD)
+#define VM_USERMAP 0
+#endif
+
+void *nv_drm_vmap(struct page **pages, unsigned long pages_count)
+{
+    return vmap(pages, pages_count, VM_USERMAP, PAGE_KERNEL);
+}
+
+void nv_drm_vunmap(void *address)
+{
+    vunmap(address);
+}
+
+bool nv_drm_workthread_init(nv_drm_workthread *worker, const char *name)
+{
+    worker->shutting_down = false;
+    if (nv_kthread_q_init(&worker->q, name)) {
+        return false;
+    }
+
+    spin_lock_init(&worker->lock);
+
+    return true;
+}
+
+void nv_drm_workthread_shutdown(nv_drm_workthread *worker)
+{
+    unsigned long flags;
+
+    spin_lock_irqsave(&worker->lock, flags);
+    worker->shutting_down = true;
+    spin_unlock_irqrestore(&worker->lock, flags);
+
+    nv_kthread_q_stop(&worker->q);
+}
+
+void nv_drm_workthread_work_init(nv_drm_work *work,
+                                 void (*callback)(void *),
+                                 void *arg)
+{
+    nv_kthread_q_item_init(work, callback, arg);
+}
+
+int nv_drm_workthread_add_work(nv_drm_workthread *worker, nv_drm_work *work)
+{
+    unsigned long flags;
+    int ret = 0;
+
+    spin_lock_irqsave(&worker->lock, flags);
+    if (!worker->shutting_down) {
+        ret = nv_kthread_q_schedule_q_item(&worker->q, work);
+    }
+    spin_unlock_irqrestore(&worker->lock, flags);
+
+    return ret;
+}
+
+void nv_drm_timer_setup(nv_drm_timer *timer, void (*callback)(nv_drm_timer *nv_drm_timer))
+{
+    nv_timer_setup(timer, callback);
+}
+
+void nv_drm_mod_timer(nv_drm_timer *timer, unsigned long timeout_native)
+{
+    mod_timer(&timer->kernel_timer, timeout_native);
+}
+
+unsigned long nv_drm_timer_now(void)
+{
+    return jiffies;
+}
+
+unsigned long nv_drm_timeout_from_ms(NvU64 relative_timeout_ms)
+{
+    return jiffies + msecs_to_jiffies(relative_timeout_ms);
+}
+
+bool nv_drm_del_timer_sync(nv_drm_timer *timer)
+{
+    if (del_timer_sync(&timer->kernel_timer)) {
+        return true;
+    } else {
+        return false;
+    }
+}
+
+#if defined(NV_DRM_FENCE_AVAILABLE)
+int nv_drm_create_sync_file(nv_dma_fence_t *fence)
+{
+#if defined(NV_LINUX_SYNC_FILE_H_PRESENT)
+    struct sync_file *sync;
+    int fd = get_unused_fd_flags(O_CLOEXEC);
+
+    if (fd < 0) {
+        return fd;
+    }
+
+    /* sync_file_create() generates its own reference to the fence */
+    sync = sync_file_create(fence);
+
+    if (IS_ERR(sync)) {
+        put_unused_fd(fd);
+        return PTR_ERR(sync);
+    }
+
+    fd_install(fd, sync->file);
+
+    return fd;
+#else /* defined(NV_LINUX_SYNC_FILE_H_PRESENT) */
+    return -EINVAL;
+#endif  /* defined(NV_LINUX_SYNC_FILE_H_PRESENT) */
+}
+
+nv_dma_fence_t *nv_drm_sync_file_get_fence(int fd)
+{
+#if defined(NV_SYNC_FILE_GET_FENCE_PRESENT)
+    return sync_file_get_fence(fd);
+#else /* defined(NV_SYNC_FILE_GET_FENCE_PRESENT) */
+    return NULL;
+#endif  /* defined(NV_SYNC_FILE_GET_FENCE_PRESENT) */
+}
+#endif /* defined(NV_DRM_FENCE_AVAILABLE) */
+
+void nv_drm_yield(void)
+{
+    set_current_state(TASK_INTERRUPTIBLE);
+    schedule_timeout(1);
+}
+
+#endif /* NV_DRM_AVAILABLE */
--- a/kernel-open/nvidia-drm/nvidia-drm-os-interface.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-os-interface.h
@@ -29,10 +29,53 @@

 #if defined(NV_DRM_AVAILABLE)

+#if defined(NV_DRM_FENCE_AVAILABLE)
+#include "nvidia-dma-fence-helper.h"
+#endif
+
+#if defined(NV_LINUX) || defined(NV_BSD)
+#include "nv-kthread-q.h"
+#include "linux/spinlock.h"
+
+typedef struct nv_drm_workthread {
+    spinlock_t lock;
+    struct nv_kthread_q q;
+    bool shutting_down;
+} nv_drm_workthread;
+
+typedef nv_kthread_q_item_t nv_drm_work;
+
+#else
+#error "Need to define deferred work primitives for this OS"
+#endif
+
+#if defined(NV_LINUX) || defined(NV_BSD)
+#include "nv-timer.h"
+
+typedef struct nv_timer nv_drm_timer;
+
+#else
+#error "Need to define kernel timer callback primitives for this OS"
+#endif
+
+#if defined(NV_DRM_FBDEV_GENERIC_SETUP_PRESENT) && defined(NV_DRM_APERTURE_REMOVE_CONFLICTING_PCI_FRAMEBUFFERS_PRESENT)
+#define NV_DRM_FBDEV_AVAILABLE
+#define NV_DRM_FBDEV_GENERIC_AVAILABLE
+#endif
+
+#if defined(NV_DRM_FBDEV_TTM_SETUP_PRESENT) && defined(NV_DRM_APERTURE_REMOVE_CONFLICTING_PCI_FRAMEBUFFERS_PRESENT)
+#define NV_DRM_FBDEV_AVAILABLE
+#define NV_DRM_FBDEV_TTM_AVAILABLE
+#endif
+
 struct page;

 /* Set to true when the atomic modeset feature is enabled. */
 extern bool nv_drm_modeset_module_param;
+#if defined(NV_DRM_FBDEV_AVAILABLE)
+/* Set to true when the nvidia-drm driver should install a framebuffer device */
+extern bool nv_drm_fbdev_module_param;
+#endif

 void *nv_drm_calloc(size_t nmemb, size_t size);

@@ -51,6 +94,37 @@ void *nv_drm_vmap(struct page **pages, unsigned long pages_count);

 void nv_drm_vunmap(void *address);

-#endif
+bool nv_drm_workthread_init(nv_drm_workthread *worker, const char *name);
+
+/* Can be called concurrently with nv_drm_workthread_add_work() */
+void nv_drm_workthread_shutdown(nv_drm_workthread *worker);
+
+void nv_drm_workthread_work_init(nv_drm_work *work,
+                                 void (*callback)(void *),
+                                 void *arg);
+
+/* Can be called concurrently with nv_drm_workthread_shutdown() */
+int nv_drm_workthread_add_work(nv_drm_workthread *worker, nv_drm_work *work);
+
+void nv_drm_timer_setup(nv_drm_timer *timer,
+                        void (*callback)(nv_drm_timer *nv_drm_timer));
+
+void nv_drm_mod_timer(nv_drm_timer *timer, unsigned long relative_timeout_ms);
+
+bool nv_drm_del_timer_sync(nv_drm_timer *timer);
+
+unsigned long nv_drm_timer_now(void);
+
+unsigned long nv_drm_timeout_from_ms(NvU64 relative_timeout_ms);
+
+#if defined(NV_DRM_FENCE_AVAILABLE)
+int nv_drm_create_sync_file(nv_dma_fence_t *fence);
+
+nv_dma_fence_t *nv_drm_sync_file_get_fence(int fd);
+#endif /* defined(NV_DRM_FENCE_AVAILABLE) */
+
+void nv_drm_yield(void);
+
+#endif /* defined(NV_DRM_AVAILABLE) */

 #endif /* __NVIDIA_DRM_OS_INTERFACE_H__ */
--- a/kernel-open/nvidia-drm/nvidia-drm-prime-fence.c
+++ b/kernel-open/nvidia-drm/nvidia-drm-prime-fence.c
@@ -1,527 +0,0 @@
-/*
- * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
- *
- * Permission is hereby granted, free of charge, to any person obtaining a
- * copy of this software and associated documentation files (the "Software"),
- * to deal in the Software without restriction, including without limitation
- * the rights to use, copy, modify, merge, publish, distribute, sublicense,
- * and/or sell copies of the Software, and to permit persons to whom the
- * Software is furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- */
-
-#include "nvidia-drm-conftest.h"
-
-#if defined(NV_DRM_AVAILABLE)
-
-#if defined(NV_DRM_DRMP_H_PRESENT)
-#include <drm/drmP.h>
-#endif
-
-#include "nvidia-drm-priv.h"
-#include "nvidia-drm-ioctl.h"
-#include "nvidia-drm-gem.h"
-#include "nvidia-drm-prime-fence.h"
-#include "nvidia-dma-resv-helper.h"
-
-#if defined(NV_DRM_FENCE_AVAILABLE)
-
-#include "nvidia-dma-fence-helper.h"
-
-struct nv_drm_fence_context {
-    struct nv_drm_device *nv_dev;
-
-    uint32_t context;
-
-    NvU64 fenceSemIndex; /* Index into semaphore surface */
-
-    /* Mapped semaphore surface */
-    struct NvKmsKapiMemory *pSemSurface;
-    NvU32 *pLinearAddress;
-
-    /* Protects nv_drm_fence_context::{pending, last_seqno} */
-    spinlock_t lock;
-
-    /*
-     * Software signaling structures. __nv_drm_fence_context_new()
-     * allocates channel event and __nv_drm_fence_context_destroy() frees it.
-     * There are no simultaneous read/write access to 'cb', therefore it does
-     * not require spin-lock protection.
-     */
-    struct NvKmsKapiChannelEvent *cb;
-
-    /* List of pending fences which are not yet signaled */
-    struct list_head pending;
-
-    unsigned last_seqno;
-};
-
-struct nv_drm_prime_fence {
-    struct list_head list_entry;
-    nv_dma_fence_t base;
-    spinlock_t lock;
-};
-
-static inline
-struct nv_drm_prime_fence *to_nv_drm_prime_fence(nv_dma_fence_t *fence)
-{
-    return container_of(fence, struct nv_drm_prime_fence, base);
-}
-
-static const char*
-nv_drm_gem_prime_fence_op_get_driver_name(nv_dma_fence_t *fence)
-{
-    return "NVIDIA";
-}
-
-static const char*
-nv_drm_gem_prime_fence_op_get_timeline_name(nv_dma_fence_t *fence)
-{
-    return "nvidia.prime";
-}
-
-static bool nv_drm_gem_prime_fence_op_enable_signaling(nv_dma_fence_t *fence)
-{
-    // DO NOTHING
-    return true;
-}
-
-static void nv_drm_gem_prime_fence_op_release(nv_dma_fence_t *fence)
-{
-    struct nv_drm_prime_fence *nv_fence = to_nv_drm_prime_fence(fence);
-    nv_drm_free(nv_fence);
-}
-
-static signed long
-nv_drm_gem_prime_fence_op_wait(nv_dma_fence_t *fence,
-                               bool intr, signed long timeout)
-{
-    /*
-     * If the waiter requests to wait with no timeout, force a timeout to ensure
-     * that it won't get stuck forever in the kernel if something were to go
-     * wrong with signaling, such as a malicious userspace not releasing the
-     * semaphore.
-     *
-     * 96 ms (roughly 6 frames @ 60 Hz) is arbitrarily chosen to be long enough
-     * that it should never get hit during normal operation, but not so long
-     * that the system becomes unresponsive.
-     */
-    return nv_dma_fence_default_wait(fence, intr,
-                              (timeout == MAX_SCHEDULE_TIMEOUT) ?
-                                  msecs_to_jiffies(96) : timeout);
-}
-
-static const nv_dma_fence_ops_t nv_drm_gem_prime_fence_ops = {
-    .get_driver_name = nv_drm_gem_prime_fence_op_get_driver_name,
-    .get_timeline_name = nv_drm_gem_prime_fence_op_get_timeline_name,
-    .enable_signaling = nv_drm_gem_prime_fence_op_enable_signaling,
-    .release = nv_drm_gem_prime_fence_op_release,
-    .wait = nv_drm_gem_prime_fence_op_wait,
-};
-
-static inline void
-__nv_drm_prime_fence_signal(struct nv_drm_prime_fence *nv_fence)
-{
-    list_del(&nv_fence->list_entry);
-    nv_dma_fence_signal(&nv_fence->base);
-    nv_dma_fence_put(&nv_fence->base);
-}
-
-static void nv_drm_gem_prime_force_fence_signal(
-    struct nv_drm_fence_context *nv_fence_context)
-{
-    WARN_ON(!spin_is_locked(&nv_fence_context->lock));
-
-    while (!list_empty(&nv_fence_context->pending)) {
-        struct nv_drm_prime_fence *nv_fence = list_first_entry(
-            &nv_fence_context->pending,
-            typeof(*nv_fence),
-            list_entry);
-
-        __nv_drm_prime_fence_signal(nv_fence);
-    }
-}
-
-static void nv_drm_gem_prime_fence_event
-(
-    void *dataPtr,
-    NvU32 dataU32
-)
-{
-    struct nv_drm_fence_context *nv_fence_context = dataPtr;
-
-    spin_lock(&nv_fence_context->lock);
-
-    while (!list_empty(&nv_fence_context->pending)) {
-        struct nv_drm_prime_fence *nv_fence = list_first_entry(
-            &nv_fence_context->pending,
-            typeof(*nv_fence),
-            list_entry);
-
-        /* Index into surface with 16 byte stride */
-        unsigned int seqno = *((nv_fence_context->pLinearAddress) +
-                               (nv_fence_context->fenceSemIndex * 4));
-
-        if (nv_fence->base.seqno > seqno) {
-            /*
-             * Fences in list are placed in increasing order of sequence
-             * number, breaks a loop once found first fence not
-             * ready to signal.
-             */
-            break;
-        }
-
-        __nv_drm_prime_fence_signal(nv_fence);
-    }
-
-    spin_unlock(&nv_fence_context->lock);
-}
-
-static inline struct nv_drm_fence_context *__nv_drm_fence_context_new(
-    struct nv_drm_device *nv_dev,
-    struct drm_nvidia_fence_context_create_params *p)
-{
-    struct nv_drm_fence_context *nv_fence_context;
-    struct NvKmsKapiMemory *pSemSurface;
-    NvU32 *pLinearAddress;
-
-    /* Allocate backup nvkms resources */
-
-    pSemSurface = nvKms->importMemory(nv_dev->pDevice,
-                                      p->size,
-                                      p->import_mem_nvkms_params_ptr,
-                                      p->import_mem_nvkms_params_size);
-    if (!pSemSurface) {
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to import fence semaphore surface");
-
-        goto failed;
-    }
-
-    if (!nvKms->mapMemory(nv_dev->pDevice,
-                          pSemSurface,
-                          NVKMS_KAPI_MAPPING_TYPE_KERNEL,
-                          (void **) &pLinearAddress)) {
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to map fence semaphore surface");
-
-        goto failed_to_map_memory;
-    }
-
-    /*
-     * Allocate a fence context object, initialize it and allocate channel
-     * event for it.
-     */
-
-    if ((nv_fence_context = nv_drm_calloc(
-                    1,
-                    sizeof(*nv_fence_context))) == NULL) {
-        goto failed_alloc_fence_context;
-    }
-
-    /*
-     * nv_dma_fence_context_alloc() cannot fail, so we do not need
-     * to check a return value.
-     */
-
-    *nv_fence_context = (struct nv_drm_fence_context) {
-        .nv_dev = nv_dev,
-        .context = nv_dma_fence_context_alloc(1),
-        .pSemSurface = pSemSurface,
-        .pLinearAddress = pLinearAddress,
-        .fenceSemIndex = p->index,
-    };
-
-    INIT_LIST_HEAD(&nv_fence_context->pending);
-
-    spin_lock_init(&nv_fence_context->lock);
-
-    /*
-     * Except 'cb', the fence context should be completely initialized
-     * before channel event allocation because the fence context may start
-     * receiving events immediately after allocation.
-     *
-     * There are no simultaneous read/write access to 'cb', therefore it does
-     * not require spin-lock protection.
-     */
-    nv_fence_context->cb =
-        nvKms->allocateChannelEvent(nv_dev->pDevice,
-                                    nv_drm_gem_prime_fence_event,
-                                    nv_fence_context,
-                                    p->event_nvkms_params_ptr,
-                                    p->event_nvkms_params_size);
-    if (!nv_fence_context->cb) {
-        NV_DRM_DEV_LOG_ERR(nv_dev,
-                           "Failed to allocate fence signaling event");
-        goto failed_to_allocate_channel_event;
-    }
-
-    return nv_fence_context;
-
-failed_to_allocate_channel_event:
-    nv_drm_free(nv_fence_context);
-
-failed_alloc_fence_context:
-
-    nvKms->unmapMemory(nv_dev->pDevice,
-                       pSemSurface,
-                       NVKMS_KAPI_MAPPING_TYPE_KERNEL,
-                       (void *) pLinearAddress);
-
-failed_to_map_memory:
-    nvKms->freeMemory(nv_dev->pDevice, pSemSurface);
-
-failed:
-    return NULL;
-}
-
-static void __nv_drm_fence_context_destroy(
-    struct nv_drm_fence_context *nv_fence_context)
-{
-    struct nv_drm_device *nv_dev = nv_fence_context->nv_dev;
-
-    /*
-     * Free channel event before destroying the fence context, otherwise event
-     * callback continue to get called.
-     */
-    nvKms->freeChannelEvent(nv_dev->pDevice, nv_fence_context->cb);
-
-    /* Force signal all pending fences and empty pending list */
-    spin_lock(&nv_fence_context->lock);
-
-    nv_drm_gem_prime_force_fence_signal(nv_fence_context);
-
-    spin_unlock(&nv_fence_context->lock);
-
-    /* Free nvkms resources */
-
-    nvKms->unmapMemory(nv_dev->pDevice,
-                       nv_fence_context->pSemSurface,
-                       NVKMS_KAPI_MAPPING_TYPE_KERNEL,
-                       (void *) nv_fence_context->pLinearAddress);
-
-    nvKms->freeMemory(nv_dev->pDevice, nv_fence_context->pSemSurface);
-
-    nv_drm_free(nv_fence_context);
-}
-
-static nv_dma_fence_t *__nv_drm_fence_context_create_fence(
-    struct nv_drm_fence_context *nv_fence_context,
-    unsigned int seqno)
-{
-    struct nv_drm_prime_fence *nv_fence;
-    int ret = 0;
-
-    if ((nv_fence = nv_drm_calloc(1, sizeof(*nv_fence))) == NULL) {
-        ret = -ENOMEM;
-        goto out;
-    }
-
-    spin_lock(&nv_fence_context->lock);
-
-    /*
-     * If seqno wrapped, force signal fences to make sure none of them
-     * get stuck.
-     */
-    if (seqno < nv_fence_context->last_seqno) {
-        nv_drm_gem_prime_force_fence_signal(nv_fence_context);
-    }
-
-    INIT_LIST_HEAD(&nv_fence->list_entry);
-
-    spin_lock_init(&nv_fence->lock);
-
-    nv_dma_fence_init(&nv_fence->base, &nv_drm_gem_prime_fence_ops,
-                      &nv_fence->lock, nv_fence_context->context,
-                      seqno);
-
-    list_add_tail(&nv_fence->list_entry, &nv_fence_context->pending);
-
-    nv_fence_context->last_seqno = seqno;
-
-    spin_unlock(&nv_fence_context->lock);
-
-out:
-    return ret != 0 ? ERR_PTR(ret) : &nv_fence->base;
-}
-
-int nv_drm_fence_supported_ioctl(struct drm_device *dev,
-                                 void *data, struct drm_file *filep)
-{
-    struct nv_drm_device *nv_dev = to_nv_device(dev);
-    return nv_dev->pDevice ? 0 : -EINVAL;
-}
-
-struct nv_drm_gem_fence_context {
-    struct nv_drm_gem_object base;
-    struct nv_drm_fence_context *nv_fence_context;
-};
-
-static inline struct nv_drm_gem_fence_context *to_gem_fence_context(
-    struct nv_drm_gem_object *nv_gem)
-{
-    if (nv_gem != NULL) {
-        return container_of(nv_gem, struct nv_drm_gem_fence_context, base);
-    }
-
-    return NULL;
-}
-
-/*
- * Tear down of the 'struct nv_drm_gem_fence_context' object is not expected
- * to be happen from any worker thread, if that happen it causes dead-lock
- * because tear down sequence calls to flush all existing
- * worker thread.
- */
-static void __nv_drm_gem_fence_context_free(struct nv_drm_gem_object *nv_gem)
-{
-    struct nv_drm_gem_fence_context *nv_gem_fence_context =
-        to_gem_fence_context(nv_gem);
-
-    __nv_drm_fence_context_destroy(nv_gem_fence_context->nv_fence_context);
-
-    nv_drm_free(nv_gem_fence_context);
-}
-
-const struct nv_drm_gem_object_funcs nv_gem_fence_context_ops = {
-    .free = __nv_drm_gem_fence_context_free,
-};
-
-static inline
-struct nv_drm_gem_fence_context *__nv_drm_gem_object_fence_context_lookup(
-    struct drm_device *dev,
-    struct drm_file *filp,
-    u32 handle)
-{
-    struct nv_drm_gem_object *nv_gem =
-            nv_drm_gem_object_lookup(dev, filp, handle);
-
-    if (nv_gem != NULL && nv_gem->ops != &nv_gem_fence_context_ops) {
-        nv_drm_gem_object_unreference_unlocked(nv_gem);
-        return NULL;
-    }
-
-    return to_gem_fence_context(nv_gem);
-}
-
-int nv_drm_fence_context_create_ioctl(struct drm_device *dev,
-                                      void *data, struct drm_file *filep)
-{
-    struct nv_drm_device *nv_dev = to_nv_device(dev);
-    struct drm_nvidia_fence_context_create_params *p = data;
-    struct nv_drm_gem_fence_context *nv_gem_fence_context = NULL;
-
-    if ((nv_gem_fence_context = nv_drm_calloc(
-                1,
-                sizeof(struct nv_drm_gem_fence_context))) == NULL) {
-        goto done;
-    }
-
-    if ((nv_gem_fence_context->nv_fence_context =
-                __nv_drm_fence_context_new(nv_dev, p)) == NULL) {
-        goto fence_context_new_failed;
-    }
-
-    nv_drm_gem_object_init(nv_dev,
-                           &nv_gem_fence_context->base,
-                           &nv_gem_fence_context_ops,
-                           0 /* size */,
-                           NULL /* pMemory */);
-
-    return nv_drm_gem_handle_create_drop_reference(filep,
-                                                   &nv_gem_fence_context->base,
-                                                   &p->handle);
-
-fence_context_new_failed:
-    nv_drm_free(nv_gem_fence_context);
-
-done:
-    return -ENOMEM;
-}
-
-int nv_drm_gem_fence_attach_ioctl(struct drm_device *dev,
-                                  void *data, struct drm_file *filep)
-{
-    int ret = -EINVAL;
-    struct nv_drm_device *nv_dev = to_nv_device(dev);
-    struct drm_nvidia_gem_fence_attach_params *p = data;
-
-    struct nv_drm_gem_object *nv_gem;
-    struct nv_drm_gem_fence_context *nv_gem_fence_context;
-
-    nv_dma_fence_t *fence;
-
-    nv_gem = nv_drm_gem_object_lookup(nv_dev->dev, filep, p->handle);
-
-    if (!nv_gem) {
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to lookup gem object for fence attach: 0x%08x",
-            p->handle);
-
-        goto done;
-    }
-
-    if((nv_gem_fence_context = __nv_drm_gem_object_fence_context_lookup(
-                nv_dev->dev,
-                filep,
-                p->fence_context_handle)) == NULL) {
-
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to lookup gem object for fence context: 0x%08x",
-            p->fence_context_handle);
-
-        goto fence_context_lookup_failed;
-    }
-
-    if (IS_ERR(fence = __nv_drm_fence_context_create_fence(
-                            nv_gem_fence_context->nv_fence_context,
-                            p->sem_thresh))) {
-        ret = PTR_ERR(fence);
-
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to allocate fence: 0x%08x", p->handle);
-
-        goto fence_context_create_fence_failed;
-    }
-
-    nv_dma_resv_lock(&nv_gem->resv, NULL);
-
-    ret = nv_dma_resv_reserve_fences(&nv_gem->resv, 1, false);
-    if (ret == 0) {
-        nv_dma_resv_add_excl_fence(&nv_gem->resv, fence);
-    } else {
-        NV_DRM_DEV_LOG_ERR(
-            nv_dev,
-            "Failed to reserve fence. Error code: %d", ret);
-    }
-
-    nv_dma_resv_unlock(&nv_gem->resv);
-
-fence_context_create_fence_failed:
-    nv_drm_gem_object_unreference_unlocked(&nv_gem_fence_context->base);
-
-fence_context_lookup_failed:
-    nv_drm_gem_object_unreference_unlocked(nv_gem);
-
-done:
-    return ret;
-}
-
-#endif /* NV_DRM_FENCE_AVAILABLE */
-
-#endif /* NV_DRM_AVAILABLE */
--- a/kernel-open/nvidia-drm/nvidia-drm-priv.h
+++ b/kernel-open/nvidia-drm/nvidia-drm-priv.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
+ * Copyright (c) 2015-2022, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
@@ -46,12 +46,33 @@
 #define NV_DRM_LOG_ERR(__fmt, ...) \
    DRM_ERROR("[nvidia-drm] " __fmt "\n", ##__VA_ARGS__)

+/*
+ * DRM_WARN() was added in v4.9 by kernel commit
+ * 30b0da8d556e65ff935a56cd82c05ba0516d3e4a
+ *
+ * Before this commit, only DRM_INFO and DRM_ERROR were defined and
+ * DRM_INFO(fmt, ...) was defined as
+ * printk(KERN_INFO "[" DRM_NAME "] " fmt, ##__VA_ARGS__). So, if
+ * DRM_WARN is undefined this defines NV_DRM_LOG_WARN following the
+ * same pattern as DRM_INFO.
+ */
+#ifdef DRM_WARN
+#define NV_DRM_LOG_WARN(__fmt, ...) \
+    DRM_WARN("[nvidia-drm] " __fmt "\n", ##__VA_ARGS__)
+#else
+#define NV_DRM_LOG_WARN(__fmt, ...) \
+    printk(KERN_WARNING "[" DRM_NAME "] [nvidia-drm] " __fmt "\n", ##__VA_ARGS__)
+#endif
+
 #define NV_DRM_LOG_INFO(__fmt, ...) \
    DRM_INFO("[nvidia-drm] " __fmt "\n", ##__VA_ARGS__)

 #define NV_DRM_DEV_LOG_INFO(__dev, __fmt, ...) \
    NV_DRM_LOG_INFO("[GPU ID 0x%08x] " __fmt, __dev->gpu_info.gpu_id, ##__VA_ARGS__)

+#define NV_DRM_DEV_LOG_WARN(__dev, __fmt, ...) \
+    NV_DRM_LOG_WARN("[GPU ID 0x%08x] " __fmt, __dev->gpu_info.gpu_id, ##__VA_ARGS__)
+
 #define NV_DRM_DEV_LOG_ERR(__dev, __fmt, ...) \
    NV_DRM_LOG_ERR("[GPU ID 0x%08x] " __fmt, __dev->gpu_info.gpu_id, ##__VA_ARGS__)

@@ -105,6 +126,7 @@ struct nv_drm_device {
    NvU64 modifiers[6 /* block linear */ + 1 /* linear */ + 1 /* terminator */];
 #endif

+    struct delayed_work hotplug_event_work;
    atomic_t enable_event_handling;

    /**
@@ -117,15 +139,64 @@ struct nv_drm_device {

 #endif

+#if defined(NV_DRM_FENCE_AVAILABLE)
+    NvU64 semsurf_stride;
+    NvU64 semsurf_max_submitted_offset;
+#endif
+
    NvBool hasVideoMemory;

    NvBool supportsSyncpts;
+    NvBool requiresVrrSemaphores;
+    NvBool subOwnershipGranted;
+    NvBool hasFramebufferConsole;

    struct drm_property *nv_out_fence_property;
+    struct drm_property *nv_input_colorspace_property;
+
+    struct {
+        NvU32 count;
+        NvU32 next_index;
+    } display_semaphores;
+
+#if defined(NV_DRM_HAS_HDR_OUTPUT_METADATA)
+    struct drm_property *nv_hdr_output_metadata_property;
+#endif
+
+    struct drm_property *nv_plane_lms_ctm_property;
+    struct drm_property *nv_plane_lms_to_itp_ctm_property;
+    struct drm_property *nv_plane_itp_to_lms_ctm_property;
+    struct drm_property *nv_plane_blend_ctm_property;
+
+    struct drm_property *nv_plane_degamma_tf_property;
+    struct drm_property *nv_plane_degamma_lut_property;
+    struct drm_property *nv_plane_degamma_lut_size_property;
+    struct drm_property *nv_plane_degamma_multiplier_property;
+
+    struct drm_property *nv_plane_tmo_lut_property;
+    struct drm_property *nv_plane_tmo_lut_size_property;
+
+    struct drm_property *nv_crtc_regamma_tf_property;
+    struct drm_property *nv_crtc_regamma_lut_property;
+    struct drm_property *nv_crtc_regamma_lut_size_property;
+    struct drm_property *nv_crtc_regamma_divisor_property;

    struct nv_drm_device *next;
 };

+static inline NvU32 nv_drm_next_display_semaphore(
+    struct nv_drm_device *nv_dev)
+{
+    NvU32 current_index = nv_dev->display_semaphores.next_index++;
+
+    if (nv_dev->display_semaphores.next_index >=
+        nv_dev->display_semaphores.count) {
+        nv_dev->display_semaphores.next_index = 0;
+    }
+
+    return current_index;
+}
+
 static inline struct nv_drm_device *to_nv_device(
    struct drm_device *dev)
 {
--- a/kernel-open/nvidia-drm/nvidia-drm-sources.mk
+++ b/kernel-open/nvidia-drm/nvidia-drm-sources.mk
@@ -0,0 +1,142 @@
+###########################################################################
+# Kbuild fragment for nvidia-drm.ko
+###########################################################################
+
+#
+# Define NVIDIA_DRM_SOURCES
+#
+
+NVIDIA_DRM_SOURCES =
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-drv.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-utils.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-crtc.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-encoder.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-connector.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-fb.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-modeset.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-fence.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-helper.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nv-kthread-q.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nv-pci-table.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-nvkms-memory.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-user-memory.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-dma-buf.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-format.c
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-os-interface.c
+
+#
+# Register the conftests needed by nvidia-drm.ko
+#
+
+NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_available
+NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_atomic_available
+NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_gpl_refcount_inc
+NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_gpl_refcount_dec_and_test
+NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_alpha_blending_available
+NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_present_drm_gem_prime_fd_to_handle
+NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_present_drm_gem_prime_handle_to_fd
+
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_dev_unref
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_reinit_primary_mode_group
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += get_user_pages_remote
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += get_user_pages
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += pin_user_pages_remote
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += pin_user_pages
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_lookup
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_atomic_state_ref_counting
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_driver_has_gem_prime_res_obj
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_atomic_helper_connector_dpms
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_funcs_have_mode_in_name
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_has_vrr_capable_property
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += vmf_insert_pfn
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_framebuffer_get
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_get
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_dev_put
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_format_num_planes
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_for_each_possible_encoder
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_rotation_available
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_vma_offset_exact_lookup_locked
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_put_unlocked
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += nvhost_dma_fence_unpack
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += list_is_first
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += timer_setup
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += dma_fence_set_error
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += fence_set_error
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += sync_file_get_fence
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_aperture_remove_conflicting_pci_framebuffers
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_fbdev_generic_setup
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_fbdev_ttm_setup
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_attach_hdr_output_metadata_property
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_helper_crtc_enable_color_mgmt
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_crtc_enable_color_mgmt
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_atomic_helper_legacy_gamma_set
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += vmf_insert_mixed
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += pfn_to_pfn_t
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_prime_mmap
+
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_bus_type
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_get_irq
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_get_name
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_device_list
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_legacy_dev_list
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_set_busid
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_connectors_changed
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_init_function_args
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_helper_mode_fill_fb_struct
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_master_drop_has_from_release_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_unload_has_int_return_type
+NV_CONFTEST_TYPE_COMPILE_TESTS += vm_fault_has_address
+NV_CONFTEST_TYPE_COMPILE_TESTS += vm_ops_fault_removed_vma_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_crtc_destroy_state_has_crtc_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_plane_destroy_state_has_plane_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_mode_object_find_has_file_priv_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += dma_buf_owner
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_connector_list_iter
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_swap_state_has_stall_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_prime_flag_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += vm_fault_t
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_gem_object_has_resv
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_async_flip
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_pageflip_flags
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_vrr_enabled
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_format_modifiers_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += mm_has_mmap_lock
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_vma_node_is_allowed_has_tag_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_vma_offset_node_has_readonly
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_display_mode_has_vrefresh
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_master_set_has_int_return_type
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_gem_free_object
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_prime_pages_to_sg_has_drm_device_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_gem_prime_callbacks
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_atomic_check_has_atomic_state_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_gem_object_vmap_has_map_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_plane_atomic_check_has_atomic_state_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_device_has_pdev
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_no_vblank
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_mode_config_has_allow_fb_modifiers
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_has_hdr_output_metadata
+NV_CONFTEST_TYPE_COMPILE_TESTS += dma_resv_add_fence
+NV_CONFTEST_TYPE_COMPILE_TESTS += dma_resv_reserve_fences
+NV_CONFTEST_TYPE_COMPILE_TESTS += reservation_object_reserve_shared_has_num_fences_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_connector_has_override_edid
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_master_has_leases
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_file_get_master
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_modeset_lock_all_end
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_connector_lookup
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_connector_put
+NV_CONFTEST_TYPE_COMPILE_TESTS += vm_area_struct_has_const_vm_flags
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_dumb_destroy
+NV_CONFTEST_TYPE_COMPILE_TESTS += fence_ops_use_64bit_seqno
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_aperture_remove_conflicting_pci_framebuffers_has_driver_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_mode_create_dp_colorspace_property_has_supported_colorspaces_arg
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_syncobj_features_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_unlocked_ioctl_flag_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_framebuffer_obj_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_color_ctm_3x4_present
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_color_lut
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_property_blob_put
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_gem_prime_mmap
+NV_CONFTEST_TYPE_COMPILE_TESTS += drm_output_poll_changed
--- a/kernel-open/nvidia-drm/nvidia-drm.Kbuild
+++ b/kernel-open/nvidia-drm/nvidia-drm.Kbuild
@@ -2,29 +2,16 @@
 # Kbuild fragment for nvidia-drm.ko
 ###########################################################################

+# Get our source file list and conftest list from the common file
+include $(src)/nvidia-drm/nvidia-drm-sources.mk
+
+# Linux-specific sources
+NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-linux.c
+
 #
 # Define NVIDIA_DRM_{SOURCES,OBJECTS}
 #

-NVIDIA_DRM_SOURCES =
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-drv.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-utils.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-crtc.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-encoder.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-connector.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-fb.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-modeset.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-prime-fence.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-linux.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-helper.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nv-pci-table.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-nvkms-memory.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-user-memory.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-gem-dma-buf.c
-NVIDIA_DRM_SOURCES += nvidia-drm/nvidia-drm-format.c
-
 NVIDIA_DRM_OBJECTS = $(patsubst %.c,%.o,$(NVIDIA_DRM_SOURCES))

 obj-m += nvidia-drm.o
@@ -43,78 +30,4 @@ NVIDIA_DRM_CFLAGS += -UDEBUG -U_DEBUG -DNDEBUG -DNV_BUILD_MODULE_INSTANCES=0

 $(call ASSIGN_PER_OBJ_CFLAGS, $(NVIDIA_DRM_OBJECTS), $(NVIDIA_DRM_CFLAGS))

-#
-# Register the conftests needed by nvidia-drm.ko
-#
-
 NV_OBJECTS_DEPEND_ON_CONFTEST += $(NVIDIA_DRM_OBJECTS)
-
-NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_available
-NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_atomic_available
-NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_gpl_refcount_inc
-NV_CONFTEST_GENERIC_COMPILE_TESTS += is_export_symbol_gpl_refcount_dec_and_test
-NV_CONFTEST_GENERIC_COMPILE_TESTS += drm_alpha_blending_available
-
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_dev_unref
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_reinit_primary_mode_group
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += get_user_pages_remote
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += get_user_pages
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_lookup
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_atomic_state_ref_counting
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_driver_has_gem_prime_res_obj
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_atomic_helper_connector_dpms
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_funcs_have_mode_in_name
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += vmf_insert_pfn
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_framebuffer_get
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_get
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_dev_put
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_format_num_planes
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_connector_for_each_possible_encoder
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_rotation_available
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_vma_offset_exact_lookup_locked
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += drm_gem_object_put_unlocked
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += nvhost_dma_fence_unpack
-
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_present
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_bus_type
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_get_irq
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_bus_has_get_name
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_device_list
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_legacy_dev_list
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_set_busid
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_connectors_changed
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_init_function_args
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_helper_mode_fill_fb_struct
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_master_drop_has_from_release_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_unload_has_int_return_type
-NV_CONFTEST_TYPE_COMPILE_TESTS += vm_fault_has_address
-NV_CONFTEST_TYPE_COMPILE_TESTS += vm_ops_fault_removed_vma_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_crtc_destroy_state_has_crtc_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_plane_destroy_state_has_plane_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_mode_object_find_has_file_priv_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += dma_buf_owner
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_connector_list_iter
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_atomic_helper_swap_state_has_stall_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_prime_flag_present
-NV_CONFTEST_TYPE_COMPILE_TESTS += vm_fault_t
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_gem_object_has_resv
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_async_flip
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_pageflip_flags
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_format_modifiers_present
-NV_CONFTEST_TYPE_COMPILE_TESTS += mm_has_mmap_lock
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_vma_node_is_allowed_has_tag_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_vma_offset_node_has_readonly
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_display_mode_has_vrefresh
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_master_set_has_int_return_type
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_gem_free_object
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_prime_pages_to_sg_has_drm_device_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_driver_has_gem_prime_callbacks
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_atomic_check_has_atomic_state_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_gem_object_vmap_has_map_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_plane_atomic_check_has_atomic_state_arg
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_device_has_pdev
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_crtc_state_has_no_vblank
-NV_CONFTEST_TYPE_COMPILE_TESTS += drm_mode_config_has_allow_fb_modifiers
-NV_CONFTEST_TYPE_COMPILE_TESTS += dma_resv_add_fence
-NV_CONFTEST_TYPE_COMPILE_TESTS += dma_resv_reserve_fences
-NV_CONFTEST_TYPE_COMPILE_TESTS += reservation_object_reserve_shared_has_num_fences_arg
--- a/kernel-open/nvidia-drm/nvidia-drm.c
+++ b/kernel-open/nvidia-drm/nvidia-drm.c
@@ -45,6 +45,7 @@ int nv_drm_init(void)
        return -EINVAL;
    }

+    nvKms->setSuspendResumeCallback(nv_drm_suspend_resume);
    return nv_drm_probe_devices();
 #else
    return 0;
@@ -54,6 +55,7 @@ int nv_drm_init(void)
 void nv_drm_exit(void)
 {
 #if defined(NV_DRM_AVAILABLE)
+    nvKms->setSuspendResumeCallback(NULL);
    nv_drm_remove_devices();
 #endif
 }
--- a/kernel-open/nvidia-modeset/nv-kthread-q.c
+++ b/kernel-open/nvidia-modeset/nv-kthread-q.c
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2016 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2016-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -169,7 +169,6 @@ void nv_kthread_q_stop(nv_kthread_q_t *q)
 //
 // This function is never invoked when there is no NUMA preference (preferred
 // node is NUMA_NO_NODE).
-#if NV_KTHREAD_Q_SUPPORTS_AFFINITY() == 1
 static struct task_struct *thread_create_on_node(int (*threadfn)(void *data),
                                                 nv_kthread_q_t *q,
                                                 int preferred_node,
@@ -177,7 +176,7 @@ static struct task_struct *thread_create_on_node(int (*threadfn)(void *data),
 {

    unsigned i, j;
-    const static unsigned attempts = 3;
+    static const unsigned attempts = 3;
    struct task_struct *thread[3];

    for (i = 0;; i++) {
@@ -202,7 +201,7 @@ static struct task_struct *thread_create_on_node(int (*threadfn)(void *data),

        // Ran out of attempts - return thread even if its stack may not be
        // allocated on the preferred node
-        if ((i == (attempts - 1)))
+        if (i == (attempts - 1))
            break;

        // Get the NUMA node where the first page of the stack is resident. If
@@ -217,7 +216,6 @@ static struct task_struct *thread_create_on_node(int (*threadfn)(void *data),

    return thread[i];
 }
-#endif

 int nv_kthread_q_init_on_node(nv_kthread_q_t *q, const char *q_name, int preferred_node)
 {
@@ -231,11 +229,7 @@ int nv_kthread_q_init_on_node(nv_kthread_q_t *q, const char *q_name, int preferr
        q->q_kthread = kthread_create(_main_loop, q, q_name);
    }
    else {
-#if NV_KTHREAD_Q_SUPPORTS_AFFINITY() == 1
        q->q_kthread = thread_create_on_node(_main_loop, q, preferred_node, q_name);
-#else
-        return -ENOTSUPP;
-#endif
    }

    if (IS_ERR(q->q_kthread)) {
@@ -253,6 +247,11 @@ int nv_kthread_q_init_on_node(nv_kthread_q_t *q, const char *q_name, int preferr
    return 0;
 }

+int nv_kthread_q_init(nv_kthread_q_t *q, const char *qname)
+{
+    return nv_kthread_q_init_on_node(q, qname, NV_KTHREAD_NO_NODE);
+}
+
 // Returns true (non-zero) if the item was actually scheduled, and false if the
 // item was already pending in a queue.
 static int _raw_q_schedule(nv_kthread_q_t *q, nv_kthread_q_item_t *q_item)
@@ -307,7 +306,7 @@ static void _q_flush_function(void *args)
 static void _raw_q_flush(nv_kthread_q_t *q)
 {
    nv_kthread_q_item_t q_item;
-    DECLARE_COMPLETION(completion);
+    DECLARE_COMPLETION_ONSTACK(completion);

    nv_kthread_q_item_init(&q_item, _q_flush_function, &completion);

--- a/kernel-open/nvidia-modeset/nvidia-modeset-linux.c
+++ b/kernel-open/nvidia-modeset/nvidia-modeset-linux.c
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2015-21 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2015-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -34,10 +34,14 @@
 #include <linux/file.h>
 #include <linux/list.h>
 #include <linux/rwsem.h>
+#include <linux/freezer.h>
+#include <linux/poll.h>
+#include <linux/cdev.h>
+
+#include <acpi/video.h>

 #include "nvstatus.h"

-#include "nv-register-module.h"
 #include "nv-modeset-interface.h"
 #include "nv-kref.h"

@@ -50,8 +54,13 @@
 #include "nv-kthread-q.h"
 #include "nv-time.h"
 #include "nv-lock.h"
+#include "nv-chardev-numbers.h"

-#if !defined(CONFIG_RETPOLINE)
+/*
+ * Commit aefb2f2e619b ("x86/bugs: Rename CONFIG_RETPOLINE =>
+ * CONFIG_MITIGATION_RETPOLINE) in v6.8 renamed CONFIG_RETPOLINE.
+ */
+#if !defined(CONFIG_RETPOLINE) && !defined(CONFIG_MITIGATION_RETPOLINE)
 #include "nv-retpoline.h"
 #endif

@@ -59,6 +68,27 @@

 #define NVKMS_LOG_PREFIX "nvidia-modeset: "

+static bool output_rounding_fix = true;
+module_param_named(output_rounding_fix, output_rounding_fix, bool, 0400);
+
+static bool disable_hdmi_frl = false;
+module_param_named(disable_hdmi_frl, disable_hdmi_frl, bool, 0400);
+
+static bool disable_vrr_memclk_switch = false;
+module_param_named(disable_vrr_memclk_switch, disable_vrr_memclk_switch, bool, 0400);
+
+static bool hdmi_deepcolor = true;
+module_param_named(hdmi_deepcolor, hdmi_deepcolor, bool, 0400);
+
+static bool vblank_sem_control = true;
+module_param_named(vblank_sem_control, vblank_sem_control, bool, 0400);
+
+static bool opportunistic_display_sync = true;
+module_param_named(opportunistic_display_sync, opportunistic_display_sync, bool, 0400);
+
+static enum NvKmsDebugForceColorSpace debug_force_color_space = NVKMS_DEBUG_FORCE_COLOR_SPACE_NONE;
+module_param_named(debug_force_color_space, debug_force_color_space, uint, 0400);
+
 /* These parameters are used for fault injection tests.  Normally the defaults
 * should be used. */
 MODULE_PARM_DESC(fail_malloc, "Fail the Nth call to nvkms_alloc");
@@ -69,8 +99,70 @@ MODULE_PARM_DESC(malloc_verbose, "Report information about malloc calls on modul
 static bool malloc_verbose = false;
 module_param_named(malloc_verbose, malloc_verbose, bool, 0400);

+#if NVKMS_CONFIG_FILE_SUPPORTED
+/* This parameter is used to find the dpy override conf file */
+#define NVKMS_CONF_FILE_SPECIFIED (nvkms_conf != NULL)
+
+MODULE_PARM_DESC(config_file,
+                 "Path to the nvidia-modeset configuration file "
+                 "(default: disabled)");
+static char *nvkms_conf = NULL;
+module_param_named(config_file, nvkms_conf, charp, 0400);
+#endif
+
 static atomic_t nvkms_alloc_called_count;

+NvBool nvkms_output_rounding_fix(void)
+{
+    return output_rounding_fix;
+}
+
+NvBool nvkms_disable_hdmi_frl(void)
+{
+    return disable_hdmi_frl;
+}
+
+NvBool nvkms_disable_vrr_memclk_switch(void)
+{
+    return disable_vrr_memclk_switch;
+}
+
+NvBool nvkms_hdmi_deepcolor(void)
+{
+    return hdmi_deepcolor;
+}
+
+NvBool nvkms_vblank_sem_control(void)
+{
+    return vblank_sem_control;
+}
+
+NvBool nvkms_opportunistic_display_sync(void)
+{
+    return opportunistic_display_sync;
+}
+
+enum NvKmsDebugForceColorSpace nvkms_debug_force_color_space(void)
+{
+    if (debug_force_color_space >= NVKMS_DEBUG_FORCE_COLOR_SPACE_MAX) {
+        return NVKMS_DEBUG_FORCE_COLOR_SPACE_NONE;
+    }
+    return debug_force_color_space;
+}
+
+NvBool nvkms_kernel_supports_syncpts(void)
+{
+/*
+ * Note this only checks that the kernel has the prerequisite
+ * support for syncpts; callers must also check that the hardware
+ * supports syncpts.
+ */
+#if (defined(CONFIG_TEGRA_GRHOST) || defined(NV_LINUX_HOST1X_NEXT_H_PRESENT))
+    return NV_TRUE;
+#else
+    return NV_FALSE;
+#endif
+}

 #define NVKMS_SYNCPT_STUBS_NEEDED

@@ -78,110 +170,6 @@ static atomic_t nvkms_alloc_called_count;
 * NVKMS interface for nvhost unit for sync point APIs.
 *************************************************************************/

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
 #ifdef NVKMS_SYNCPT_STUBS_NEEDED
 /* Unsupported STUB for nvkms_syncpt APIs */
 NvBool nvkms_syncpt_op(
@@ -277,7 +265,24 @@ static inline int nvkms_read_trylock_pm_lock(void)

 static inline void nvkms_read_lock_pm_lock(void)
 {
-    down_read(&nvkms_pm_lock);
+    if ((current->flags & PF_NOFREEZE)) {
+        /*
+         * Non-freezable tasks (i.e. kthreads in this case) don't have to worry
+         * about being frozen during system suspend, but do need to block so
+         * that the CPU can go idle during s2idle. Do a normal uninterruptible
+         * blocking wait for the PM lock.
+         */
+        down_read(&nvkms_pm_lock);
+    } else {
+        /*
+         * For freezable tasks, make sure we give the kernel an opportunity to
+         * freeze if taking the PM lock fails.
+         */
+        while (!down_read_trylock(&nvkms_pm_lock)) {
+            try_to_freeze();
+            cond_resched();
+        }
+    }
 }

 static inline void nvkms_read_unlock_pm_lock(void)
@@ -409,7 +414,7 @@ NvU64 nvkms_get_usec(void)
    struct timespec64 ts;
    NvU64 ns;

-    ktime_get_real_ts64(&ts);
+    ktime_get_raw_ts64(&ts);

    ns = timespec64_to_ns(&ts);
    return ns / 1000;
@@ -523,6 +528,8 @@ nvkms_event_queue_changed(nvkms_per_open_handle_t *pOpenKernel,

 static void nvkms_suspend(NvU32 gpuId)
 {
+    nvKmsKapiSuspendResume(NV_TRUE /* suspend */);
+
    if (gpuId == 0) {
        nvkms_write_lock_pm_lock();
    }
@@ -541,6 +548,8 @@ static void nvkms_resume(NvU32 gpuId)
    if (gpuId == 0) {
        nvkms_write_unlock_pm_lock();
    }
+
+    nvKmsKapiSuspendResume(NV_FALSE /* suspend */);
 }


@@ -869,49 +878,6 @@ void nvkms_free_timer(nvkms_timer_handle_t *handle)
    timer->cancel = NV_TRUE;
 }

-void* nvkms_get_per_open_data(int fd)
-{
-    struct file *filp = fget(fd);
-    struct nvkms_per_open *popen = NULL;
-    dev_t rdev = 0;
-    void *data = NULL;
-
-    if (filp == NULL) {
-        return NULL;
-    }
-
-    if (filp->f_inode == NULL) {
-        goto done;
-    }
-    rdev = filp->f_inode->i_rdev;
-
-    if ((MAJOR(rdev) != NVKMS_MAJOR_DEVICE_NUMBER) ||
-        (MINOR(rdev) != NVKMS_MINOR_DEVICE_NUMBER)) {
-        goto done;
-    }
-
-    popen = filp->private_data;
-    if (popen == NULL) {
-        goto done;
-    }
-
-    data = popen->data;
-
-done:
-    /*
-     * fget() incremented the struct file's reference count, which
-     * needs to be balanced with a call to fput().  It is safe to
-     * decrement the reference count before returning
-     * filp->private_data because core NVKMS is currently holding the
-     * nvkms_lock, which prevents the nvkms_close() => nvKmsClose()
-     * call chain from freeing the file out from under the caller of
-     * nvkms_get_per_open_data().
-     */
-    fput(filp);
-
-    return data;
-}
-
 NvBool nvkms_fd_is_nvidia_chardev(int fd)
 {
    struct file *filp = fget(fd);
@@ -1053,6 +1019,12 @@ nvkms_register_backlight(NvU32 gpu_id, NvU32 display_id, void *drv_priv,
    struct nvkms_backlight_device *nvkms_bd = NULL;
    int i;

+#if defined(NV_ACPI_VIDEO_BACKLIGHT_USE_NATIVE)
+    if (!acpi_video_backlight_use_native()) {
+        return NULL;
+    }
+#endif
+
    gpu_info = nvkms_alloc(NV_MAX_GPUS * sizeof(*gpu_info), NV_TRUE);
    if (gpu_info == NULL) {
        return NULL;
@@ -1123,7 +1095,7 @@ static void nvkms_kapi_event_kthread_q_callback(void *arg)
    nvKmsKapiHandleEventQueueChange(device);
 }

-struct nvkms_per_open *nvkms_open_common(enum NvKmsClientType type,
+static struct nvkms_per_open *nvkms_open_common(enum NvKmsClientType type,
                                         struct NvKmsKapiDevice *device,
                                         int *status)
 {
@@ -1175,7 +1147,7 @@ failed:
    return NULL;
 }

-void nvkms_close_common(struct nvkms_per_open *popen)
+static void nvkms_close_pm_locked(struct nvkms_per_open *popen)
 {
    /*
     * Don't use down_interruptible(): we need to free resources
@@ -1213,13 +1185,13 @@ void nvkms_close_common(struct nvkms_per_open *popen)
    nvkms_free(popen, sizeof(*popen));
 }

-static void nvkms_close_deferred(void *data)
+static void nvkms_close_pm_unlocked(void *data)
 {
    struct nvkms_per_open *popen = data;

    nvkms_read_lock_pm_lock();

-    nvkms_close_common(popen);
+    nvkms_close_pm_locked(popen);

    nvkms_read_unlock_pm_lock();
 }
@@ -1227,18 +1199,18 @@ static void nvkms_close_deferred(void *data)
 static void nvkms_close_popen(struct nvkms_per_open *popen)
 {
    if (nvkms_read_trylock_pm_lock() == 0) {
-        nvkms_close_common(popen);
+        nvkms_close_pm_locked(popen);
        nvkms_read_unlock_pm_lock();
    } else {
        nv_kthread_q_item_init(&popen->deferred_close_q_item,
-                               nvkms_close_deferred,
+                               nvkms_close_pm_unlocked,
                               popen);
        nvkms_queue_work(&nvkms_deferred_close_kthread_q,
                         &popen->deferred_close_q_item);
    }
 }

-int nvkms_ioctl_common
+static int nvkms_ioctl_common
 (
    struct nvkms_per_open *popen,
    NvU32 cmd, NvU64 address, const size_t size
@@ -1284,7 +1256,27 @@ struct nvkms_per_open* nvkms_open_from_kapi

 void nvkms_close_from_kapi(struct nvkms_per_open *popen)
 {
-    nvkms_close_popen(popen);
+    nvkms_close_pm_unlocked(popen);
+}
+
+NvBool nvkms_ioctl_from_kapi_try_pmlock
+(
+    struct nvkms_per_open *popen,
+    NvU32 cmd, void *params_address, const size_t param_size
+)
+{
+    NvBool ret;
+
+    if (nvkms_read_trylock_pm_lock()) {
+        return NV_FALSE;
+    }
+
+    ret = nvkms_ioctl_common(popen,
+                             cmd,
+                             (NvU64)(NvUPtr)params_address, param_size) == 0;
+    nvkms_read_unlock_pm_lock();
+
+    return ret;
 }

 NvBool nvkms_ioctl_from_kapi
@@ -1443,32 +1435,127 @@ static void nvkms_proc_exit(void)
        return;
    }

-#if defined(NV_PROC_REMOVE_PRESENT)
    proc_remove(nvkms_proc_dir);
-#else
-    /*
-     * On kernel versions without proc_remove(), we need to explicitly
-     * remove each proc file beneath nvkms_proc_dir.
-     * nvkms_proc_init() only creates files directly under
-     * nvkms_proc_dir, so those are the only files we need to remove
-     * here: warn if there is any deeper directory nesting.
-     */
-    {
-        struct proc_dir_entry *entry = nvkms_proc_dir->subdir;
-
-        while (entry != NULL) {
-            struct proc_dir_entry *next = entry->next;
-            WARN_ON(entry->subdir != NULL);
-            remove_proc_entry(entry->name, entry->parent);
-            entry = next;
-        }
-    }
-
-    remove_proc_entry(nvkms_proc_dir->name, nvkms_proc_dir->parent);
-#endif /* NV_PROC_REMOVE_PRESENT */
 #endif /* CONFIG_PROC_FS */
 }

+/*************************************************************************
+ * NVKMS Config File Read
+ ************************************************************************/
+#if NVKMS_CONFIG_FILE_SUPPORTED
+static NvBool nvkms_fs_mounted(void)
+{
+    return current->fs != NULL;
+}
+
+static size_t nvkms_config_file_open
+(
+    char *fname,
+    char ** const buff
+)
+{
+    int i = 0;
+    struct file *file;
+    struct inode *file_inode;
+    size_t file_size = 0;
+    size_t read_size = 0;
+#if defined(NV_KERNEL_READ_HAS_POINTER_POS_ARG)
+    loff_t pos = 0;
+#endif
+
+    if (!nvkms_fs_mounted()) {
+        printk(KERN_ERR NVKMS_LOG_PREFIX "ERROR: Filesystems not mounted\n");
+        return 0;
+    }
+
+    file = filp_open(fname, O_RDONLY, 0);
+    if (file == NULL || IS_ERR(file)) {
+        printk(KERN_WARNING NVKMS_LOG_PREFIX "WARNING: Failed to open %s\n",
+               fname);
+        return 0;
+    }
+
+    file_inode = file->f_inode;
+    if (file_inode == NULL || IS_ERR(file_inode)) {
+        printk(KERN_WARNING NVKMS_LOG_PREFIX "WARNING: Inode is invalid\n");
+        goto done;
+    }
+    file_size = file_inode->i_size;
+    if (file_size > NVKMS_READ_FILE_MAX_SIZE) {
+        printk(KERN_WARNING NVKMS_LOG_PREFIX "WARNING: File exceeds maximum size\n");
+        goto done;
+    }
+
+    *buff = nvkms_alloc(file_size, NV_FALSE);
+    if (*buff == NULL) {
+        printk(KERN_WARNING NVKMS_LOG_PREFIX "WARNING: Out of memory\n");
+        goto done;
+    }
+
+    /*
+     * TODO: Once we have access to GPL symbols, this can be replaced with
+     * kernel_read_file for kernels >= 4.6
+     */
+    while ((read_size < file_size) && (i++ < NVKMS_READ_FILE_MAX_LOOPS)) {
+#if defined(NV_KERNEL_READ_HAS_POINTER_POS_ARG)
+        ssize_t ret = kernel_read(file, *buff + read_size,
+                                  file_size - read_size, &pos);
+#else
+        ssize_t ret = kernel_read(file, read_size,
+                                  *buff + read_size,
+                                  file_size - read_size);
+#endif
+        if (ret <= 0) {
+            break;
+        }
+        read_size += ret;
+    }
+
+    if (read_size != file_size) {
+        printk(KERN_WARNING NVKMS_LOG_PREFIX "WARNING: Failed to read %s\n",
+               fname);
+        goto done;
+    }
+
+    filp_close(file, current->files);
+    return file_size;
+
+done:
+    nvkms_free(*buff, file_size);
+    filp_close(file, current->files);
+    return 0;
+}
+
+/* must be called with nvkms_lock locked */
+static void nvkms_read_config_file_locked(void)
+{
+    char *buffer = NULL;
+    size_t buf_size = 0;
+
+    /* only read the config file if the kernel parameter is set */
+    if (!NVKMS_CONF_FILE_SPECIFIED) {
+        return;
+    }
+
+    buf_size = nvkms_config_file_open(nvkms_conf, &buffer);
+
+    if (buf_size == 0) {
+        return;
+    }
+
+    if (nvKmsReadConf(buffer, buf_size, nvkms_config_file_open)) {
+        printk(KERN_INFO NVKMS_LOG_PREFIX "Successfully read %s\n",
+               nvkms_conf);
+    }
+
+    nvkms_free(buffer, buf_size);
+}
+#else
+static void nvkms_read_config_file_locked(void)
+{
+}
+#endif
+
 /*************************************************************************
 * NVKMS KAPI functions
 ************************************************************************/
@@ -1482,6 +1569,48 @@ NvBool nvKmsKapiGetFunctionsTable
 }
 EXPORT_SYMBOL(nvKmsKapiGetFunctionsTable);

+NvU32 nvKmsKapiF16ToF32(NvU16 a)
+{
+    return nvKmsKapiF16ToF32Internal(a);
+}
+EXPORT_SYMBOL(nvKmsKapiF16ToF32);
+
+NvU16 nvKmsKapiF32ToF16(NvU32 a)
+{
+    return nvKmsKapiF32ToF16Internal(a);
+}
+EXPORT_SYMBOL(nvKmsKapiF32ToF16);
+
+NvU32 nvKmsKapiF32Mul(NvU32 a, NvU32 b)
+{
+    return nvKmsKapiF32MulInternal(a, b);
+}
+EXPORT_SYMBOL(nvKmsKapiF32Mul);
+
+NvU32 nvKmsKapiF32Div(NvU32 a, NvU32 b)
+{
+    return nvKmsKapiF32DivInternal(a, b);
+}
+EXPORT_SYMBOL(nvKmsKapiF32Div);
+
+NvU32 nvKmsKapiF32Add(NvU32 a, NvU32 b)
+{
+    return nvKmsKapiF32AddInternal(a, b);
+}
+EXPORT_SYMBOL(nvKmsKapiF32Add);
+
+NvU32 nvKmsKapiF32ToUI32RMinMag(NvU32 a, NvBool exact)
+{
+    return nvKmsKapiF32ToUI32RMinMagInternal(a, exact);
+}
+EXPORT_SYMBOL(nvKmsKapiF32ToUI32RMinMag);
+
+NvU32 nvKmsKapiUI32ToF32(NvU32 a)
+{
+    return nvKmsKapiUI32ToF32Internal(a);
+}
+EXPORT_SYMBOL(nvKmsKapiUI32ToF32);
+
 /*************************************************************************
 * File operation callback functions.
 *************************************************************************/
@@ -1562,6 +1691,12 @@ static int nvkms_ioctl(struct inode *inode, struct file *filp,
    return status;
 }

+static long nvkms_unlocked_ioctl(struct file *filp, unsigned int cmd,
+                                 unsigned long arg)
+{
+    return nvkms_ioctl(filp->f_inode, filp, cmd, arg);
+}
+
 static unsigned int nvkms_poll(struct file *filp, poll_table *wait)
 {
    unsigned int mask = 0;
@@ -1589,17 +1724,73 @@ static unsigned int nvkms_poll(struct file *filp, poll_table *wait)
 * Module loading support code.
 *************************************************************************/

-static nvidia_module_t nvidia_modeset_module = {
+#define NVKMS_RDEV  (MKDEV(NV_MAJOR_DEVICE_NUMBER, \
+                           NV_MINOR_DEVICE_NUMBER_MODESET_DEVICE))
+
+static struct file_operations nvkms_fops = {
    .owner       = THIS_MODULE,
-    .module_name = "nvidia-modeset",
-    .instance    = 1, /* minor number: 255-1=254 */
-    .open        = nvkms_open,
-    .close       = nvkms_close,
-    .mmap        = nvkms_mmap,
-    .ioctl       = nvkms_ioctl,
    .poll        = nvkms_poll,
+    .unlocked_ioctl = nvkms_unlocked_ioctl,
+#if NVCPU_IS_X86_64 || NVCPU_IS_AARCH64
+    .compat_ioctl = nvkms_unlocked_ioctl,
+#endif
+    .mmap        = nvkms_mmap,
+    .open        = nvkms_open,
+    .release     = nvkms_close,
 };

+static struct cdev nvkms_device_cdev;
+
+static int __init nvkms_register_chrdev(void)
+{
+    int ret;
+
+    ret = register_chrdev_region(NVKMS_RDEV, 1, "nvidia-modeset");
+    if (ret < 0) {
+        return ret;
+    }
+
+    cdev_init(&nvkms_device_cdev, &nvkms_fops);
+    ret = cdev_add(&nvkms_device_cdev, NVKMS_RDEV, 1);
+    if (ret < 0) {
+        unregister_chrdev_region(NVKMS_RDEV, 1);
+        return ret;
+    }
+
+    return ret;
+}
+
+static void nvkms_unregister_chrdev(void)
+{
+    cdev_del(&nvkms_device_cdev);
+    unregister_chrdev_region(NVKMS_RDEV, 1);
+}
+
+void* nvkms_get_per_open_data(int fd)
+{
+    struct file *filp = fget(fd);
+    void *data = NULL;
+
+    if (filp) {
+        if (filp->f_op == &nvkms_fops && filp->private_data) {
+            struct nvkms_per_open *popen = filp->private_data;
+            data = popen->data;
+        }
+
+        /*
+         * fget() incremented the struct file's reference count, which needs to
+         * be balanced with a call to fput().  It is safe to decrement the
+         * reference count before returning filp->private_data because core
+         * NVKMS is currently holding the nvkms_lock, which prevents the
+         * nvkms_close() => nvKmsClose() call chain from freeing the file out
+         * from under the caller of nvkms_get_per_open_data().
+         */
+        fput(filp);
+    }
+
+    return data;
+}
+
 static int __init nvkms_init(void)
 {
    int ret;
@@ -1630,28 +1821,29 @@ static int __init nvkms_init(void)
    INIT_LIST_HEAD(&nvkms_timers.list);
    spin_lock_init(&nvkms_timers.lock);

-    ret = nvidia_register_module(&nvidia_modeset_module);
-
+    ret = nvkms_register_chrdev();
    if (ret != 0) {
-        goto fail_register_module;
+        goto fail_register_chrdev;
    }

    down(&nvkms_lock);
    if (!nvKmsModuleLoad()) {
        ret = -ENOMEM;
    }
-    up(&nvkms_lock);
    if (ret != 0) {
+        up(&nvkms_lock);
        goto fail_module_load;
    }
+    nvkms_read_config_file_locked();
+    up(&nvkms_lock);

    nvkms_proc_init();

    return 0;

 fail_module_load:
-    nvidia_unregister_module(&nvidia_modeset_module);
-fail_register_module:
+    nvkms_unregister_chrdev();
+fail_register_chrdev:
    nv_kthread_q_stop(&nvkms_deferred_close_kthread_q);
 fail_deferred_close_kthread:
    nv_kthread_q_stop(&nvkms_kthread_q);
@@ -1715,7 +1907,7 @@ restart:
    nv_kthread_q_stop(&nvkms_deferred_close_kthread_q);
    nv_kthread_q_stop(&nvkms_kthread_q);

-    nvidia_unregister_module(&nvidia_modeset_module);
+    nvkms_unregister_chrdev();
    nvkms_free_rm();

    if (malloc_verbose) {
@@ -1727,16 +1919,7 @@ restart:
 module_init(nvkms_init);
 module_exit(nvkms_exit);

-#if defined(MODULE_LICENSE)
-
  MODULE_LICENSE("Dual MIT/GPL");

-
-
-#endif
-#if defined(MODULE_INFO)
-  MODULE_INFO(supported, "external");
-#endif
-#if defined(MODULE_VERSION)
-  MODULE_VERSION(NV_VERSION_STRING);
-#endif
+MODULE_INFO(supported, "external");
+MODULE_VERSION(NV_VERSION_STRING);
--- a/kernel-open/nvidia-modeset/nvidia-modeset-os-interface.h
+++ b/kernel-open/nvidia-modeset/nvidia-modeset-os-interface.h
@@ -40,17 +40,39 @@
 #include "nv_stdarg.h"

 enum NvKmsSyncPtOp {
+    /*
+     * Call into Tegra's kernel nvhost driver, and allocate a syncpoint that can
+     * be exclusively used by the caller. Internally, this operation will call
+     * get() to set the initial refcount of the syncpoint to 1.
+     */
    NVKMS_SYNCPT_OP_ALLOC,
-    NVKMS_SYNCPT_OP_GET,
+    /*
+     * Decrease the refcount of an already allocated syncpoint. Once the
+     * refcount drops to 0, the syncpoint will be returned to the free pool that
+     * nvhost manages, so PUT can also be used to balance out an ALLOC.
+     */
    NVKMS_SYNCPT_OP_PUT,
-    NVKMS_SYNCPT_OP_INCR_MAX,
-    NVKMS_SYNCPT_OP_CPU_INCR,
+    /*
+     * Extract syncpt id and thresh from the sync-file file descriptor
+     */
    NVKMS_SYNCPT_OP_FD_TO_ID_AND_THRESH,
+    /*
+     * Create dma-fence from syncpt id and thresh value and create sync_file
+     * file descriptor for the dma-fence handle created.
+     */
    NVKMS_SYNCPT_OP_ID_AND_THRESH_TO_FD,
+    /*
+     * read syncpt minimum value of given syncpt
+     */
    NVKMS_SYNCPT_OP_READ_MINVAL,
-    NVKMS_SYNCPT_OP_READ_MAXVAL,
-    NVKMS_SYNCPT_OP_SET_MIN_EQ_MAX,
-    NVKMS_SYNCPT_OP_SET_MAXVAL,
+};
+
+enum NvKmsDebugForceColorSpace {
+    NVKMS_DEBUG_FORCE_COLOR_SPACE_NONE,
+    NVKMS_DEBUG_FORCE_COLOR_SPACE_RGB,
+    NVKMS_DEBUG_FORCE_COLOR_SPACE_YUV444,
+    NVKMS_DEBUG_FORCE_COLOR_SPACE_YUV422,
+    NVKMS_DEBUG_FORCE_COLOR_SPACE_MAX,
 };

 typedef struct {
@@ -60,24 +82,10 @@ typedef struct {
        NvU32 id;                       /*  out  */
    } alloc;

-    struct {
-        NvU32 id;                       /*  in   */
-    } get;
-
    struct {
        NvU32 id;                       /*  in   */
    } put;

-    struct {
-        NvU32 id;                       /*  in   */
-        NvU32 incr;                     /*  in   */
-        NvU32 value;                    /*  out  */
-    } incr_max;
-
-    struct {
-        NvU32 id;                       /*  in   */
-    } cpu_incr;
-
    struct {
        NvS32 fd;                       /*  in   */
        NvU32 id;                       /*  out  */
@@ -94,22 +102,15 @@ typedef struct {
        NvU32 id;                       /*  in   */
        NvU32 minval;                   /*  out  */
    } read_minval;
-
-    struct {
-        NvU32 id;                       /*  in   */
-        NvU32 maxval;                   /*  out  */
-    } read_maxval;
-
-    struct {
-        NvU32 id;                       /*  in   */
-    } set_min_eq_max;
-
-    struct {
-        NvU32 id;                       /*  in   */
-        NvU32 val;                      /*  in   */
-    } set_maxval;
 } NvKmsSyncPtOpParams;

+NvBool nvkms_output_rounding_fix(void);
+NvBool nvkms_disable_hdmi_frl(void);
+NvBool nvkms_disable_vrr_memclk_switch(void);
+NvBool nvkms_hdmi_deepcolor(void);
+NvBool nvkms_vblank_sem_control(void);
+NvBool nvkms_opportunistic_display_sync(void);
+enum NvKmsDebugForceColorSpace nvkms_debug_force_color_space(void);

 void   nvkms_call_rm    (void *ops);
 void*  nvkms_alloc      (size_t size,
@@ -312,6 +313,11 @@ NvU32 nvkms_enumerate_gpus(nv_gpu_info_t *gpu_info);

 NvBool nvkms_allow_write_combining(void);

+/*!
+ * Check if OS supports syncpoints.
+ */
+NvBool nvkms_kernel_supports_syncpts(void);
+
 /*!
 * Checks whether the fd is associated with an nvidia character device.
 */
@@ -336,6 +342,16 @@ NvBool nvkms_ioctl_from_kapi
    NvU32 cmd, void *params_address, const size_t params_size
 );

+/*!
+ * Like nvkms_ioctl_from_kapi, but return NV_FALSE instead of waiting if the
+ * power management read lock cannot be acquired.
+ */
+NvBool nvkms_ioctl_from_kapi_try_pmlock
+(
+    struct nvkms_per_open *popen,
+    NvU32 cmd, void *params_address, const size_t params_size
+);
+
 /*!
 * APIs for locking.
 */
--- a/kernel-open/nvidia-modeset/nvidia-modeset.Kbuild
+++ b/kernel-open/nvidia-modeset/nvidia-modeset.Kbuild
@@ -58,6 +58,18 @@ nvidia-modeset-y += $(NVIDIA_MODESET_BINARY_OBJECT_O)
 NVIDIA_MODESET_CFLAGS += -I$(src)/nvidia-modeset
 NVIDIA_MODESET_CFLAGS += -UDEBUG -U_DEBUG -DNDEBUG -DNV_BUILD_MODULE_INSTANCES=0

+# Some Android kernels prohibit driver use of filesystem functions like
+# filp_open() and kernel_read().  Disable the NVKMS_CONFIG_FILE_SUPPORTED
+# functionality that uses those functions when building for Android.
+
+PLATFORM_IS_ANDROID ?= 0
+
+ifeq ($(PLATFORM_IS_ANDROID),1)
+  NVIDIA_MODESET_CFLAGS += -DNVKMS_CONFIG_FILE_SUPPORTED=0
+else
+  NVIDIA_MODESET_CFLAGS += -DNVKMS_CONFIG_FILE_SUPPORTED=1
+endif
+
 $(call ASSIGN_PER_OBJ_CFLAGS, $(NVIDIA_MODESET_OBJECTS), $(NVIDIA_MODESET_CFLAGS))


@@ -85,15 +97,12 @@ $(obj)/$(NVIDIA_MODESET_INTERFACE): $(addprefix $(obj)/,$(NVIDIA_MODESET_OBJECTS

 NV_OBJECTS_DEPEND_ON_CONFTEST += $(NVIDIA_MODESET_OBJECTS)

-NV_CONFTEST_TYPE_COMPILE_TESTS += file_operations
-NV_CONFTEST_TYPE_COMPILE_TESTS += node_states_n_memory
 NV_CONFTEST_TYPE_COMPILE_TESTS += timespec64
 NV_CONFTEST_TYPE_COMPILE_TESTS += proc_ops
 NV_CONFTEST_FUNCTION_COMPILE_TESTS += pde_data
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += proc_remove
 NV_CONFTEST_FUNCTION_COMPILE_TESTS += timer_setup
-NV_CONFTEST_FUNCTION_COMPILE_TESTS += kthread_create_on_node
 NV_CONFTEST_FUNCTION_COMPILE_TESTS += list_is_first
 NV_CONFTEST_FUNCTION_COMPILE_TESTS += ktime_get_real_ts64
 NV_CONFTEST_FUNCTION_COMPILE_TESTS += ktime_get_raw_ts64
-NV_CONFTEST_SYMBOL_COMPILE_TESTS += is_export_symbol_present_kthread_create_on_node
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += acpi_video_backlight_use_native
+NV_CONFTEST_FUNCTION_COMPILE_TESTS += kernel_read_has_pointer_pos_arg
--- a/kernel-open/nvidia-modeset/nvkms.h
+++ b/kernel-open/nvidia-modeset/nvkms.h
@@ -42,6 +42,20 @@ typedef void nvkms_procfs_proc_t(void *data,
                                 char *buffer, size_t size,
                                 nvkms_procfs_out_string_func_t *outString);

+/* max number of loops to prevent hanging the kernel if an edge case is hit */
+#define NVKMS_READ_FILE_MAX_LOOPS 1000
+/* max size for any file read by the config system */
+#define NVKMS_READ_FILE_MAX_SIZE  8192
+
+/*
+ * The read file callback should allocate a buffer pointed to by *buff, fill it
+ * with the contents of fname, and return the size of the buffer. Buffer is not
+ * guaranteed to be null-terminated. The caller is responsible for freeing the
+ * buffer with nvkms_free, not nvFree.
+ */
+typedef size_t nvkms_config_read_file_func_t(char *fname,
+                                             char ** const buff);
+
 typedef struct {
    const char *name;
    nvkms_procfs_proc_t *func;
@@ -52,6 +66,8 @@ enum NvKmsClientType {
    NVKMS_CLIENT_KERNEL_SPACE,
 };

+struct NvKmsPerOpenDev;
+
 NvBool nvKmsIoctl(
    void *pOpenVoid,
    NvU32 cmd,
@@ -74,6 +90,9 @@ void nvKmsResume(NvU32 gpuId);

 void nvKmsGetProcFiles(const nvkms_procfs_file_t **ppProcFiles);

+NvBool nvKmsReadConf(const char *buff, size_t size,
+                     nvkms_config_read_file_func_t readfile);
+
 void nvKmsKapiHandleEventQueueChange
 (
    struct NvKmsKapiDevice *device
@@ -84,7 +103,25 @@ NvBool nvKmsKapiGetFunctionsTableInternal
    struct NvKmsKapiFunctionsTable *funcsTable
 );

+void nvKmsKapiSuspendResume(NvBool suspend);
+
 NvBool nvKmsGetBacklight(NvU32 display_id, void *drv_priv, NvU32 *brightness);
 NvBool nvKmsSetBacklight(NvU32 display_id, void *drv_priv, NvU32 brightness);

+NvBool nvKmsOpenDevHasSubOwnerPermissionOrBetter(const struct NvKmsPerOpenDev *pOpenDev);
+
+NvU32 nvKmsKapiF16ToF32Internal(NvU16 a);
+
+NvU16 nvKmsKapiF32ToF16Internal(NvU32 a);
+
+NvU32 nvKmsKapiF32MulInternal(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32DivInternal(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32AddInternal(NvU32 a, NvU32 b);
+
+NvU32 nvKmsKapiF32ToUI32RMinMagInternal(NvU32 a, NvBool exact);
+
+NvU32 nvKmsKapiUI32ToF32Internal(NvU32 a);
+
 #endif /* __NV_KMS_H__ */
--- a/kernel-open/nvidia-peermem/nv-p2p.h
+++ b/kernel-open/nvidia-peermem/nv-p2p.h
@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2011-2016 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-FileCopyrightText: Copyright (c) 2011-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: MIT
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
@@ -94,11 +94,10 @@ struct nvidia_p2p_params {
 } nvidia_p2p_params_t;

 /*
- * Capability flag for users to detect
+ * Macro for users to detect
 * driver support for persistent pages.
 */
-extern int nvidia_p2p_cap_persistent_pages;
-#define NVIDIA_P2P_CAP_PERSISTENT_PAGES
+#define NVIDIA_P2P_CAP_GET_PAGES_PERSISTENT_API

 /*
 * This API is not supported.
@@ -173,15 +172,6 @@ struct nvidia_p2p_page_table {
 *   A pointer to the function to be invoked when the pages
 *   underlying the virtual address range are freed
 *   implicitly.
- *   If NULL, persistent pages will be returned.
- *   This means the pages underlying the range of GPU virtual memory
- *   will persist until explicitly freed by nvidia_p2p_put_pages().
- *   Persistent GPU memory mappings are not supported on PowerPC,
-
-
-
- *   MIG-enabled devices and vGPU.
-
 * @param[in]     data
 *   A non-NULL opaque pointer to private data to be passed to the
 *   callback function.
@@ -194,12 +184,48 @@ struct nvidia_p2p_page_table {
 *     insufficient resources were available to complete the operation.
 *   -EIO         if an unknown error occurred.
 */
-int nvidia_p2p_get_pages(uint64_t p2p_token, uint32_t va_space,
-        uint64_t virtual_address,
+int nvidia_p2p_get_pages( uint64_t p2p_token, uint32_t va_space,
+        uint64_t virtual_address, uint64_t length,
+        struct nvidia_p2p_page_table **page_table,
+        void (*free_callback)(void *data), void *data);
+
+/*
+ * @brief
+ *   Pin and make the pages underlying a range of GPU virtual memory
+ *   accessible to a third-party device. The pages will persist until
+ *   explicitly freed by nvidia_p2p_put_pages_persistent().
+ *
+ *   Persistent GPU memory mappings are not supported on PowerPC,
+ *   MIG-enabled devices and vGPU.
+ *
+ *   This API only supports pinned, GPU-resident memory, such as that provided
+ *   by cudaMalloc().
+ *
+ *   This API may sleep.
+ *
+ * @param[in]     virtual_address
+ *   The start address in the specified virtual address space.
+ *   Address must be aligned to the 64KB boundary.
+ * @param[in]     length
+ *   The length of the requested P2P mapping.
+ *   Length must be a multiple of 64KB.
+ * @param[out]    page_table
+ *   A pointer to an array of structures with P2P PTEs.
+ * @param[in]     flags
+ *   Must be set to zero for now.
+ *
+ * @return
+ *    0           upon successful completion.
+ *   -EINVAL      if an invalid argument was supplied.
+ *   -ENOTSUPP    if the requested operation is not supported.
+ *   -ENOMEM      if the driver failed to allocate memory or if
+ *     insufficient resources were available to complete the operation.
+ *   -EIO         if an unknown error occurred.
+ */
+int nvidia_p2p_get_pages_persistent(uint64_t virtual_address,
        uint64_t length,
        struct nvidia_p2p_page_table **page_table,
-        void (*free_callback)(void *data),
-        void *data);
+        uint32_t flags);

 #define NVIDIA_P2P_DMA_MAPPING_VERSION   0x00020003

@@ -272,6 +298,8 @@ int nvidia_p2p_dma_unmap_pages(struct pci_dev *peer,
 *   Release a set of pages previously made accessible to
 *   a third-party device.
 *
+ *   This API may sleep.
+ *
 * @param[in]     p2p_token
 *   A token that uniquely identifies the P2P mapping.
 * @param[in]     va_space
@@ -286,10 +314,33 @@ int nvidia_p2p_dma_unmap_pages(struct pci_dev *peer,
 *   -EINVAL      if an invalid argument was supplied.
 *   -EIO         if an unknown error occurred.
 */
-int nvidia_p2p_put_pages(uint64_t p2p_token, uint32_t va_space,
-        uint64_t virtual_address,
+int nvidia_p2p_put_pages(uint64_t p2p_token,
+        uint32_t va_space, uint64_t virtual_address,
        struct nvidia_p2p_page_table *page_table);

+/*
+ * @brief
+ *   Release a set of persistent pages previously made accessible to
+ *   a third-party device.
+ *
+ *   This API may sleep.
+ *
+ * @param[in]     virtual_address
+ *   The start address in the specified virtual address space.
+ * @param[in]     page_table
+ *   A pointer to the array of structures with P2P PTEs.
+ * @param[in]     flags
+ *   Must be set to zero for now.
+ *
+ * @return
+ *    0           upon successful completion.
+ *   -EINVAL      if an invalid argument was supplied.
+ *   -EIO         if an unknown error occurred.
+ */
+int nvidia_p2p_put_pages_persistent(uint64_t virtual_address,
+        struct nvidia_p2p_page_table *page_table,
+        uint32_t flags);
+
 /*
 * @brief
 *    Free a third-party P2P page table. (This function is a no-op.)
--- a/kernel-open/nvidia-peermem/nvidia-peermem.Kbuild
+++ b/kernel-open/nvidia-peermem/nvidia-peermem.Kbuild
@@ -30,8 +30,18 @@ NVIDIA_PEERMEM_CFLAGS += -UDEBUG -U_DEBUG -DNDEBUG -DNV_BUILD_MODULE_INSTANCES=0
 # MOFED's Module.symvers is needed for the build
 # to find the additional ib_* symbols.
 #
+# Also, MOFED doesn't use kbuild ARCH names.
+# So adapt OFA_ARCH to match MOFED's conventions.
+#
+ifeq ($(ARCH), arm64)
+    OFA_ARCH := aarch64
+else ifeq ($(ARCH), powerpc)
+    OFA_ARCH := ppc64le
+else
+    OFA_ARCH := $(ARCH)
+endif
 OFA_DIR := /usr/src/ofa_kernel
-OFA_CANDIDATES = $(OFA_DIR)/$(ARCH)/$(KERNELRELEASE) $(OFA_DIR)/$(KERNELRELEASE) $(OFA_DIR)/default /var/lib/dkms/mlnx-ofed-kernel
+OFA_CANDIDATES = $(OFA_DIR)/$(OFA_ARCH)/$(KERNELRELEASE) $(OFA_DIR)/$(KERNELRELEASE) $(OFA_DIR)/default /var/lib/dkms/mlnx-ofed-kernel
 MLNX_OFED_KERNEL := $(shell for d in $(OFA_CANDIDATES); do \
                              if [ -d "$$d" ]; then \
                                echo "$$d"; \
--- a/kernel-open/nvidia-peermem/nvidia-peermem.c
+++ b/kernel-open/nvidia-peermem/nvidia-peermem.c
@@ -1,20 +1,25 @@
-/* SPDX-License-Identifier: Linux-OpenIB */
 /*
 * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
 * Copyright (c) 2007, 2008 Mellanox Technologies. All rights reserved.
 *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
 *
- *  - Redistributions of source code must retain the above
- *    copyright notice, this list of conditions and the following
- *    disclaimer.
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
 *
- *  - Redistributions in binary form must reproduce the above
- *    copyright notice, this list of conditions and the following
- *    disclaimer in the documentation and/or other materials
- *    provided with the distribution.
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
@@ -43,7 +48,9 @@

 MODULE_AUTHOR("Yishai Hadas");
 MODULE_DESCRIPTION("NVIDIA GPU memory plug-in");
-MODULE_LICENSE("Linux-OpenIB");
+
+MODULE_LICENSE("Dual BSD/GPL");
+
 MODULE_VERSION(DRV_VERSION);
 enum {
        NV_MEM_PEERDIRECT_SUPPORT_DEFAULT = 0,
@@ -53,7 +60,20 @@ static int peerdirect_support = NV_MEM_PEERDIRECT_SUPPORT_DEFAULT;
 module_param(peerdirect_support, int, S_IRUGO);
 MODULE_PARM_DESC(peerdirect_support, "Set level of support for Peer-direct, 0 [default] or 1 [legacy, for example MLNX_OFED 4.9 LTS]");

-#define peer_err(FMT, ARGS...) printk(KERN_ERR "nvidia-peermem" " %s:%d " FMT, __FUNCTION__, __LINE__, ## ARGS)
+enum {
+        NV_MEM_PERSISTENT_API_SUPPORT_LEGACY = 0,
+        NV_MEM_PERSISTENT_API_SUPPORT_DEFAULT = 1,
+};
+static int persistent_api_support = NV_MEM_PERSISTENT_API_SUPPORT_DEFAULT;
+module_param(persistent_api_support, int, S_IRUGO);
+MODULE_PARM_DESC(persistent_api_support, "Set level of support for persistent APIs, 0 [legacy] or 1 [default]");
+
+#define peer_err(FMT, ARGS...) printk(KERN_ERR "nvidia-peermem" " %s:%d ERROR " FMT, __FUNCTION__, __LINE__, ## ARGS)
+#ifdef NV_MEM_DEBUG
+#define peer_trace(FMT, ARGS...) printk(KERN_DEBUG "nvidia-peermem" " %s:%d TRACE " FMT, __FUNCTION__, __LINE__, ## ARGS)
+#else
+#define peer_trace(FMT, ARGS...) do {} while (0)
+#endif

 #if defined(NV_MLNX_IB_PEER_MEM_SYMBOLS_PRESENT)

@@ -74,7 +94,10 @@ invalidate_peer_memory mem_invalidate_callback;
 static void *reg_handle = NULL;
 static void *reg_handle_nc = NULL;

+#define NV_MEM_CONTEXT_MAGIC ((u64)0xF1F4F1D0FEF0DAD0ULL)
+
 struct nv_mem_context {
+    u64 pad1;
    struct nvidia_p2p_page_table *page_table;
    struct nvidia_p2p_dma_mapping *dma_mapping;
    u64 core_context;
@@ -86,8 +109,22 @@ struct nv_mem_context {
    struct task_struct *callback_task;
    int sg_allocated;
    struct sg_table sg_head;
+    u64 pad2;
 };

+#define NV_MEM_CONTEXT_CHECK_OK(MC) ({                                  \
+    struct nv_mem_context *mc = (MC);                                   \
+    int rc = ((0 != mc) &&                                              \
+              (READ_ONCE(mc->pad1) == NV_MEM_CONTEXT_MAGIC) &&          \
+              (READ_ONCE(mc->pad2) == NV_MEM_CONTEXT_MAGIC));           \
+    if (!rc) {                                                          \
+        peer_trace("invalid nv_mem_context=%px pad1=%016llx pad2=%016llx\n", \
+                   mc,                                                  \
+                   mc?mc->pad1:0,                                       \
+                   mc?mc->pad2:0);                                      \
+    }                                                                   \
+    rc;                                                                 \
+})

 static void nv_get_p2p_free_callback(void *data)
 {
@@ -97,8 +134,9 @@ static void nv_get_p2p_free_callback(void *data)
    struct nvidia_p2p_dma_mapping *dma_mapping = NULL;

    __module_get(THIS_MODULE);
-    if (!nv_mem_context) {
-        peer_err("nv_get_p2p_free_callback -- invalid nv_mem_context\n");
+
+    if (!NV_MEM_CONTEXT_CHECK_OK(nv_mem_context)) {
+        peer_err("detected invalid context, skipping further processing\n");
        goto out;
    }

@@ -169,9 +207,11 @@ static int nv_mem_acquire(unsigned long addr, size_t size, void *peer_mem_privat
        /* Error case handled as not mine */
        return 0;

+    nv_mem_context->pad1 = NV_MEM_CONTEXT_MAGIC;
    nv_mem_context->page_virt_start = addr & GPU_PAGE_MASK;
    nv_mem_context->page_virt_end   = (addr + size + GPU_PAGE_SIZE - 1) & GPU_PAGE_MASK;
    nv_mem_context->mapped_size  = nv_mem_context->page_virt_end - nv_mem_context->page_virt_start;
+    nv_mem_context->pad2 = NV_MEM_CONTEXT_MAGIC;

    ret = nvidia_p2p_get_pages(0, 0, nv_mem_context->page_virt_start, nv_mem_context->mapped_size,
                               &nv_mem_context->page_table, nv_mem_dummy_callback, nv_mem_context);
@@ -195,6 +235,7 @@ static int nv_mem_acquire(unsigned long addr, size_t size, void *peer_mem_privat
    return 1;

 err:
+    memset(nv_mem_context, 0, sizeof(*nv_mem_context));
    kfree(nv_mem_context);

    /* Error case handled as not mine */
@@ -249,8 +290,8 @@ static int nv_dma_map(struct sg_table *sg_head, void *context,
    nv_mem_context->sg_allocated = 1;
    for_each_sg(sg_head->sgl, sg, nv_mem_context->npages, i) {
        sg_set_page(sg, NULL, nv_mem_context->page_size, 0);
-        sg->dma_address = dma_mapping->dma_addresses[i];
-        sg->dma_length = nv_mem_context->page_size;
+        sg_dma_address(sg) = dma_mapping->dma_addresses[i];
+        sg_dma_len(sg) = nv_mem_context->page_size;
    }
    nv_mem_context->sg_head = *sg_head;
    *nmap = nv_mem_context->npages;
@@ -284,8 +325,9 @@ out:
    return 0;
 }

-
-static void nv_mem_put_pages(struct sg_table *sg_head, void *context)
+static void nv_mem_put_pages_common(int nc,
+                                    struct sg_table *sg_head,
+                                    void *context)
 {
    int ret = 0;
    struct nv_mem_context *nv_mem_context =
@@ -302,8 +344,18 @@ static void nv_mem_put_pages(struct sg_table *sg_head, void *context)
    if (nv_mem_context->callback_task == current)
        return;

-    ret = nvidia_p2p_put_pages(0, 0, nv_mem_context->page_virt_start,
-                               nv_mem_context->page_table);
+    if (nc) {
+#ifdef NVIDIA_P2P_CAP_GET_PAGES_PERSISTENT_API
+        ret = nvidia_p2p_put_pages_persistent(nv_mem_context->page_virt_start,
+                                              nv_mem_context->page_table, 0);
+#else
+        ret = nvidia_p2p_put_pages(0, 0, nv_mem_context->page_virt_start,
+                                   nv_mem_context->page_table);
+#endif
+    } else {
+        ret = nvidia_p2p_put_pages(0, 0, nv_mem_context->page_virt_start,
+                                   nv_mem_context->page_table);
+    }

 #ifdef _DEBUG_ONLY_
    /* Here we expect an error in real life cases that should be ignored - not printed.
@@ -318,6 +370,16 @@ static void nv_mem_put_pages(struct sg_table *sg_head, void *context)
    return;
 }

+static void nv_mem_put_pages(struct sg_table *sg_head, void *context)
+{
+    nv_mem_put_pages_common(0, sg_head, context);
+}
+
+static void nv_mem_put_pages_nc(struct sg_table *sg_head, void *context)
+{
+    nv_mem_put_pages_common(1, sg_head, context);
+}
+
 static void nv_mem_release(void *context)
 {
    struct nv_mem_context *nv_mem_context =
@@ -326,6 +388,7 @@ static void nv_mem_release(void *context)
        sg_free_table(&nv_mem_context->sg_head);
        nv_mem_context->sg_allocated = 0;
    }
+    memset(nv_mem_context, 0, sizeof(*nv_mem_context));
    kfree(nv_mem_context);
    module_put(THIS_MODULE);
    return;
@@ -396,8 +459,15 @@ static int nv_mem_get_pages_nc(unsigned long addr,
    nv_mem_context->core_context = core_context;
    nv_mem_context->page_size = GPU_PAGE_SIZE;

+#ifdef NVIDIA_P2P_CAP_GET_PAGES_PERSISTENT_API
+    ret = nvidia_p2p_get_pages_persistent(nv_mem_context->page_virt_start,
+                                          nv_mem_context->mapped_size,
+                                          &nv_mem_context->page_table, 0);
+#else
    ret = nvidia_p2p_get_pages(0, 0, nv_mem_context->page_virt_start, nv_mem_context->mapped_size,
                               &nv_mem_context->page_table, NULL, NULL);
+#endif
+
    if (ret < 0) {
        peer_err("error %d while calling nvidia_p2p_get_pages() with NULL callback\n", ret);
        return ret;
@@ -407,43 +477,17 @@ static int nv_mem_get_pages_nc(unsigned long addr,
 }

 static struct peer_memory_client nv_mem_client_nc = {
-	.acquire        = nv_mem_acquire,
-	.get_pages      = nv_mem_get_pages_nc,
-	.dma_map        = nv_dma_map,
-	.dma_unmap      = nv_dma_unmap,
-	.put_pages      = nv_mem_put_pages,
-	.get_page_size  = nv_mem_get_page_size,
-	.release        = nv_mem_release,
+    .acquire        = nv_mem_acquire,
+    .get_pages      = nv_mem_get_pages_nc,
+    .dma_map        = nv_dma_map,
+    .dma_unmap      = nv_dma_unmap,
+    .put_pages      = nv_mem_put_pages_nc,
+    .get_page_size  = nv_mem_get_page_size,
+    .release        = nv_mem_release,
 };

-#endif /* NV_MLNX_IB_PEER_MEM_SYMBOLS_PRESENT */
-
-static int nv_mem_param_conf_check(void)
+static int nv_mem_legacy_client_init(void)
 {
-    int rc = 0;
-    switch (peerdirect_support) {
-    case NV_MEM_PEERDIRECT_SUPPORT_DEFAULT:
-    case NV_MEM_PEERDIRECT_SUPPORT_LEGACY:
-        break;
-    default:
-        peer_err("invalid peerdirect_support param value %d\n", peerdirect_support);
-        rc = -EINVAL;
-        break;
-    }
-    return rc;
-}
-
-static int __init nv_mem_client_init(void)
-{
-    int rc;
-    rc = nv_mem_param_conf_check();
-    if (rc) {
-        return rc;
-    }
-
-#if defined (NV_MLNX_IB_PEER_MEM_SYMBOLS_PRESENT)
-    int status = 0;
-
    // off by one, to leave space for the trailing '1' which is flagging
    // the new client type
    BUG_ON(strlen(DRV_NAME) > IB_PEER_MEMORY_NAME_MAX-1);
@@ -472,25 +516,99 @@ static int __init nv_mem_client_init(void)
                         &mem_invalidate_callback);
    if (!reg_handle) {
        peer_err("nv_mem_client_init -- error while registering traditional client\n");
-        status = -EINVAL;
-        goto out;
+        return -EINVAL;
+    }
+    return 0;
+}
+
+static int nv_mem_nc_client_init(void)
+{
+    // The nc client enables support for persistent pages.
+    if (persistent_api_support == NV_MEM_PERSISTENT_API_SUPPORT_LEGACY)
+    {
+        //
+        // If legacy behavior is forced via module param,
+        // both legacy and persistent clients are registered and are named
+        // "nv_mem"(legacy) and "nv_mem_nc"(persistent).
+        //
+        strcpy(nv_mem_client_nc.name, DRV_NAME "_nc");
+    }
+    else
+    {
+        //
+        // With default persistent behavior, the client name shall be "nv_mem"
+        // so that libraries can use the persistent client under the same name.
+        //
+        strcpy(nv_mem_client_nc.name, DRV_NAME);
    }

-    // The nc client enables support for persistent pages.
-    // Thanks to this check, nvidia-peermem requires the new symbol from nvidia.ko, which 
-    // prevents users to unintentionally load this module with unsupported nvidia.ko.
-    BUG_ON(!nvidia_p2p_cap_persistent_pages);
-    strcpy(nv_mem_client_nc.name, DRV_NAME "_nc");
    strcpy(nv_mem_client_nc.version, DRV_VERSION);
    reg_handle_nc = ib_register_peer_memory_client(&nv_mem_client_nc, NULL);
    if (!reg_handle_nc) {
        peer_err("nv_mem_client_init -- error while registering nc client\n");
-        status = -EINVAL;
-        goto out;
+        return -EINVAL;
+    }
+    return 0;
+}
+
+#endif /* NV_MLNX_IB_PEER_MEM_SYMBOLS_PRESENT */
+
+static int nv_mem_param_peerdirect_conf_check(void)
+{
+    int rc = 0;
+    switch (peerdirect_support) {
+    case NV_MEM_PEERDIRECT_SUPPORT_DEFAULT:
+    case NV_MEM_PEERDIRECT_SUPPORT_LEGACY:
+        break;
+    default:
+        peer_err("invalid peerdirect_support param value %d\n", peerdirect_support);
+        rc = -EINVAL;
+        break;
+    }
+    return rc;
+}
+
+static int nv_mem_param_persistent_api_conf_check(void)
+{
+    int rc = 0;
+    switch (persistent_api_support) {
+    case NV_MEM_PERSISTENT_API_SUPPORT_DEFAULT:
+    case NV_MEM_PERSISTENT_API_SUPPORT_LEGACY:
+        break;
+    default:
+        peer_err("invalid persistent_api_support param value %d\n", persistent_api_support);
+        rc = -EINVAL;
+        break;
+    }
+    return rc;
+}
+
+static int __init nv_mem_client_init(void)
+{
+#if defined (NV_MLNX_IB_PEER_MEM_SYMBOLS_PRESENT)
+    int rc;
+    rc = nv_mem_param_peerdirect_conf_check();
+    if (rc) {
+        return rc;
    }

+    rc = nv_mem_param_persistent_api_conf_check();
+    if (rc) {
+        return rc;
+    }
+
+    if (persistent_api_support == NV_MEM_PERSISTENT_API_SUPPORT_LEGACY) {
+        rc = nv_mem_legacy_client_init();
+        if (rc)
+            goto out;
+    }
+
+    rc = nv_mem_nc_client_init();
+    if (rc)
+        goto out;
+
 out:
-    if (status) {
+    if (rc) {
        if (reg_handle) {
            ib_unregister_peer_memory_client(reg_handle);
            reg_handle = NULL;
@@ -502,7 +620,7 @@ out:
        }
    }

-    return status;
+    return rc;
 #else
    return -EINVAL;
 #endif
--- a/kernel-open/nvidia-uvm/cla06fsubch.h
+++ b/kernel-open/nvidia-uvm/cla06fsubch.h
@@ -1,29 +1,33 @@
-/*******************************************************************************
-    Copyright (c) 2013 NVIDIA Corporation
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2003-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */

-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-    DEALINGS IN THE SOFTWARE.
-
-*******************************************************************************/
-
-#ifndef __cla06fsubch_h__
-#define __cla06fsubch_h__
+#ifndef _cla06fsubch_h_
+#define _cla06fsubch_h_

+#define NVA06F_SUBCHANNEL_2D                            3
+#define NVA06F_SUBCHANNEL_3D                            0
+#define NVA06F_SUBCHANNEL_COMPUTE                       1
 #define NVA06F_SUBCHANNEL_COPY_ENGINE                   4
+#define NVA06F_SUBCHANNEL_I2M                           2

-#endif // {__cla06fsubch_h__}
+#endif // _cla06fsubch_h_
--- a/kernel-open/nvidia-uvm/cla16f.h
+++ b/kernel-open/nvidia-uvm/cla16f.h
@@ -1,25 +1,25 @@
-/*******************************************************************************
-    Copyright (c) 2021 NVIDIA Corporation
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-    DEALINGS IN THE SOFTWARE.
-
-*******************************************************************************/
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */

 #ifndef _cla16f_h_
 #define _cla16f_h_
@@ -30,9 +30,48 @@ extern "C" {

 #include "nvtypes.h"

-#define KEPLER_CHANNEL_GPFIFO_B                                    (0x0000A16F)
+/* class KEPLER_CHANNEL_GPFIFO  */
+/*
+ * Documentation for KEPLER_CHANNEL_GPFIFO can be found in dev_pbdma.ref,
+ * chapter "User Control Registers". It is documented as device NV_UDMA.
+ * The GPFIFO format itself is also documented in dev_pbdma.ref,
+ * NV_PPBDMA_GP_ENTRY_*. The pushbuffer format is documented in dev_ram.ref,
+ * chapter "FIFO DMA RAM", NV_FIFO_DMA_*.
+ *
+ */
+#define  KEPLER_CHANNEL_GPFIFO_B                           (0x0000A16F)

+/* pio method data structure */
+typedef volatile struct _cla16f_tag0 {
+ NvV32 Reserved00[0x7c0];
+} NvA16FTypedef, KEPLER_ChannelGPFifoB;
+#define NVA16F_TYPEDEF                               KEPLER_CHANNELChannelGPFifo
+/* dma flow control data structure */
+typedef volatile struct _cla16f_tag1 {
+ NvU32 Ignored00[0x010];        /*                                  0000-003f*/
+ NvU32 Put;                     /* put offset, read/write           0040-0043*/
+ NvU32 Get;                     /* get offset, read only            0044-0047*/
+ NvU32 Reference;               /* reference value, read only       0048-004b*/
+ NvU32 PutHi;                   /* high order put offset bits       004c-004f*/
+ NvU32 Ignored01[0x002];        /*                                  0050-0057*/
+ NvU32 TopLevelGet;             /* top level get offset, read only  0058-005b*/
+ NvU32 TopLevelGetHi;           /* high order top level get bits    005c-005f*/
+ NvU32 GetHi;                   /* high order get offset bits       0060-0063*/
+ NvU32 Ignored02[0x007];        /*                                  0064-007f*/
+ NvU32 Ignored03;               /* used to be engine yield          0080-0083*/
+ NvU32 Ignored04[0x001];        /*                                  0084-0087*/
+ NvU32 GPGet;                   /* GP FIFO get offset, read only    0088-008b*/
+ NvU32 GPPut;                   /* GP FIFO put offset               008c-008f*/
+ NvU32 Ignored05[0x5c];
+} NvA16FControl, KeplerBControlGPFifo;
+/* fields and values */
+#define NVA16F_NUMBER_OF_SUBCHANNELS                               (8)
 #define NVA16F_SET_OBJECT                                          (0x00000000)
+#define NVA16F_SET_OBJECT_NVCLASS                                         15:0
+#define NVA16F_SET_OBJECT_ENGINE                                         20:16
+#define NVA16F_SET_OBJECT_ENGINE_SW                                 0x0000001f
+#define NVA16F_ILLEGAL                                             (0x00000004)
+#define NVA16F_ILLEGAL_HANDLE                                             31:0
 #define NVA16F_NOP                                                 (0x00000008)
 #define NVA16F_NOP_HANDLE                                                 31:0
 #define NVA16F_SEMAPHOREA                                          (0x00000010)
@@ -100,6 +139,12 @@ extern "C" {
 #define NVA16F_SET_REFERENCE_COUNT                                        31:0
 #define NVA16F_WFI                                                 (0x00000078)
 #define NVA16F_WFI_HANDLE                                                 31:0
+#define NVA16F_CRC_CHECK                                           (0x0000007c)
+#define NVA16F_CRC_CHECK_VALUE                                            31:0
+#define NVA16F_YIELD                                               (0x00000080)
+#define NVA16F_YIELD_OP                                                    1:0
+#define NVA16F_YIELD_OP_NOP                                         0x00000000
+

 /* GPFIFO entry format */
 #define NVA16F_GP_ENTRY__SIZE                                   8
@@ -116,15 +161,38 @@ extern "C" {
 #define NVA16F_GP_ENTRY1_LEVEL_MAIN                    0x00000000
 #define NVA16F_GP_ENTRY1_LEVEL_SUBROUTINE              0x00000001
 #define NVA16F_GP_ENTRY1_LENGTH                             30:10
+#define NVA16F_GP_ENTRY1_SYNC                               31:31
+#define NVA16F_GP_ENTRY1_SYNC_PROCEED                  0x00000000
+#define NVA16F_GP_ENTRY1_SYNC_WAIT                     0x00000001
+#define NVA16F_GP_ENTRY1_OPCODE                               7:0
+#define NVA16F_GP_ENTRY1_OPCODE_NOP                    0x00000000
+#define NVA16F_GP_ENTRY1_OPCODE_ILLEGAL                0x00000001
+#define NVA16F_GP_ENTRY1_OPCODE_GP_CRC                 0x00000002
+#define NVA16F_GP_ENTRY1_OPCODE_PB_CRC                 0x00000003

 /* dma method formats */
+#define NVA16F_DMA_METHOD_ADDRESS_OLD                              12:2
 #define NVA16F_DMA_METHOD_ADDRESS                                  11:0
+#define NVA16F_DMA_SUBDEVICE_MASK                                  15:4
 #define NVA16F_DMA_METHOD_SUBCHANNEL                               15:13
+#define NVA16F_DMA_TERT_OP                                         17:16
+#define NVA16F_DMA_TERT_OP_GRP0_INC_METHOD                         (0x00000000)
+#define NVA16F_DMA_TERT_OP_GRP0_SET_SUB_DEV_MASK                   (0x00000001)
+#define NVA16F_DMA_TERT_OP_GRP0_STORE_SUB_DEV_MASK                 (0x00000002)
+#define NVA16F_DMA_TERT_OP_GRP0_USE_SUB_DEV_MASK                   (0x00000003)
+#define NVA16F_DMA_TERT_OP_GRP2_NON_INC_METHOD                     (0x00000000)
+#define NVA16F_DMA_METHOD_COUNT_OLD                                28:18
 #define NVA16F_DMA_METHOD_COUNT                                    28:16
+#define NVA16F_DMA_IMMD_DATA                                       28:16
 #define NVA16F_DMA_SEC_OP                                          31:29
+#define NVA16F_DMA_SEC_OP_GRP0_USE_TERT                            (0x00000000)
 #define NVA16F_DMA_SEC_OP_INC_METHOD                               (0x00000001)
+#define NVA16F_DMA_SEC_OP_GRP2_USE_TERT                            (0x00000002)
 #define NVA16F_DMA_SEC_OP_NON_INC_METHOD                           (0x00000003)
-
+#define NVA16F_DMA_SEC_OP_IMMD_DATA_METHOD                         (0x00000004)
+#define NVA16F_DMA_SEC_OP_ONE_INC                                  (0x00000005)
+#define NVA16F_DMA_SEC_OP_RESERVED6                                (0x00000006)
+#define NVA16F_DMA_SEC_OP_END_PB_SEGMENT                           (0x00000007)
 /* dma incrementing method format */
 #define NVA16F_DMA_INCR_ADDRESS                                    11:0
 #define NVA16F_DMA_INCR_SUBCHANNEL                                 15:13
@@ -132,7 +200,6 @@ extern "C" {
 #define NVA16F_DMA_INCR_OPCODE                                     31:29
 #define NVA16F_DMA_INCR_OPCODE_VALUE                               (0x00000001)
 #define NVA16F_DMA_INCR_DATA                                       31:0
-
 /* dma non-incrementing method format */
 #define NVA16F_DMA_NONINCR_ADDRESS                                 11:0
 #define NVA16F_DMA_NONINCR_SUBCHANNEL                              15:13
@@ -140,13 +207,45 @@ extern "C" {
 #define NVA16F_DMA_NONINCR_OPCODE                                  31:29
 #define NVA16F_DMA_NONINCR_OPCODE_VALUE                            (0x00000003)
 #define NVA16F_DMA_NONINCR_DATA                                    31:0
-
+/* dma increment-once method format */
+#define NVA16F_DMA_ONEINCR_ADDRESS                                 11:0
+#define NVA16F_DMA_ONEINCR_SUBCHANNEL                              15:13
+#define NVA16F_DMA_ONEINCR_COUNT                                   28:16
+#define NVA16F_DMA_ONEINCR_OPCODE                                  31:29
+#define NVA16F_DMA_ONEINCR_OPCODE_VALUE                            (0x00000005)
+#define NVA16F_DMA_ONEINCR_DATA                                    31:0
+/* dma no-operation format */
+#define NVA16F_DMA_NOP                                             (0x00000000)
 /* dma immediate-data format */
 #define NVA16F_DMA_IMMD_ADDRESS                                    11:0
 #define NVA16F_DMA_IMMD_SUBCHANNEL                                 15:13
 #define NVA16F_DMA_IMMD_DATA                                       28:16
 #define NVA16F_DMA_IMMD_OPCODE                                     31:29
 #define NVA16F_DMA_IMMD_OPCODE_VALUE                               (0x00000004)
+/* dma set sub-device mask format */
+#define NVA16F_DMA_SET_SUBDEVICE_MASK_VALUE                        15:4
+#define NVA16F_DMA_SET_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVA16F_DMA_SET_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000001)
+/* dma store sub-device mask format */
+#define NVA16F_DMA_STORE_SUBDEVICE_MASK_VALUE                      15:4
+#define NVA16F_DMA_STORE_SUBDEVICE_MASK_OPCODE                     31:16
+#define NVA16F_DMA_STORE_SUBDEVICE_MASK_OPCODE_VALUE               (0x00000002)
+/* dma use sub-device mask format */
+#define NVA16F_DMA_USE_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVA16F_DMA_USE_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000003)
+/* dma end-segment format */
+#define NVA16F_DMA_ENDSEG_OPCODE                                   31:29
+#define NVA16F_DMA_ENDSEG_OPCODE_VALUE                             (0x00000007)
+/* dma legacy incrementing/non-incrementing formats */
+#define NVA16F_DMA_ADDRESS                                         12:2
+#define NVA16F_DMA_SUBCH                                           15:13
+#define NVA16F_DMA_OPCODE3                                         17:16
+#define NVA16F_DMA_OPCODE3_NONE                                    (0x00000000)
+#define NVA16F_DMA_COUNT                                           28:18
+#define NVA16F_DMA_OPCODE                                          31:29
+#define NVA16F_DMA_OPCODE_METHOD                                   (0x00000000)
+#define NVA16F_DMA_OPCODE_NONINC_METHOD                            (0x00000002)
+#define NVA16F_DMA_DATA                                            31:0

 #ifdef __cplusplus
 };     /* extern "C" */
--- a/kernel-open/nvidia-uvm/clb069.h
+++ b/kernel-open/nvidia-uvm/clb069.h
@@ -1,24 +1,26 @@
-/*******************************************************************************
-    Copyright (c) 2014 NVidia Corporation
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */

-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-    DEALINGS IN THE SOFTWARE.
-*******************************************************************************/
 #ifndef _clb069_h_
 #define _clb069_h_

--- a/kernel-open/nvidia-uvm/clb06f.h
+++ b/kernel-open/nvidia-uvm/clb06f.h
@@ -1,28 +1,28 @@
-/*******************************************************************************
-    Copyright (c) 2014 NVIDIA Corporation
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */

-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-    DEALINGS IN THE SOFTWARE.
-
-*******************************************************************************/
-
-#ifndef _clB06f_h_
-#define _clB06f_h_
+#ifndef _clb06f_h_
+#define _clb06f_h_

 #ifdef __cplusplus
 extern "C" {
@@ -30,10 +30,46 @@ extern "C" {

 #include "nvtypes.h"

+/* class MAXWELL_CHANNEL_GPFIFO  */
+/*
+ * Documentation for MAXWELL_CHANNEL_GPFIFO can be found in dev_pbdma.ref,
+ * chapter "User Control Registers". It is documented as device NV_UDMA.
+ * The GPFIFO format itself is also documented in dev_pbdma.ref,
+ * NV_PPBDMA_GP_ENTRY_*. The pushbuffer format is documented in dev_ram.ref,
+ * chapter "FIFO DMA RAM", NV_FIFO_DMA_*.
+ *
+ */
 #define  MAXWELL_CHANNEL_GPFIFO_A                           (0x0000B06F)

-/* class MAXWELL_CHANNEL_GPFIFO  */
+#define NVB06F_TYPEDEF                             MAXWELL_CHANNELChannelGPFifoA
+
+/* dma flow control data structure */
+typedef volatile struct _clb06f_tag0 {
+ NvU32 Ignored00[0x010];        /*                                  0000-003f*/
+ NvU32 Put;                     /* put offset, read/write           0040-0043*/
+ NvU32 Get;                     /* get offset, read only            0044-0047*/
+ NvU32 Reference;               /* reference value, read only       0048-004b*/
+ NvU32 PutHi;                   /* high order put offset bits       004c-004f*/
+ NvU32 Ignored01[0x002];        /*                                  0050-0057*/
+ NvU32 TopLevelGet;             /* top level get offset, read only  0058-005b*/
+ NvU32 TopLevelGetHi;           /* high order top level get bits    005c-005f*/
+ NvU32 GetHi;                   /* high order get offset bits       0060-0063*/
+ NvU32 Ignored02[0x007];        /*                                  0064-007f*/
+ NvU32 Ignored03;               /* used to be engine yield          0080-0083*/
+ NvU32 Ignored04[0x001];        /*                                  0084-0087*/
+ NvU32 GPGet;                   /* GP FIFO get offset, read only    0088-008b*/
+ NvU32 GPPut;                   /* GP FIFO put offset               008c-008f*/
+ NvU32 Ignored05[0x5c];
+} Nvb06FControl, MaxwellAControlGPFifo;
+
+/* fields and values */
+#define NVB06F_NUMBER_OF_SUBCHANNELS                               (8)
 #define NVB06F_SET_OBJECT                                          (0x00000000)
+#define NVB06F_SET_OBJECT_NVCLASS                                         15:0
+#define NVB06F_SET_OBJECT_ENGINE                                         20:16
+#define NVB06F_SET_OBJECT_ENGINE_SW                                 0x0000001f
+#define NVB06F_ILLEGAL                                             (0x00000004)
+#define NVB06F_ILLEGAL_HANDLE                                             31:0
 #define NVB06F_NOP                                                 (0x00000008)
 #define NVB06F_NOP_HANDLE                                                 31:0
 #define NVB06F_SEMAPHOREA                                          (0x00000010)
@@ -47,6 +83,8 @@ extern "C" {
 #define NVB06F_SEMAPHORED_OPERATION_ACQUIRE                         0x00000001
 #define NVB06F_SEMAPHORED_OPERATION_RELEASE                         0x00000002
 #define NVB06F_SEMAPHORED_OPERATION_ACQ_GEQ                         0x00000004
+#define NVB06F_SEMAPHORED_OPERATION_ACQ_AND                         0x00000008
+#define NVB06F_SEMAPHORED_OPERATION_REDUCTION                       0x00000010
 #define NVB06F_SEMAPHORED_ACQUIRE_SWITCH                                 12:12
 #define NVB06F_SEMAPHORED_ACQUIRE_SWITCH_DISABLED                   0x00000000
 #define NVB06F_SEMAPHORED_ACQUIRE_SWITCH_ENABLED                    0x00000001
@@ -56,8 +94,22 @@ extern "C" {
 #define NVB06F_SEMAPHORED_RELEASE_SIZE                                   24:24
 #define NVB06F_SEMAPHORED_RELEASE_SIZE_16BYTE                       0x00000000
 #define NVB06F_SEMAPHORED_RELEASE_SIZE_4BYTE                        0x00000001
-
+#define NVB06F_SEMAPHORED_REDUCTION                                      30:27
+#define NVB06F_SEMAPHORED_REDUCTION_MIN                             0x00000000
+#define NVB06F_SEMAPHORED_REDUCTION_MAX                             0x00000001
+#define NVB06F_SEMAPHORED_REDUCTION_XOR                             0x00000002
+#define NVB06F_SEMAPHORED_REDUCTION_AND                             0x00000003
+#define NVB06F_SEMAPHORED_REDUCTION_OR                              0x00000004
+#define NVB06F_SEMAPHORED_REDUCTION_ADD                             0x00000005
+#define NVB06F_SEMAPHORED_REDUCTION_INC                             0x00000006
+#define NVB06F_SEMAPHORED_REDUCTION_DEC                             0x00000007
+#define NVB06F_SEMAPHORED_FORMAT                                         31:31
+#define NVB06F_SEMAPHORED_FORMAT_SIGNED                             0x00000000
+#define NVB06F_SEMAPHORED_FORMAT_UNSIGNED                           0x00000001
 #define NVB06F_NON_STALL_INTERRUPT                                 (0x00000020)
+#define NVB06F_NON_STALL_INTERRUPT_HANDLE                                 31:0
+#define NVB06F_FB_FLUSH                                            (0x00000024)
+#define NVB06F_FB_FLUSH_HANDLE                                            31:0
 // NOTE - MEM_OP_A and MEM_OP_B have been removed for gm20x to make room for
 // possible future MEM_OP features.  MEM_OP_C/D have identical functionality
 // to the previous MEM_OP_A/B methods.
@@ -84,10 +136,27 @@ extern "C" {
 #define NVB06F_MEM_OP_D_OPERATION_L2_CLEAN_COMPTAGS                 0x0000000f
 #define NVB06F_MEM_OP_D_OPERATION_L2_FLUSH_DIRTY                    0x00000010
 #define NVB06F_MEM_OP_D_TLB_INVALIDATE_ADDR_HI                             7:0
+#define NVB06F_SET_REFERENCE                                       (0x00000050)
+#define NVB06F_SET_REFERENCE_COUNT                                        31:0
 #define NVB06F_WFI                                                 (0x00000078)
+#define NVB06F_WFI_SCOPE                                                   0:0
+#define NVB06F_WFI_SCOPE_CURRENT_SCG_TYPE                           0x00000000
+#define NVB06F_WFI_SCOPE_ALL                                        0x00000001
+#define NVB06F_CRC_CHECK                                           (0x0000007c)
+#define NVB06F_CRC_CHECK_VALUE                                            31:0
+#define NVB06F_YIELD                                               (0x00000080)
+#define NVB06F_YIELD_OP                                                    1:0
+#define NVB06F_YIELD_OP_NOP                                         0x00000000
+#define NVB06F_YIELD_OP_PBDMA_TIMESLICE                             0x00000001
+#define NVB06F_YIELD_OP_RUNLIST_TIMESLICE                           0x00000002
+#define NVB06F_YIELD_OP_TSG                                         0x00000003
+

 /* GPFIFO entry format */
 #define NVB06F_GP_ENTRY__SIZE                                   8
+#define NVB06F_GP_ENTRY0_FETCH                                0:0
+#define NVB06F_GP_ENTRY0_FETCH_UNCONDITIONAL           0x00000000
+#define NVB06F_GP_ENTRY0_FETCH_CONDITIONAL             0x00000001
 #define NVB06F_GP_ENTRY0_GET                                 31:2
 #define NVB06F_GP_ENTRY0_OPERAND                             31:0
 #define NVB06F_GP_ENTRY1_GET_HI                               7:0
@@ -98,11 +167,38 @@ extern "C" {
 #define NVB06F_GP_ENTRY1_LEVEL_MAIN                    0x00000000
 #define NVB06F_GP_ENTRY1_LEVEL_SUBROUTINE              0x00000001
 #define NVB06F_GP_ENTRY1_LENGTH                             30:10
+#define NVB06F_GP_ENTRY1_SYNC                               31:31
+#define NVB06F_GP_ENTRY1_SYNC_PROCEED                  0x00000000
+#define NVB06F_GP_ENTRY1_SYNC_WAIT                     0x00000001
+#define NVB06F_GP_ENTRY1_OPCODE                               7:0
+#define NVB06F_GP_ENTRY1_OPCODE_NOP                    0x00000000
+#define NVB06F_GP_ENTRY1_OPCODE_ILLEGAL                0x00000001
+#define NVB06F_GP_ENTRY1_OPCODE_GP_CRC                 0x00000002
+#define NVB06F_GP_ENTRY1_OPCODE_PB_CRC                 0x00000003

 /* dma method formats */
+#define NVB06F_DMA_METHOD_ADDRESS_OLD                              12:2
+#define NVB06F_DMA_METHOD_ADDRESS                                  11:0
+#define NVB06F_DMA_SUBDEVICE_MASK                                  15:4
+#define NVB06F_DMA_METHOD_SUBCHANNEL                               15:13
+#define NVB06F_DMA_TERT_OP                                         17:16
+#define NVB06F_DMA_TERT_OP_GRP0_INC_METHOD                         (0x00000000)
+#define NVB06F_DMA_TERT_OP_GRP0_SET_SUB_DEV_MASK                   (0x00000001)
+#define NVB06F_DMA_TERT_OP_GRP0_STORE_SUB_DEV_MASK                 (0x00000002)
+#define NVB06F_DMA_TERT_OP_GRP0_USE_SUB_DEV_MASK                   (0x00000003)
+#define NVB06F_DMA_TERT_OP_GRP2_NON_INC_METHOD                     (0x00000000)
+#define NVB06F_DMA_METHOD_COUNT_OLD                                28:18
+#define NVB06F_DMA_METHOD_COUNT                                    28:16
+#define NVB06F_DMA_IMMD_DATA                                       28:16
 #define NVB06F_DMA_SEC_OP                                          31:29
+#define NVB06F_DMA_SEC_OP_GRP0_USE_TERT                            (0x00000000)
 #define NVB06F_DMA_SEC_OP_INC_METHOD                               (0x00000001)
+#define NVB06F_DMA_SEC_OP_GRP2_USE_TERT                            (0x00000002)
 #define NVB06F_DMA_SEC_OP_NON_INC_METHOD                           (0x00000003)
+#define NVB06F_DMA_SEC_OP_IMMD_DATA_METHOD                         (0x00000004)
+#define NVB06F_DMA_SEC_OP_ONE_INC                                  (0x00000005)
+#define NVB06F_DMA_SEC_OP_RESERVED6                                (0x00000006)
+#define NVB06F_DMA_SEC_OP_END_PB_SEGMENT                           (0x00000007)
 /* dma incrementing method format */
 #define NVB06F_DMA_INCR_ADDRESS                                    11:0
 #define NVB06F_DMA_INCR_SUBCHANNEL                                 15:13
@@ -132,9 +228,33 @@ extern "C" {
 #define NVB06F_DMA_IMMD_DATA                                       28:16
 #define NVB06F_DMA_IMMD_OPCODE                                     31:29
 #define NVB06F_DMA_IMMD_OPCODE_VALUE                               (0x00000004)
+/* dma set sub-device mask format */
+#define NVB06F_DMA_SET_SUBDEVICE_MASK_VALUE                        15:4
+#define NVB06F_DMA_SET_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVB06F_DMA_SET_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000001)
+/* dma store sub-device mask format */
+#define NVB06F_DMA_STORE_SUBDEVICE_MASK_VALUE                      15:4
+#define NVB06F_DMA_STORE_SUBDEVICE_MASK_OPCODE                     31:16
+#define NVB06F_DMA_STORE_SUBDEVICE_MASK_OPCODE_VALUE               (0x00000002)
+/* dma use sub-device mask format */
+#define NVB06F_DMA_USE_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVB06F_DMA_USE_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000003)
+/* dma end-segment format */
+#define NVB06F_DMA_ENDSEG_OPCODE                                   31:29
+#define NVB06F_DMA_ENDSEG_OPCODE_VALUE                             (0x00000007)
+/* dma legacy incrementing/non-incrementing formats */
+#define NVB06F_DMA_ADDRESS                                         12:2
+#define NVB06F_DMA_SUBCH                                           15:13
+#define NVB06F_DMA_OPCODE3                                         17:16
+#define NVB06F_DMA_OPCODE3_NONE                                    (0x00000000)
+#define NVB06F_DMA_COUNT                                           28:18
+#define NVB06F_DMA_OPCODE                                          31:29
+#define NVB06F_DMA_OPCODE_METHOD                                   (0x00000000)
+#define NVB06F_DMA_OPCODE_NONINC_METHOD                            (0x00000002)
+#define NVB06F_DMA_DATA                                            31:0

 #ifdef __cplusplus
 };     /* extern "C" */
 #endif

-#endif /* _clB06F_h_ */
+#endif /* _clb06f_h_ */
--- a/kernel-open/nvidia-uvm/clb0b5.h
+++ b/kernel-open/nvidia-uvm/clb0b5.h
@@ -1,19 +1,19 @@
 /*******************************************************************************
-    Copyright (c) 2014 NVIDIA Corporation
+    Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.

-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
+    Permission is hereby granted, free of charge, to any person obtaining a
+    copy of this software and associated documentation files (the "Software"),
+    to deal in the Software without restriction, including without limitation
+    the rights to use, copy, modify, merge, publish, distribute, sublicense,
+    and/or sell copies of the Software, and to permit persons to whom the
+    Software is furnished to do so, subject to the following conditions:

-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
+    The above copyright notice and this permission notice shall be included in
+    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
@@ -32,6 +32,10 @@ extern "C" {

 #define MAXWELL_DMA_COPY_A                                                            (0x0000B0B5)

+#define NVB0B5_NOP                                                              (0x00000100)
+#define NVB0B5_NOP_PARAMETER                                                    31:0
+#define NVB0B5_PM_TRIGGER                                                       (0x00000140)
+#define NVB0B5_PM_TRIGGER_V                                                     31:0
 #define NVB0B5_SET_SEMAPHORE_A                                                  (0x00000240)
 #define NVB0B5_SET_SEMAPHORE_A_UPPER                                            7:0
 #define NVB0B5_SET_SEMAPHORE_B                                                  (0x00000244)
@@ -183,9 +187,75 @@ extern "C" {
 #define NVB0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_TWO                      (0x00000001)
 #define NVB0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_THREE                    (0x00000002)
 #define NVB0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_FOUR                     (0x00000003)
-
+#define NVB0B5_SET_DST_BLOCK_SIZE                                               (0x0000070C)
+#define NVB0B5_SET_DST_BLOCK_SIZE_WIDTH                                         3:0
+#define NVB0B5_SET_DST_BLOCK_SIZE_WIDTH_QUARTER_GOB                             (0x0000000E)
+#define NVB0B5_SET_DST_BLOCK_SIZE_WIDTH_ONE_GOB                                 (0x00000000)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT                                        7:4
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_ONE_GOB                                (0x00000000)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_TWO_GOBS                               (0x00000001)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_FOUR_GOBS                              (0x00000002)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_EIGHT_GOBS                             (0x00000003)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_SIXTEEN_GOBS                           (0x00000004)
+#define NVB0B5_SET_DST_BLOCK_SIZE_HEIGHT_THIRTYTWO_GOBS                         (0x00000005)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH                                         11:8
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_ONE_GOB                                 (0x00000000)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_TWO_GOBS                                (0x00000001)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_FOUR_GOBS                               (0x00000002)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_EIGHT_GOBS                              (0x00000003)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_SIXTEEN_GOBS                            (0x00000004)
+#define NVB0B5_SET_DST_BLOCK_SIZE_DEPTH_THIRTYTWO_GOBS                          (0x00000005)
+#define NVB0B5_SET_DST_BLOCK_SIZE_GOB_HEIGHT                                    15:12
+#define NVB0B5_SET_DST_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_TESLA_4                 (0x00000000)
+#define NVB0B5_SET_DST_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_FERMI_8                 (0x00000001)
+#define NVB0B5_SET_DST_WIDTH                                                    (0x00000710)
+#define NVB0B5_SET_DST_WIDTH_V                                                  31:0
+#define NVB0B5_SET_DST_HEIGHT                                                   (0x00000714)
+#define NVB0B5_SET_DST_HEIGHT_V                                                 31:0
+#define NVB0B5_SET_DST_DEPTH                                                    (0x00000718)
+#define NVB0B5_SET_DST_DEPTH_V                                                  31:0
+#define NVB0B5_SET_DST_LAYER                                                    (0x0000071C)
+#define NVB0B5_SET_DST_LAYER_V                                                  31:0
+#define NVB0B5_SET_DST_ORIGIN                                                   (0x00000720)
+#define NVB0B5_SET_DST_ORIGIN_X                                                 15:0
+#define NVB0B5_SET_DST_ORIGIN_Y                                                 31:16
+#define NVB0B5_SET_SRC_BLOCK_SIZE                                               (0x00000728)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_WIDTH                                         3:0
+#define NVB0B5_SET_SRC_BLOCK_SIZE_WIDTH_QUARTER_GOB                             (0x0000000E)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_WIDTH_ONE_GOB                                 (0x00000000)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT                                        7:4
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_ONE_GOB                                (0x00000000)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_TWO_GOBS                               (0x00000001)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_FOUR_GOBS                              (0x00000002)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_EIGHT_GOBS                             (0x00000003)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_SIXTEEN_GOBS                           (0x00000004)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_HEIGHT_THIRTYTWO_GOBS                         (0x00000005)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH                                         11:8
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_ONE_GOB                                 (0x00000000)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_TWO_GOBS                                (0x00000001)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_FOUR_GOBS                               (0x00000002)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_EIGHT_GOBS                              (0x00000003)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_SIXTEEN_GOBS                            (0x00000004)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_DEPTH_THIRTYTWO_GOBS                          (0x00000005)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_GOB_HEIGHT                                    15:12
+#define NVB0B5_SET_SRC_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_TESLA_4                 (0x00000000)
+#define NVB0B5_SET_SRC_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_FERMI_8                 (0x00000001)
+#define NVB0B5_SET_SRC_WIDTH                                                    (0x0000072C)
+#define NVB0B5_SET_SRC_WIDTH_V                                                  31:0
+#define NVB0B5_SET_SRC_HEIGHT                                                   (0x00000730)
+#define NVB0B5_SET_SRC_HEIGHT_V                                                 31:0
+#define NVB0B5_SET_SRC_DEPTH                                                    (0x00000734)
+#define NVB0B5_SET_SRC_DEPTH_V                                                  31:0
+#define NVB0B5_SET_SRC_LAYER                                                    (0x00000738)
+#define NVB0B5_SET_SRC_LAYER_V                                                  31:0
+#define NVB0B5_SET_SRC_ORIGIN                                                   (0x0000073C)
+#define NVB0B5_SET_SRC_ORIGIN_X                                                 15:0
+#define NVB0B5_SET_SRC_ORIGIN_Y                                                 31:16
+#define NVB0B5_PM_TRIGGER_END                                                   (0x00001114)
+#define NVB0B5_PM_TRIGGER_END_V                                                 31:0

 #ifdef __cplusplus
 };     /* extern "C" */
 #endif
 #endif // _clb0b5_h
+
--- a/kernel-open/nvidia-uvm/clc06f.h
+++ b/kernel-open/nvidia-uvm/clc06f.h
@@ -1,25 +1,25 @@
-/*******************************************************************************
-    Copyright (c) 2014 NVIDIA Corporation
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-    DEALINGS IN THE SOFTWARE.
-
-*******************************************************************************/
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2020-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */

 #ifndef _clc06f_h_
 #define _clc06f_h_
@@ -30,10 +30,47 @@ extern "C" {

 #include "nvtypes.h"

+/* class PASCAL_CHANNEL_GPFIFO  */
+/*
+ * Documentation for PASCAL_CHANNEL_GPFIFO can be found in dev_pbdma.ref,
+ * chapter "User Control Registers". It is documented as device NV_UDMA.
+ * The GPFIFO format itself is also documented in dev_pbdma.ref,
+ * NV_PPBDMA_GP_ENTRY_*. The pushbuffer format is documented in dev_ram.ref,
+ * chapter "FIFO DMA RAM", NV_FIFO_DMA_*.
+ *
+ * Note there is no .mfs file for this class.
+ */
 #define  PASCAL_CHANNEL_GPFIFO_A                           (0x0000C06F)

-/* class PASCAL_CHANNEL_GPFIFO_A */
+#define NVC06F_TYPEDEF                             PASCAL_CHANNELChannelGPFifoA
+
+/* dma flow control data structure */
+typedef volatile struct Nvc06fControl_struct {
+ NvU32 Ignored00[0x010];        /*                                  0000-003f*/
+ NvU32 Put;                     /* put offset, read/write           0040-0043*/
+ NvU32 Get;                     /* get offset, read only            0044-0047*/
+ NvU32 Reference;               /* reference value, read only       0048-004b*/
+ NvU32 PutHi;                   /* high order put offset bits       004c-004f*/
+ NvU32 Ignored01[0x002];        /*                                  0050-0057*/
+ NvU32 TopLevelGet;             /* top level get offset, read only  0058-005b*/
+ NvU32 TopLevelGetHi;           /* high order top level get bits    005c-005f*/
+ NvU32 GetHi;                   /* high order get offset bits       0060-0063*/
+ NvU32 Ignored02[0x007];        /*                                  0064-007f*/
+ NvU32 Ignored03;               /* used to be engine yield          0080-0083*/
+ NvU32 Ignored04[0x001];        /*                                  0084-0087*/
+ NvU32 GPGet;                   /* GP FIFO get offset, read only    0088-008b*/
+ NvU32 GPPut;                   /* GP FIFO put offset               008c-008f*/
+ NvU32 Ignored05[0x5c];
+} Nvc06fControl, PascalAControlGPFifo;
+
+/* fields and values */
+#define NVC06F_NUMBER_OF_SUBCHANNELS                               (8)
 #define NVC06F_SET_OBJECT                                          (0x00000000)
+#define NVC06F_SET_OBJECT_NVCLASS                                         15:0
+#define NVC06F_SET_OBJECT_ENGINE                                         20:16
+#define NVC06F_SET_OBJECT_ENGINE_SW                                 0x0000001f
+#define NVC06F_ILLEGAL                                             (0x00000004)
+#define NVC06F_ILLEGAL_HANDLE                                             31:0
 #define NVC06F_NOP                                                 (0x00000008)
 #define NVC06F_NOP_HANDLE                                                 31:0
 #define NVC06F_SEMAPHOREA                                          (0x00000010)
@@ -47,54 +84,33 @@ extern "C" {
 #define NVC06F_SEMAPHORED_OPERATION_ACQUIRE                         0x00000001
 #define NVC06F_SEMAPHORED_OPERATION_RELEASE                         0x00000002
 #define NVC06F_SEMAPHORED_OPERATION_ACQ_GEQ                         0x00000004
+#define NVC06F_SEMAPHORED_OPERATION_ACQ_AND                         0x00000008
+#define NVC06F_SEMAPHORED_OPERATION_REDUCTION                       0x00000010
 #define NVC06F_SEMAPHORED_ACQUIRE_SWITCH                                 12:12
 #define NVC06F_SEMAPHORED_ACQUIRE_SWITCH_DISABLED                   0x00000000
 #define NVC06F_SEMAPHORED_ACQUIRE_SWITCH_ENABLED                    0x00000001
-
-
-/* GPFIFO entry format */
-#define NVC06F_GP_ENTRY__SIZE                                   8
-#define NVC06F_GP_ENTRY0_GET                                 31:2
-#define NVC06F_GP_ENTRY0_OPERAND                             31:0
-#define NVC06F_GP_ENTRY1_GET_HI                               7:0
-#define NVC06F_GP_ENTRY1_PRIV                                 8:8
-#define NVC06F_GP_ENTRY1_PRIV_USER                     0x00000000
-#define NVC06F_GP_ENTRY1_PRIV_KERNEL                   0x00000001
-#define NVC06F_GP_ENTRY1_LEVEL                                9:9
-#define NVC06F_GP_ENTRY1_LEVEL_MAIN                    0x00000000
-#define NVC06F_GP_ENTRY1_LEVEL_SUBROUTINE              0x00000001
-#define NVC06F_GP_ENTRY1_LENGTH                             30:10
-
-/* dma incrementing method format */
-#define NVC06F_DMA_INCR_ADDRESS                                    11:0
-#define NVC06F_DMA_INCR_SUBCHANNEL                                 15:13
-#define NVC06F_DMA_INCR_COUNT                                      28:16
-#define NVC06F_DMA_INCR_OPCODE                                     31:29
-#define NVC06F_DMA_INCR_OPCODE_VALUE                               (0x00000001)
-#define NVC06F_DMA_INCR_DATA                                       31:0
-/* dma non-incrementing method format */
-#define NVC06F_DMA_NONINCR_ADDRESS                                 11:0
-#define NVC06F_DMA_NONINCR_SUBCHANNEL                              15:13
-#define NVC06F_DMA_NONINCR_COUNT                                   28:16
-#define NVC06F_DMA_NONINCR_OPCODE                                  31:29
-#define NVC06F_DMA_NONINCR_OPCODE_VALUE                            (0x00000003)
-#define NVC06F_DMA_NONINCR_DATA                                    31:0
-/* dma increment-once method format */
-#define NVC06F_DMA_ONEINCR_ADDRESS                                 11:0
-#define NVC06F_DMA_ONEINCR_SUBCHANNEL                              15:13
-#define NVC06F_DMA_ONEINCR_COUNT                                   28:16
-#define NVC06F_DMA_ONEINCR_OPCODE                                  31:29
-#define NVC06F_DMA_ONEINCR_OPCODE_VALUE                            (0x00000005)
-#define NVC06F_DMA_ONEINCR_DATA                                    31:0
-/* dma no-operation format */
-#define NVC06F_DMA_NOP                                             (0x00000000)
-/* dma immediate-data format */
-#define NVC06F_DMA_IMMD_ADDRESS                                    11:0
-#define NVC06F_DMA_IMMD_SUBCHANNEL                                 15:13
-#define NVC06F_DMA_IMMD_DATA                                       28:16
-#define NVC06F_DMA_IMMD_OPCODE                                     31:29
-#define NVC06F_DMA_IMMD_OPCODE_VALUE                               (0x00000004)
-
+#define NVC06F_SEMAPHORED_RELEASE_WFI                                    20:20
+#define NVC06F_SEMAPHORED_RELEASE_WFI_EN                            0x00000000
+#define NVC06F_SEMAPHORED_RELEASE_WFI_DIS                           0x00000001
+#define NVC06F_SEMAPHORED_RELEASE_SIZE                                   24:24
+#define NVC06F_SEMAPHORED_RELEASE_SIZE_16BYTE                       0x00000000
+#define NVC06F_SEMAPHORED_RELEASE_SIZE_4BYTE                        0x00000001
+#define NVC06F_SEMAPHORED_REDUCTION                                      30:27
+#define NVC06F_SEMAPHORED_REDUCTION_MIN                             0x00000000
+#define NVC06F_SEMAPHORED_REDUCTION_MAX                             0x00000001
+#define NVC06F_SEMAPHORED_REDUCTION_XOR                             0x00000002
+#define NVC06F_SEMAPHORED_REDUCTION_AND                             0x00000003
+#define NVC06F_SEMAPHORED_REDUCTION_OR                              0x00000004
+#define NVC06F_SEMAPHORED_REDUCTION_ADD                             0x00000005
+#define NVC06F_SEMAPHORED_REDUCTION_INC                             0x00000006
+#define NVC06F_SEMAPHORED_REDUCTION_DEC                             0x00000007
+#define NVC06F_SEMAPHORED_FORMAT                                         31:31
+#define NVC06F_SEMAPHORED_FORMAT_SIGNED                             0x00000000
+#define NVC06F_SEMAPHORED_FORMAT_UNSIGNED                           0x00000001
+#define NVC06F_NON_STALL_INTERRUPT                                 (0x00000020)
+#define NVC06F_NON_STALL_INTERRUPT_HANDLE                                 31:0
+#define NVC06F_FB_FLUSH                                            (0x00000024) // Deprecated - use MEMBAR TYPE SYS_MEMBAR
+#define NVC06F_FB_FLUSH_HANDLE                                            31:0
 // NOTE - MEM_OP_A and MEM_OP_B have been replaced in gp100 with methods for
 // specifying the page address for a targeted TLB invalidate and the uTLB for
 // a targeted REPLAY_CANCEL for UVM.
@@ -153,19 +169,142 @@ extern "C" {
 #define NVC06F_MEM_OP_D_OPERATION_L2_PEERMEM_INVALIDATE             0x0000000d
 #define NVC06F_MEM_OP_D_OPERATION_L2_SYSMEM_INVALIDATE              0x0000000e
 // CLEAN_LINES is an alias for Tegra/GPU IP usage
+#define NVC06F_MEM_OP_D_OPERATION_L2_INVALIDATE_CLEAN_LINES         0x0000000e
+// This B alias is confusing but it was missed as part of the update. Left here
+// for compatibility.
 #define NVC06F_MEM_OP_B_OPERATION_L2_INVALIDATE_CLEAN_LINES         0x0000000e
 #define NVC06F_MEM_OP_D_OPERATION_L2_CLEAN_COMPTAGS                 0x0000000f
 #define NVC06F_MEM_OP_D_OPERATION_L2_FLUSH_DIRTY                    0x00000010
 #define NVC06F_MEM_OP_D_OPERATION_L2_WAIT_FOR_SYS_PENDING_READS     0x00000015
 #define NVC06F_SET_REFERENCE                                       (0x00000050)
 #define NVC06F_SET_REFERENCE_COUNT                                        31:0
-
+// Syncpoint methods are only available on Tegra parts.  Attempting to use
+// them on discrete GPUs will result in Host raising NV_PPBDMA_INTR_0_METHOD.
+#define NVC06F_SYNCPOINTA                                          (0x00000070)
+#define NVC06F_SYNCPOINTA_PAYLOAD                                         31:0
+#define NVC06F_SYNCPOINTB                                          (0x00000074)
+#define NVC06F_SYNCPOINTB_OPERATION                                        0:0
+#define NVC06F_SYNCPOINTB_OPERATION_WAIT                            0x00000000
+#define NVC06F_SYNCPOINTB_OPERATION_INCR                            0x00000001
+#define NVC06F_SYNCPOINTB_WAIT_SWITCH                                      4:4
+#define NVC06F_SYNCPOINTB_WAIT_SWITCH_DIS                           0x00000000
+#define NVC06F_SYNCPOINTB_WAIT_SWITCH_EN                            0x00000001
+#define NVC06F_SYNCPOINTB_SYNCPT_INDEX                                    19:8
 #define NVC06F_WFI                                                 (0x00000078)
 #define NVC06F_WFI_SCOPE                                                   0:0
 #define NVC06F_WFI_SCOPE_CURRENT_SCG_TYPE                           0x00000000
 #define NVC06F_WFI_SCOPE_ALL                                        0x00000001
+#define NVC06F_CRC_CHECK                                           (0x0000007c)
+#define NVC06F_CRC_CHECK_VALUE                                            31:0
+#define NVC06F_YIELD                                               (0x00000080)
+#define NVC06F_YIELD_OP                                                    1:0
+#define NVC06F_YIELD_OP_NOP                                         0x00000000
+#define NVC06F_YIELD_OP_PBDMA_TIMESLICE                             0x00000001
+#define NVC06F_YIELD_OP_RUNLIST_TIMESLICE                           0x00000002
+#define NVC06F_YIELD_OP_TSG                                         0x00000003


+/* GPFIFO entry format */
+#define NVC06F_GP_ENTRY__SIZE                                   8
+#define NVC06F_GP_ENTRY0_FETCH                                0:0
+#define NVC06F_GP_ENTRY0_FETCH_UNCONDITIONAL           0x00000000
+#define NVC06F_GP_ENTRY0_FETCH_CONDITIONAL             0x00000001
+#define NVC06F_GP_ENTRY0_GET                                 31:2
+#define NVC06F_GP_ENTRY0_OPERAND                             31:0
+#define NVC06F_GP_ENTRY1_GET_HI                               7:0
+#define NVC06F_GP_ENTRY1_PRIV                                 8:8
+#define NVC06F_GP_ENTRY1_PRIV_USER                     0x00000000
+#define NVC06F_GP_ENTRY1_PRIV_KERNEL                   0x00000001
+#define NVC06F_GP_ENTRY1_LEVEL                                9:9
+#define NVC06F_GP_ENTRY1_LEVEL_MAIN                    0x00000000
+#define NVC06F_GP_ENTRY1_LEVEL_SUBROUTINE              0x00000001
+#define NVC06F_GP_ENTRY1_LENGTH                             30:10
+#define NVC06F_GP_ENTRY1_SYNC                               31:31
+#define NVC06F_GP_ENTRY1_SYNC_PROCEED                  0x00000000
+#define NVC06F_GP_ENTRY1_SYNC_WAIT                     0x00000001
+#define NVC06F_GP_ENTRY1_OPCODE                               7:0
+#define NVC06F_GP_ENTRY1_OPCODE_NOP                    0x00000000
+#define NVC06F_GP_ENTRY1_OPCODE_ILLEGAL                0x00000001
+#define NVC06F_GP_ENTRY1_OPCODE_GP_CRC                 0x00000002
+#define NVC06F_GP_ENTRY1_OPCODE_PB_CRC                 0x00000003
+
+/* dma method formats */
+#define NVC06F_DMA_METHOD_ADDRESS_OLD                              12:2
+#define NVC06F_DMA_METHOD_ADDRESS                                  11:0
+#define NVC06F_DMA_SUBDEVICE_MASK                                  15:4
+#define NVC06F_DMA_METHOD_SUBCHANNEL                               15:13
+#define NVC06F_DMA_TERT_OP                                         17:16
+#define NVC06F_DMA_TERT_OP_GRP0_INC_METHOD                         (0x00000000)
+#define NVC06F_DMA_TERT_OP_GRP0_SET_SUB_DEV_MASK                   (0x00000001)
+#define NVC06F_DMA_TERT_OP_GRP0_STORE_SUB_DEV_MASK                 (0x00000002)
+#define NVC06F_DMA_TERT_OP_GRP0_USE_SUB_DEV_MASK                   (0x00000003)
+#define NVC06F_DMA_TERT_OP_GRP2_NON_INC_METHOD                     (0x00000000)
+#define NVC06F_DMA_METHOD_COUNT_OLD                                28:18
+#define NVC06F_DMA_METHOD_COUNT                                    28:16
+#define NVC06F_DMA_IMMD_DATA                                       28:16
+#define NVC06F_DMA_SEC_OP                                          31:29
+#define NVC06F_DMA_SEC_OP_GRP0_USE_TERT                            (0x00000000)
+#define NVC06F_DMA_SEC_OP_INC_METHOD                               (0x00000001)
+#define NVC06F_DMA_SEC_OP_GRP2_USE_TERT                            (0x00000002)
+#define NVC06F_DMA_SEC_OP_NON_INC_METHOD                           (0x00000003)
+#define NVC06F_DMA_SEC_OP_IMMD_DATA_METHOD                         (0x00000004)
+#define NVC06F_DMA_SEC_OP_ONE_INC                                  (0x00000005)
+#define NVC06F_DMA_SEC_OP_RESERVED6                                (0x00000006)
+#define NVC06F_DMA_SEC_OP_END_PB_SEGMENT                           (0x00000007)
+/* dma incrementing method format */
+#define NVC06F_DMA_INCR_ADDRESS                                    11:0
+#define NVC06F_DMA_INCR_SUBCHANNEL                                 15:13
+#define NVC06F_DMA_INCR_COUNT                                      28:16
+#define NVC06F_DMA_INCR_OPCODE                                     31:29
+#define NVC06F_DMA_INCR_OPCODE_VALUE                               (0x00000001)
+#define NVC06F_DMA_INCR_DATA                                       31:0
+/* dma non-incrementing method format */
+#define NVC06F_DMA_NONINCR_ADDRESS                                 11:0
+#define NVC06F_DMA_NONINCR_SUBCHANNEL                              15:13
+#define NVC06F_DMA_NONINCR_COUNT                                   28:16
+#define NVC06F_DMA_NONINCR_OPCODE                                  31:29
+#define NVC06F_DMA_NONINCR_OPCODE_VALUE                            (0x00000003)
+#define NVC06F_DMA_NONINCR_DATA                                    31:0
+/* dma increment-once method format */
+#define NVC06F_DMA_ONEINCR_ADDRESS                                 11:0
+#define NVC06F_DMA_ONEINCR_SUBCHANNEL                              15:13
+#define NVC06F_DMA_ONEINCR_COUNT                                   28:16
+#define NVC06F_DMA_ONEINCR_OPCODE                                  31:29
+#define NVC06F_DMA_ONEINCR_OPCODE_VALUE                            (0x00000005)
+#define NVC06F_DMA_ONEINCR_DATA                                    31:0
+/* dma no-operation format */
+#define NVC06F_DMA_NOP                                             (0x00000000)
+/* dma immediate-data format */
+#define NVC06F_DMA_IMMD_ADDRESS                                    11:0
+#define NVC06F_DMA_IMMD_SUBCHANNEL                                 15:13
+#define NVC06F_DMA_IMMD_DATA                                       28:16
+#define NVC06F_DMA_IMMD_OPCODE                                     31:29
+#define NVC06F_DMA_IMMD_OPCODE_VALUE                               (0x00000004)
+/* dma set sub-device mask format */
+#define NVC06F_DMA_SET_SUBDEVICE_MASK_VALUE                        15:4
+#define NVC06F_DMA_SET_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVC06F_DMA_SET_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000001)
+/* dma store sub-device mask format */
+#define NVC06F_DMA_STORE_SUBDEVICE_MASK_VALUE                      15:4
+#define NVC06F_DMA_STORE_SUBDEVICE_MASK_OPCODE                     31:16
+#define NVC06F_DMA_STORE_SUBDEVICE_MASK_OPCODE_VALUE               (0x00000002)
+/* dma use sub-device mask format */
+#define NVC06F_DMA_USE_SUBDEVICE_MASK_OPCODE                       31:16
+#define NVC06F_DMA_USE_SUBDEVICE_MASK_OPCODE_VALUE                 (0x00000003)
+/* dma end-segment format */
+#define NVC06F_DMA_ENDSEG_OPCODE                                   31:29
+#define NVC06F_DMA_ENDSEG_OPCODE_VALUE                             (0x00000007)
+/* dma legacy incrementing/non-incrementing formats */
+#define NVC06F_DMA_ADDRESS                                         12:2
+#define NVC06F_DMA_SUBCH                                           15:13
+#define NVC06F_DMA_OPCODE3                                         17:16
+#define NVC06F_DMA_OPCODE3_NONE                                    (0x00000000)
+#define NVC06F_DMA_COUNT                                           28:18
+#define NVC06F_DMA_OPCODE                                          31:29
+#define NVC06F_DMA_OPCODE_METHOD                                   (0x00000000)
+#define NVC06F_DMA_OPCODE_NONINC_METHOD                            (0x00000002)
+#define NVC06F_DMA_DATA                                            31:0
+
 #ifdef __cplusplus
 };     /* extern "C" */
 #endif
--- a/kernel-open/nvidia-uvm/clc0b5.h
+++ b/kernel-open/nvidia-uvm/clc0b5.h
@@ -1,19 +1,19 @@
 /*******************************************************************************
-    Copyright (c) 2014 NVIDIA Corporation
+    Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.

-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to
-    deal in the Software without restriction, including without limitation the
-    rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-    sell copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
+    Permission is hereby granted, free of charge, to any person obtaining a
+    copy of this software and associated documentation files (the "Software"),
+    to deal in the Software without restriction, including without limitation
+    the rights to use, copy, modify, merge, publish, distribute, sublicense,
+    and/or sell copies of the Software, and to permit persons to whom the
+    Software is furnished to do so, subject to the following conditions:

-        The above copyright notice and this permission notice shall be
-        included in all copies or substantial portions of the Software.
+    The above copyright notice and this permission notice shall be included in
+    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
@@ -32,6 +32,10 @@ extern "C" {

 #define PASCAL_DMA_COPY_A                                                            (0x0000C0B5)

+#define NVC0B5_NOP                                                              (0x00000100)
+#define NVC0B5_NOP_PARAMETER                                                    31:0
+#define NVC0B5_PM_TRIGGER                                                       (0x00000140)
+#define NVC0B5_PM_TRIGGER_V                                                     31:0
 #define NVC0B5_SET_SEMAPHORE_A                                                  (0x00000240)
 #define NVC0B5_SET_SEMAPHORE_A_UPPER                                            16:0
 #define NVC0B5_SET_SEMAPHORE_B                                                  (0x00000244)
@@ -115,6 +119,10 @@ extern "C" {
 #define NVC0B5_LAUNCH_DMA_SRC_BYPASS_L2                                         20:20
 #define NVC0B5_LAUNCH_DMA_SRC_BYPASS_L2_USE_PTE_SETTING                         (0x00000000)
 #define NVC0B5_LAUNCH_DMA_SRC_BYPASS_L2_FORCE_VOLATILE                          (0x00000001)
+#define NVC0B5_LAUNCH_DMA_DST_BYPASS_L2                                         21:21
+#define NVC0B5_LAUNCH_DMA_DST_BYPASS_L2_USE_PTE_SETTING                         (0x00000000)
+#define NVC0B5_LAUNCH_DMA_DST_BYPASS_L2_FORCE_VOLATILE                          (0x00000001)
+#define NVC0B5_LAUNCH_DMA_RESERVED                                              31:28
 #define NVC0B5_OFFSET_IN_UPPER                                                  (0x00000400)
 #define NVC0B5_OFFSET_IN_UPPER_UPPER                                            16:0
 #define NVC0B5_OFFSET_IN_LOWER                                                  (0x00000404)
@@ -183,6 +191,68 @@ extern "C" {
 #define NVC0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_TWO                      (0x00000001)
 #define NVC0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_THREE                    (0x00000002)
 #define NVC0B5_SET_REMAP_COMPONENTS_NUM_DST_COMPONENTS_FOUR                     (0x00000003)
+#define NVC0B5_SET_DST_BLOCK_SIZE                                               (0x0000070C)
+#define NVC0B5_SET_DST_BLOCK_SIZE_WIDTH                                         3:0
+#define NVC0B5_SET_DST_BLOCK_SIZE_WIDTH_ONE_GOB                                 (0x00000000)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT                                        7:4
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_ONE_GOB                                (0x00000000)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_TWO_GOBS                               (0x00000001)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_FOUR_GOBS                              (0x00000002)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_EIGHT_GOBS                             (0x00000003)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_SIXTEEN_GOBS                           (0x00000004)
+#define NVC0B5_SET_DST_BLOCK_SIZE_HEIGHT_THIRTYTWO_GOBS                         (0x00000005)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH                                         11:8
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_ONE_GOB                                 (0x00000000)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_TWO_GOBS                                (0x00000001)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_FOUR_GOBS                               (0x00000002)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_EIGHT_GOBS                              (0x00000003)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_SIXTEEN_GOBS                            (0x00000004)
+#define NVC0B5_SET_DST_BLOCK_SIZE_DEPTH_THIRTYTWO_GOBS                          (0x00000005)
+#define NVC0B5_SET_DST_BLOCK_SIZE_GOB_HEIGHT                                    15:12
+#define NVC0B5_SET_DST_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_FERMI_8                 (0x00000001)
+#define NVC0B5_SET_DST_WIDTH                                                    (0x00000710)
+#define NVC0B5_SET_DST_WIDTH_V                                                  31:0
+#define NVC0B5_SET_DST_HEIGHT                                                   (0x00000714)
+#define NVC0B5_SET_DST_HEIGHT_V                                                 31:0
+#define NVC0B5_SET_DST_DEPTH                                                    (0x00000718)
+#define NVC0B5_SET_DST_DEPTH_V                                                  31:0
+#define NVC0B5_SET_DST_LAYER                                                    (0x0000071C)
+#define NVC0B5_SET_DST_LAYER_V                                                  31:0
+#define NVC0B5_SET_DST_ORIGIN                                                   (0x00000720)
+#define NVC0B5_SET_DST_ORIGIN_X                                                 15:0
+#define NVC0B5_SET_DST_ORIGIN_Y                                                 31:16
+#define NVC0B5_SET_SRC_BLOCK_SIZE                                               (0x00000728)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_WIDTH                                         3:0
+#define NVC0B5_SET_SRC_BLOCK_SIZE_WIDTH_ONE_GOB                                 (0x00000000)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT                                        7:4
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_ONE_GOB                                (0x00000000)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_TWO_GOBS                               (0x00000001)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_FOUR_GOBS                              (0x00000002)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_EIGHT_GOBS                             (0x00000003)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_SIXTEEN_GOBS                           (0x00000004)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_HEIGHT_THIRTYTWO_GOBS                         (0x00000005)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH                                         11:8
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_ONE_GOB                                 (0x00000000)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_TWO_GOBS                                (0x00000001)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_FOUR_GOBS                               (0x00000002)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_EIGHT_GOBS                              (0x00000003)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_SIXTEEN_GOBS                            (0x00000004)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_DEPTH_THIRTYTWO_GOBS                          (0x00000005)
+#define NVC0B5_SET_SRC_BLOCK_SIZE_GOB_HEIGHT                                    15:12
+#define NVC0B5_SET_SRC_BLOCK_SIZE_GOB_HEIGHT_GOB_HEIGHT_FERMI_8                 (0x00000001)
+#define NVC0B5_SET_SRC_WIDTH                                                    (0x0000072C)
+#define NVC0B5_SET_SRC_WIDTH_V                                                  31:0
+#define NVC0B5_SET_SRC_HEIGHT                                                   (0x00000730)
+#define NVC0B5_SET_SRC_HEIGHT_V                                                 31:0
+#define NVC0B5_SET_SRC_DEPTH                                                    (0x00000734)
+#define NVC0B5_SET_SRC_DEPTH_V                                                  31:0
+#define NVC0B5_SET_SRC_LAYER                                                    (0x00000738)
+#define NVC0B5_SET_SRC_LAYER_V                                                  31:0
+#define NVC0B5_SET_SRC_ORIGIN                                                   (0x0000073C)
+#define NVC0B5_SET_SRC_ORIGIN_X                                                 15:0
+#define NVC0B5_SET_SRC_ORIGIN_Y                                                 31:16
+#define NVC0B5_PM_TRIGGER_END                                                   (0x00001114)
+#define NVC0B5_PM_TRIGGER_END_V                                                 31:0

 #ifdef __cplusplus
 };     /* extern "C" */
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Bernhard Stoeckner	d5a0858f90	565.57.01	2024-10-22 17:38:58 +02:00
Gaurav Juvekar	ed4be64962	560.35.03	2024-08-19 10:46:21 -07:00
Gaurav Juvekar	315fd96d2d	560.31.02	2024-07-31 11:27:06 -07:00
Gaurav Juvekar	448d5cc656	560.28.03	2024-07-19 15:45:15 -07:00
Bernhard Stoeckner	5fdf5032fb	555.58.02 (cherry picked from commit `1795a8bb20`)	2024-07-19 15:38:11 -07:00
Milos Tijanic	171c735e57	555.58 (cherry picked from commit `af77e083a2`)	2024-07-19 15:38:08 -07:00
Bernhard Stoeckner	74ee05e160	555.52.04 (cherry picked from commit `78d807e001`)	2024-07-19 15:38:04 -07:00
Bernhard Stoeckner	3084c04453	555.42.02 (cherry picked from commit `5a1c474040`)	2024-07-19 15:38:00 -07:00
Bernhard Stoeckner	caa2dd11a0	550.100	2024-07-09 15:49:19 +02:00
Bernhard Stoeckner	e45d91de02	550.90.07	2024-06-04 13:48:03 +02:00
Bernhard Stoeckner	083cd9cf17	550.78	2024-04-25 16:24:58 +02:00
Bernhard Stoeckner	ea4c27fad6	550.76	2024-04-17 17:23:37 +02:00
Bernhard Stoeckner	3bf16b890c	550.67	2024-03-19 16:56:28 +01:00
Bernhard Stoeckner	12933b2d3c	550.54.15	2024-03-18 17:52:11 +01:00
Bernhard Stoeckner	476bd34534	550.54.14	2024-02-23 16:37:56 +01:00
Bernhard Stoeckner	91676d6628	550.40.07	2024-01-24 18:28:48 +01:00
Bernhard Stöckner	bb2dac1f20	Update 20_build_bug.yml	2024-01-23 15:30:14 +01:00
Maneet Singh	4c29105335	545.29.06	2023-11-21 13:38:23 -08:00
Andy Ritger	be3cd9abcb	545.29.02	2023-10-31 16:31:08 -07:00
Andy Ritger	a2f89d6b59	Revert "545.29.03" This reverts commit `f364378a65`. 545.29.03 and 545.29.02 are functionally the same for purposes of open-gpu-kernel-modules, but there was poor NVIDIA-internal communication about which driver would actually be released. Revert 545.29.03 so that a subsequent commit can provide 545.29.02 cleanly.	2023-10-31 16:28:17 -07:00
Maneet Singh	f364378a65	545.29.03	2023-10-31 09:44:03 -07:00
Andy Ritger	b5bf85a8e3	545.23.06	2023-10-17 09:25:29 -07:00
Maneet Singh	f59818b751	535.113.01	2023-09-21 10:43:43 -07:00
Bernhard Stoeckner	a8e01be6b2	535.104.05	2023-08-22 15:09:37 +02:00
Bernhard Stoeckner	12c0739352	535.98	2023-08-08 18:28:38 +02:00
Bernhard Stoeckner	29f830f1bb	535.86.10	2023-07-31 18:17:14 +02:00
Bernhard Stoeckner	337e28efda	535.86.05	2023-07-18 16:00:22 +02:00
Bernhard Stoeckner	22a077c4fe	issue template: be clearer about issues with prop driver	2023-07-10 15:58:02 +02:00
Andy Ritger	26458140be	535.54.03	2023-06-14 12:37:59 -07:00
Andy Ritger	eb5c7665a1	535.43.02	2023-05-30 10:11:36 -07:00
Andy Ritger	6dd092ddb7	530.41.03	2023-03-23 11:00:12 -07:00
Andy Ritger	4397463e73	530.30.02	2023-02-28 11:12:44 -08:00
Andy Ritger	e598191e8e	525.89.02	2023-02-08 10:15:15 -08:00
Maneet Singh	1dc88ff75e	525.85.12	2023-01-30 16:30:12 -08:00
Andy Ritger	811073c51e	525.85.05	2023-01-19 10:41:59 -08:00
Andy Ritger	dac2350c7f	525.78.01	2023-01-05 10:40:27 -08:00
Andy Ritger	9594cc0169	525.60.13	2022-12-05 10:49:53 -08:00
Andy Ritger	5f40a5aee5	525.60.11	2022-11-28 13:39:27 -08:00
Andy Ritger	758b4ee818	525.53	2022-11-10 08:39:33 -08:00
Andy Ritger	7c345b838b	520.56.06	2022-10-12 10:30:46 -07:00
Andy Ritger	90eb10774f	520.61.05	2022-10-10 14:59:24 -07:00
Andy Ritger	fe0728787f	515.76	2022-09-20 13:54:59 -07:00
Andy Ritger	9855350159	515.65.01	2022-08-02 08:35:13 -07:00
Joshua Ashton	1e2a1b5fbe	dp_connectorimpl: Use switch-case for dscBitsPerPixelIncrement This is an enum, so we can just use a switch case instead of chaning IFs constantly.	2022-08-02 08:28:14 -07:00
Joshua Ashton	7ded2136ca	dp_connectorimpl: Remove duplicate dscBitsPerPixelIncrement All other values of the BITS_PER_PIXEL_INCREMENT are handled here. This one was previously using a bitwise OR in, then also assigning directly after. Closes: #54	2022-08-02 08:28:14 -07:00
Joshua Ashton	dcdb0c9a7e	osapi: Fix dereferencing NULL pNumaMemSize in rm_get_gpu_numa_info Copy paste error. Closes: #55	2022-08-02 08:28:14 -07:00
Joshua Ashton	9aea50f52f	dp_configcaps: Avoid querying bus if both bLinkActive and bLinkReady is NULL Doesn't fix a crash -- but avoids redundant work due to a copy paste error. Closes #56	2022-08-02 08:28:14 -07:00
Joshua Ashton	28d2504766	nv-pci: Fix nullptr dereference if device was not found Closes #57	2022-08-02 08:28:14 -07:00
Bernhard Stoeckner	88636e701d	update CONTRIBUTING.md to explain cosmetics policy Also adds a blurb about the missing code style guide.	2022-07-07 11:02:42 +02:00
Andy Ritger	94eaea9726	515.57	2022-06-28 08:00:06 -07:00