[rocm-libraries] ROCm/rocm-libraries#5249 (commit 2a114bb)

[CK] [CK_TILE] Improve build and test time of CI with smart dependency parser (#5249) ## Motivation Existing dependency parser needs full build of tests to determine which tests are affected by code changes in a PR. This still takes 2-4 hours for building the tests which slows down the CI as the number of tests grow. To resolve this issue we implemented a smart dependency parser which uses CMake Configure to parse dependencies and build only the affected test cases. We have ensured that two approaches are available 1) CMake pre-build analysis for each PR to ensure fast build and test. 2) Ninja post-build analysis to enable full build for nightly tests. ## Technical Details ```bash ### 1. Configure the project with CMake cmake -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON .. ### 2. Analyze dependencies (no build required!) python3 ../script/dependency-parser/main.py cmake-parse compile_commands.json build.ninja \ --workspace-root .. --output cmake_dependency_mapping.json --parallel 8 ### 3. Find tests affected by changes python3 ../script/dependency-parser/main.py select cmake_dependency_mapping.json origin/develop \ HEAD --test-prefix --output tests_to_run.json ### 4. Build only affected tests ninja $(jq -r '.executables[]' tests_to_run.json | tr '\n' ' ') ### 5. Run affected tests ctest -R "$(jq -r '.regex' tests_to_run.json)" ``` ### Jenkins Integration - Added `buildMode` to jenkinsfile to integrate both `selective` and `full` build methods ### Known Limitations ### 1. Build-Time Generated Headers (HIGH RISK) **Problem:** Files generated during the build process (e.g., via `add_custom_command`) cannot be analyzed before building. **Example:** ```cmake add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/generated/config.hpp COMMAND generate_config.sh DEPENDS template.hpp.in ) ``` **Impact:** If a source file includes `generated/config.hpp`, the dependency won't be detected until after building. **Mitigation:** - CK analysis shows **no generated headers** currently used - If generated headers are added in the future, they must be built first - Recommendation: Generate headers in CMake configure phase (not build phase) when possible ## Test Plan **1. Modified Files:** ``` include/ck_tile/ops/common.hpp include/ck_tile/ops/gemm.hpp include/ck_tile/ops/gemm/warp/warp_gemm.hpp ``` **2. Compare tests selected between `build.ninja` and `cmake-parse` methods** ## Test Result - 1. The test completed in 5-6 minutes finding about 8000+ executables that should be built. - 2. We selected a commit 5ccc1387ea which resulted in same 7 tests with both legacy and new methods. - PR | Legacy tests | Smart tests | Notes -- | -- | -- | -- 5261 | 453 | 455 | Only 2 tests (test_amdgcn_mma and test_amdgcn_sparse_mma) 5168 | 0 | 0 | Changes in dispatcher only. No CK tests invoked. 5249 | 0 | 0 | Changes to dependency parser. No CK tests invoked 5260 | 0 | 0 | Changes in dispatcher only. No CK tests invoked. 5174 | 1 | 1 | One test from FMHA affected by this PR in both cases 5383 | 0 | 0 | Changes are only in benchmark files. Did not trigger any tests 5445 | 1 | 1 | Changes are only to tests/ck_tile/gemm_streamk. Only triggered one streamk test in both cases. 5454 | 3 | 3 | Both methods identified same test_grouped_conv_bwd tests 5427 | 234 | 234 | Core infrastructure header changes. Detected exactly same tests 5388 | 85 | 85 | modifies warp-level GEMM operations (warp_gemm.hpp, warp_gemm_dispatcher.hpp). Correctly identified all the streamK gemm tests ## Submission Checklist - [x ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
2026-04-19 22:39:03 +00:00 · 2026-03-19 05:31:35 +00:00
parent 345a56c55e
commit 652d3456ca
13 changed files with 3585 additions and 210 deletions
--- a/script/dependency-parser/main.py
+++ b/script/dependency-parser/main.py
@@ -6,7 +6,8 @@
 Unified CLI for Ninja Dependency Analysis and Selective Testing

 Features:
- Dependency parsing (from build.ninja)
+- CMake pre-build dependency parsing (using compile_commands.json + clang -MM)
+- Post-build dependency parsing (from build.ninja - legacy)
 - Selective test filtering (between git refs)
 - Code auditing (--audit)
 - Build optimization (--optimize-build)
@@ -16,6 +17,13 @@ import argparse
 import sys


+def run_cmake_dependency_analyzer(args):
+    from src.cmake_dependency_analyzer import main as cmake_main
+
+    sys.argv = ["cmake_dependency_analyzer.py"] + args
+    cmake_main()
+
+
 def run_dependency_parser(args):
    from src.enhanced_ninja_parser import main as ninja_main

@@ -32,13 +40,53 @@ def run_selective_test_filter(args):

 def main():
    parser = argparse.ArgumentParser(
-        description="Unified Ninja Dependency & Selective Testing Tool"
+        description="Unified Dependency Analysis & Selective Testing Tool"
    )
    subparsers = parser.add_subparsers(dest="command", required=True)

-    # Dependency parsing
+    # CMake pre-build dependency parsing (NEW - RECOMMENDED)
+    parser_cmake = subparsers.add_parser(
+        "cmake-parse",
+        help="[NEW] Parse compile_commands.json for pre-build dependency analysis"
+    )
+    parser_cmake.add_argument(
+        "compile_commands",
+        help="Path to compile_commands.json"
+    )
+    parser_cmake.add_argument(
+        "build_ninja",
+        help="Path to build.ninja"
+    )
+    parser_cmake.add_argument(
+        "--workspace-root",
+        default=".",
+        help="Workspace root directory (default: current directory)"
+    )
+    parser_cmake.add_argument(
+        "--output",
+        default="cmake_dependency_mapping.json",
+        help="Output JSON file (default: cmake_dependency_mapping.json)"
+    )
+    parser_cmake.add_argument(
+        "--parallel",
+        type=int,
+        default=8,
+        help="Number of parallel workers (default: 8)"
+    )
+    parser_cmake.add_argument(
+        "--quiet",
+        action="store_true",
+        help="Suppress progress output"
+    )
+    parser_cmake.add_argument(
+        "--force",
+        action="store_true",
+        help="Force regeneration even if cache is valid"
+    )
+
+    # Ninja post-build dependency parsing (LEGACY)
    parser_parse = subparsers.add_parser(
-        "parse", help="Parse build.ninja and generate dependency mapping"
+        "parse", help="[LEGACY] Parse build.ninja post-build (requires full build first)"
    )
    parser_parse.add_argument("build_ninja", help="Path to build.ninja")
    parser_parse.add_argument(
@@ -63,6 +111,15 @@ def main():
        action="store_true",
        help="Only include executables starting with 'test_'",
    )
+    parser_test.add_argument(
+        "--ctest-only",
+        action="store_true",
+        help="Only include tests registered with CTest (excludes EXCLUDE_FROM_ALL targets like benchmarks)",
+    )
+    parser_test.add_argument(
+        "--build-dir",
+        help="Build directory for ctest lookup (optional, default: inferred from depmap_json path)",
+    )
    parser_test.add_argument(
        "--output", help="Output JSON file", default="tests_to_run.json"
    )
@@ -82,7 +139,17 @@ def main():

    args = parser.parse_args()

-    if args.command == "parse":
+    if args.command == "cmake-parse":
+        cmake_args = [args.compile_commands, args.build_ninja]
+        cmake_args += ["--workspace-root", args.workspace_root]
+        cmake_args += ["--output", args.output]
+        cmake_args += ["--parallel", str(args.parallel)]
+        if args.quiet:
+            cmake_args.append("--quiet")
+        if args.force:
+            cmake_args.append("--force")
+        run_cmake_dependency_analyzer(cmake_args)
+    elif args.command == "parse":
        parse_args = [args.build_ninja, args.ninja]
        if args.workspace_root:
            parse_args.append(args.workspace_root)
@@ -93,6 +160,10 @@ def main():
            filter_args.append("--test-prefix")
        if args.all:
            filter_args.append("--all")
+        if args.ctest_only:
+            filter_args.append("--ctest-only")
+        if args.build_dir:
+            filter_args += ["--build-dir", args.build_dir]
        if args.output:
            filter_args += ["--output", args.output]
        run_selective_test_filter(filter_args)