From 14d8dc4714e22aa3c8b5a14e60c2156f27911cbf Mon Sep 17 00:00:00 2001 From: Illia Silin <98187287+illsilin@users.noreply.github.com> Date: Wed, 11 Feb 2026 20:42:43 -0800 Subject: [PATCH] [CK] Fix the launch_tests script. (#4525) ## Motivation Fix the script that filters the tests. ## Technical Details There were several places where the paths had to be updated for the launch_tests script to work correctly. ## Test Plan ## Test Result ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. --------- Co-authored-by: Vidyasagar Ananthan --- script/dependency-parser/README.md | 206 ++++++++++-------- ...e_list_of_files_not_referenced_in_tests.py | 49 +++-- .../src/enhanced_ninja_parser.py | 190 +++++++++------- .../src/selective_test_filter.py | 17 +- script/launch_tests.sh | 35 +-- 5 files changed, 291 insertions(+), 206 deletions(-) diff --git a/script/dependency-parser/README.md b/script/dependency-parser/README.md index ff4a44b9a2..4b2a755d21 100644 --- a/script/dependency-parser/README.md +++ b/script/dependency-parser/README.md @@ -8,166 +8,200 @@ The parser: - Identifies all executables in the Ninja build. - Maps object files to their source and header dependencies using `ninja -t deps`. - Constructs a reverse mapping from each file to all dependent executables. -- Handles multi-executable dependencies and supports parallel processing for scalability. +- Automatically detects monorepo structure (`projects//`) and scopes analysis accordingly. - Exports results in CSV and JSON formats for integration with other tools. ## Features - **Comprehensive Dependency Tracking**: Captures direct source file dependencies and, critically, all included header files via `ninja -t deps`. - **Executable to Object Mapping**: Parses the `build.ninja` file to understand how executables are linked from object files. -- **Object to Source/Header Mapping**: Uses `ninja -t deps` for each object file to get a complete list of its dependencies. +- **Batch Dependency Extraction**: Runs a single `ninja -t deps` call (no arguments) to dump all dependency information at once, then filters in-memory. This avoids the massive overhead of per-object subprocess calls on large build files (e.g., a 246MB `build.ninja` with 29K+ objects completes in ~2 seconds instead of ~54 minutes). +- **Monorepo Awareness**: Automatically detects `projects//` paths, strips them to project-relative paths, and scopes `git diff` to only the relevant subtree. - **File to Executable Inversion**: Inverts the dependency graph to map each file to the set of executables that depend on it. -- **Parallel Processing**: Utilizes a `ThreadPoolExecutor` to run `ninja -t deps` commands in parallel, significantly speeding up analysis for projects with many object files. -- **Filtering**: Option to filter out system files and focus on project-specific dependencies. +- **Filtering**: Filters out system files (`/usr/`, `/opt/rocm/`, etc.) and focuses on project-specific dependencies. - **Multiple Output Formats**: - - **CSV**: `enhanced_file_executable_mapping.csv` - A comma-separated values file where each row lists a file and a semicolon-separated list of executables that depend on it. - - **JSON**: `enhanced_dependency_mapping.json` - A JSON file representing a dictionary where keys are file paths and values are lists of dependent executables. + - **CSV**: `enhanced_file_executable_mapping.csv` - Each row lists a file and a semicolon-separated list of dependent executables. + - **JSON**: `enhanced_dependency_mapping.json` - Includes file-to-executable mapping, executable-to-file mapping, repo metadata, and statistics. - **Robust Error Handling**: Includes error handling for missing files and failed subprocess commands. ## Prerequisites - **Python 3.7+** - **Ninja build system**: The `ninja` executable must be in the system's PATH or its path provided as an argument. -- A **Ninja build directory** containing a `build.ninja` file and the compiled object files. The project should have been built at least once. +- A **Ninja build directory** containing a `build.ninja` file. The project should have been built at least once (even partially) so that `ninja -t deps` has dependency data. + +## Quick Start with launch_tests.sh + +The easiest way to use this tool is via the `launch_tests.sh` wrapper script: + +```bash +# From the monorepo root (or anywhere): +script/launch_tests.sh /path/to/build-dir + +# Uses default build dir (/build) if no argument given: +script/launch_tests.sh +``` + +This script: +1. Discovers the git root (monorepo root) automatically. +2. Runs the dependency parser against `build.ninja`. +3. Runs `git diff` between `origin/develop` and the current branch (scoped to CK files only). +4. Maps changed files to affected tests/examples. +5. Runs the affected tests via `ctest` in chunks. + +Environment variables: +- `CTEST_CHUNK_SIZE`: Number of tests per ctest invocation (default: 10). +- `CTEST_FAIL_FAST`: Set to `true` to stop on first failure (default: `false`). ## Using CMake with Ninja -To use this tool effectively, your C++ project should be configured with CMake to generate Ninja build files and dependency information. Follow these steps: +To use this tool effectively, your C++ project should be configured with CMake to generate Ninja build files: -1. **Configure CMake to use Ninja and generate dependencies:** +1. **Configure CMake to use Ninja:** ```bash - cmake -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release /path/to/your/source + cmake -G Ninja \ + -DCMAKE_PREFIX_PATH=/opt/rocm \ + -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \ + -DCMAKE_BUILD_TYPE=Release \ + -DGPU_TARGETS="gfx942" \ + /path/to/composablekernel ``` - - The `-G Ninja` flag tells CMake to generate Ninja build files. - - `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` is optional but useful for other tooling. - - Ensure your CMakeLists.txt uses `target_include_directories` and proper dependency declarations for accurate results. -2. **Build your project with Ninja:** +2. **Build your project (full or partial):** ```bash + # Full build ninja - ``` - - This step is required to generate all object files and dependency information (`.d` files) that the parser relies on. -3. **Run the dependency parser tool:** + # Or build specific targets + ninja example_gemm_xdl_fp16 example_gemm_xdl_fp16_v3 + ``` + The parser only extracts dependencies for objects that were actually built. + +3. **Run the dependency parser:** ```bash - python main.py parse /path/to/build.ninja --workspace-root /path/to/your/workspace + python main.py parse /path/to/build/build.ninja --workspace-root /path/to/monorepo-root ``` -**Note:** Always run Ninja to ensure all dependencies are up to date before invoking the parser. If you change source files or headers, re-run Ninja first. +**Note:** `--workspace-root` should point to the **git root** (monorepo root) for correct monorepo detection. If omitted, it defaults to `..` relative to the build directory. ## Usage -All features are available via the unified main.py CLI: +All features are available via the unified `main.py` CLI: ```bash -# Dependency parsing (now supports --workspace-root) -python main.py parse examples/build-ninja/build.ninja --workspace-root /path/to/your/workspace +# Dependency parsing +python main.py parse /path/to/build.ninja --workspace-root /path/to/monorepo-root -# Selective test filtering +# Selective test filtering (between git refs) python main.py select enhanced_dependency_mapping.json [--all | --test-prefix] [--output ] -# Code auditing +# Code auditing (list all files and their dependent executables) python main.py audit enhanced_dependency_mapping.json -# Build optimization +# Build optimization (list affected executables for specific changed files) python main.py optimize enhanced_dependency_mapping.json [ ...] ``` -**Arguments:** +### Parse arguments -1. ``: (Required) The full path to the `build.ninja` file within your Ninja build directory. -2. `[--workspace-root ]`: (Optional, recommended) The root directory of your workspace. -3. `[path_to_ninja_executable]`: (Optional) The path to the `ninja` executable if it's not in your system's PATH. Defaults to `ninja`. +| Argument | Required | Description | +|----------|----------|-------------| +| `build_ninja` | Yes | Path to the `build.ninja` file | +| `--workspace-root` | No | Root of the workspace/monorepo (default: `..`) | +| `--ninja` | No | Path to the ninja executable (default: `ninja`) | -**Example:** +### Select arguments -```bash -# Assuming your build directory is 'build-ninja' and it contains 'build.ninja' -python src/enhanced_ninja_parser.py build-ninja/build.ninja - -# With custom workspace root -python src/enhanced_ninja_parser.py build-ninja/build.ninja ninja /path/to/your/workspace - -# If ninja is installed in a custom location -python src/enhanced_ninja_parser.py /path/to/project/build/build.ninja /usr/local/bin/ninja -``` +| Argument | Required | Description | +|----------|----------|-------------| +| `depmap_json` | Yes | Path to `enhanced_dependency_mapping.json` | +| `ref1` | Yes | Source git ref (branch or commit SHA) | +| `ref2` | Yes | Target git ref (branch or commit SHA) | +| `--all` | No | Include all affected executables (default) | +| `--test-prefix` | No | Only include executables starting with `test_` | +| `--output` | No | Output JSON file (default: `tests_to_run.json`) | ## How It Works -1. **Initialization**: - * Takes the path to `build.ninja` and optionally the `ninja` executable. - * Sets up internal data structures to store mappings. +1. **Build File Parsing (`_parse_build_file`)**: + * Reads the `build.ninja` file (~246MB for the full CK monorepo build). + * Uses regular expressions to identify executable link rules and object compilation rules. + * Populates `executable_to_objects` and `object_to_source` mappings. -2. **Build File Parsing (`_parse_build_file`)**: - * Reads the `build.ninja` file. - * Uses regular expressions to identify rules for linking executables (e.g., `build my_exe: link main.o utils.o`) and compiling object files (e.g., `build main.o: cxx ../src/main.cpp`). - * Populates `executable_to_objects` (mapping an executable name to a list of its .o files) and `object_to_source` (mapping an object file to its primary source file). +2. **Batch Dependency Extraction (`_extract_object_dependencies`)**: + * Runs a single `ninja -t deps` command (no arguments) which dumps all dependency information for every built object file. + * Parses the output and filters to only the objects found in `object_to_source`. + * Strips the workspace root prefix from absolute paths to produce project-relative paths. -3. **Object Dependency Extraction (`_extract_all_object_dependencies`)**: - * Iterates through all unique object files identified in the previous step. - * For each object file, it calls `_get_object_dependencies`. - * This process is parallelized using `ThreadPoolExecutor` for efficiency. Each call to `ninja -t deps` runs in a separate thread. +3. **Monorepo Path Detection (`_build_file_to_executable_mapping`)**: + * Applies a regex to detect `projects//` in dependency paths. + * Strips the monorepo prefix so paths are relative to the CK project root (e.g., `include/ck/ck.hpp`). + * Records the detected project name for use by the selective test filter. -4. **Individual Object Dependencies (`_get_object_dependencies`)**: - * For a given object file (e.g., `main.o`), it runs the command: `ninja -t deps main.o` in the build directory. - * This command outputs a list of all files that `main.o` depends on, including its primary source (`main.cpp`) and all headers (`*.h`, `*.hpp`) it includes directly or indirectly. - * The output is parsed, cleaned, and returned as a list of file paths. +4. **File Filtering (`_is_project_file`)**: + * Excludes system files (`/usr/`, `/opt/rocm/`, etc.). + * Includes files in known CK directories (`include/`, `library/`, `test/`, `example/`, etc.). + * Also recognizes monorepo-prefixed paths (`projects/composablekernel/include/`, etc.). -5. **Building Final File-to-Executable Mapping (`_build_file_to_executable_mapping`)**: - * This is the core inversion step. It iterates through each executable and its associated object files. - * For each object file, it looks up the full list of its dependencies (source and headers) obtained in step 3 & 4. - * For every dependent file found, it adds the current executable to that file's entry in the `file_to_executables` dictionary. - * If `filter_project_files` is enabled, it checks each dependency against a list of common system paths (e.g., `/usr/include`, `_deps/`) and excludes them if they match. - -6. **Filtering (`_is_project_file`)**: - * A helper function to determine if a given file path is likely a project file or a system/external library file. This helps in focusing the dependency map on the user's own codebase. - -7. **Output Generation**: - * **`export_to_csv(csv_file)`**: Writes the `file_to_executables` mapping to a CSV file. Each row contains a file path and a semicolon-delimited string of executable names. - * **`export_to_json(json_file)`**: Dumps the `file_to_executables` mapping (where the set of executables is converted to a list) into a JSON file. - * **`print_summary()`**: Prints a summary of the findings, including the number of executables, object files, source files, and header files mapped. +5. **Selective Test Filtering (`selective_test_filter.py`)**: + * Loads the dependency mapping JSON. + * Runs `git diff --name-only` between two refs, scoped to `projects//` when in monorepo mode. + * Strips the monorepo prefix from changed file paths. + * Looks up each changed file in the dependency map to find affected executables. + * Exports the list of tests to run as JSON. ## Output Files -Running the script will generate two files in the same directory as the input `build.ninja` file: +Running the parser generates two files in the build directory: - **`enhanced_file_executable_mapping.csv`**: ```csv - File,Executables - /path/to/project/src/main.cpp,my_exe_1;my_exe_2 - /path/to/project/include/utils.h,my_exe_1;another_test - ... + source_file,executables + "include/ck/ck.hpp","bin/example_gemm_xdl_fp16;bin/example_gemm_xdl_fp16_v3" + "example/01_gemm/gemm_xdl_fp16.cpp","bin/example_gemm_xdl_fp16" ``` - **`enhanced_dependency_mapping.json`**: ```json { - "/path/to/project/src/main.cpp": ["my_exe_1", "my_exe_2"], - "/path/to/project/include/utils.h": ["my_exe_1", "another_test"], - ... + "repo": { + "type": "monorepo", + "project": "composablekernel" + }, + "file_to_executables": { + "include/ck/ck.hpp": ["bin/example_gemm_xdl_fp16", "bin/example_gemm_xdl_fp16_v3"], + "example/01_gemm/gemm_xdl_fp16.cpp": ["bin/example_gemm_xdl_fp16"] + }, + "executable_to_files": { + "bin/example_gemm_xdl_fp16": ["include/ck/ck.hpp", "example/01_gemm/gemm_xdl_fp16.cpp"] + }, + "statistics": { + "total_files": 180, + "total_executables": 20403, + "total_object_files": 29530, + "files_with_multiple_executables": 140 + } } ``` ## Use Cases -- **Impact Analysis**: Determine which executables (especially tests) need to be rebuilt or re-run when a specific source or header file changes. -- **Build Optimization**: Understand the dependency structure to potentially optimize build times. +- **Selective CI/CD Testing**: Run only the tests affected by a PR's changes, cutting CI time dramatically. +- **Impact Analysis**: Determine which executables need to be rebuilt when a header changes. +- **Build Optimization**: Identify which targets are affected by a set of file changes. - **Code Auditing**: Get a clear overview of how files are used across different executables. -- **Selective Testing**: Integrate with CI/CD systems to run only the tests affected by a given set of changes. ## Limitations - Relies on the accuracy of Ninja's dependency information (`ninja -t deps`). If the build system doesn't correctly generate `.d` (dependency) files, the header information might be incomplete. -- The definition of "project file" vs. "system file" is based on a simple path-based heuristic and might need adjustment for specific project structures. -- Performance for extremely large projects (tens of thousands of object files) might still be a consideration, though parallelization helps significantly. +- Only objects that have been **actually built** will have dependency data. A partial build means partial coverage of the dependency map. +- The definition of "project file" vs. "system file" is based on a path-based heuristic and might need adjustment for other project structures. ## Troubleshooting -- **"ninja: command not found"**: Ensure `ninja` is installed and in your PATH, or provide the full path to the executable as the second argument. +- **"ninja: command not found"**: Ensure `ninja` is installed and in your PATH, or provide the full path via `--ninja`. - **"build.ninja not found"**: Double-check the path to your `build.ninja` file. - **Empty or Incomplete Output**: * Make sure the project has been successfully built at least once. `ninja -t deps` relies on information generated during the build. - * Verify that your CMake (or other meta-build system) is configured to generate dependency files for Ninja. -- **Slow Performance**: For very large projects, the number of `ninja -t deps` calls can be substantial. While parallelized, it can still take time. Consider if all object files truly need to be analyzed or if a subset is sufficient for your needs. - -This tool provides a powerful way to gain deep insights into your Ninja project's dependency structure, enabling more intelligent build and test workflows. + * Verify that your CMake is configured to generate dependency files for Ninja (`-G Ninja`). +- **JSON shows `"type": "component"` instead of `"monorepo"`**: Ensure `--workspace-root` points to the **git/monorepo root**, not the CK project root. The parser needs to see `projects//` in the dependency paths to detect monorepo mode. diff --git a/script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py b/script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py index 58bb9e8e93..b6f1c33939 100644 --- a/script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py +++ b/script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py @@ -10,6 +10,7 @@ # Usage: python3 generate_list_of_files_not_referenced_in_tests -f /path/to/enhanced_dependency_mapping/json/file import argparse +import os import subprocess import json @@ -32,23 +33,38 @@ def main(): ref_files = json.load(file) file_to_executables = ref_files["file_to_executables"] - all_files = ( - subprocess.check_output( - 'find ../../include/ ../../library/ ../../profiler/ -type f -iname "*.cpp" -o -iname "*.hpp"', + # Determine the CK project root: go up two levels from this script's location + # (script/dependency-parser/ -> script/ -> CK root) + script_dir = os.path.dirname(os.path.abspath(__file__)) + ck_root = os.path.normpath(os.path.join(script_dir, "..", "..")) + + search_dirs = ["include", "library", "profiler"] + all_files = [] + for d in search_dirs: + search_path = os.path.join(ck_root, d) + if not os.path.isdir(search_path): + print(f"Warning: directory not found: {search_path}") + continue + result = subprocess.check_output( + f'find "{search_path}" -type f -iname "*.cpp" -o -iname "*.hpp"', shell=True, - ) - .decode("utf-8") - .split("\n") - ) - all_files = all_files[:-1] - all_files[:] = [x[6:] for x in all_files] + ).decode("utf-8").split("\n") + # Convert absolute paths to relative paths from CK root + ck_root_prefix = ck_root.rstrip("/") + "/" + for f in result: + f = f.strip() + if f: + if f.startswith(ck_root_prefix): + f = f[len(ck_root_prefix):] + all_files.append(f) all_referenced_files = [] for v in file_to_executables: - if ( - "composablekernel/include/" in v - or "composablekernel/library/" in v - or "composablekernel/profiler/" in v + # Match both standalone paths (include/, library/, profiler/) and + # monorepo paths (projects/composablekernel/include/, etc.) + if any( + d in v + for d in ["include/", "library/", "profiler/"] ): exe_list = file_to_executables[v] else: @@ -60,10 +76,13 @@ def main(): not_referenced_files = {"include": [], "library": [], "profiler": []} for f in all_files: - found = any(f in el for el in all_referenced_files) + # Check if this file appears in any referenced file path + # (handles both relative and monorepo-prefixed paths) + found = any(f in el or el.endswith(f) for el in all_referenced_files) if not found: pos = f.find("/") - not_referenced_files[f[:pos]].append(f) + if pos > 0 and f[:pos] in not_referenced_files: + not_referenced_files[f[:pos]].append(f) print(json.dumps(not_referenced_files, indent="\t")) diff --git a/script/dependency-parser/src/enhanced_ninja_parser.py b/script/dependency-parser/src/enhanced_ninja_parser.py index ec1f835a6b..00c701d0cb 100644 --- a/script/dependency-parser/src/enhanced_ninja_parser.py +++ b/script/dependency-parser/src/enhanced_ninja_parser.py @@ -16,8 +16,6 @@ import sys import subprocess from collections import defaultdict import json -from concurrent.futures import ThreadPoolExecutor, as_completed -import threading class EnhancedNinjaDependencyParser: @@ -32,9 +30,6 @@ class EnhancedNinjaDependencyParser: self.object_to_all_deps = {} # object -> [all_dependencies] self.file_to_executables = defaultdict(set) # file -> {executables} - # Thread safety - self.lock = threading.Lock() - def parse_dependencies(self): """Main method to parse all dependencies.""" print(f"Parsing ninja dependencies from: {self.build_file_path}") @@ -92,80 +87,98 @@ class EnhancedNinjaDependencyParser: print(f"Found {len(self.object_to_source)} object-to-source mappings") def _extract_object_dependencies(self): - """Extract detailed dependencies for all object files using ninja -t deps.""" - object_files = list(self.object_to_source.keys()) - # Process object files in parallel for better performance + """Extract detailed dependencies for all object files using a single ninja -t deps call. + + Previous implementation spawned ninja -t deps per object file (29K+ subprocesses), + each re-parsing the full build.ninja. For large monorepo builds (246MB+ build.ninja), + each call takes 2-14 seconds, making the total ~54 minutes. + + This implementation calls ninja -t deps once (no args) to dump ALL deps in ~2 seconds, + then parses the output and filters to only the objects we care about. + """ + object_files = set(self.object_to_source.keys()) if not object_files: print("No object files found - skipping dependency extraction") return - max_workers = min(128, len(object_files)) # Limit concurrent processes + print(f"Running single 'ninja -t deps' call for all built objects...") - with ThreadPoolExecutor(max_workers=max_workers) as executor: - # Submit all object files for processing - future_to_obj = { - executor.submit(self._get_object_dependencies, obj): obj - for obj in object_files - } - # Process completed futures - completed = 0 - for future in as_completed(future_to_obj): - obj_file = future_to_obj[future] - try: - dependencies = future.result() - with self.lock: - self.object_to_all_deps[obj_file] = dependencies - completed += 1 - if completed % 100 == 0: - print( - f"Processed {completed}/{len(object_files)} object files..." - ) - except Exception as e: - print(f"Error processing {obj_file}: {e}") - - print( - f"Completed dependency extraction for {len(self.object_to_all_deps)} object files" - ) - - def _get_object_dependencies(self, object_file): - """Get all dependencies for a single object file using ninja -t deps.""" try: - # Run ninja -t deps for this object file - cmd = [self.ninja_executable, "-t", "deps", object_file] + cmd = [self.ninja_executable, "-t", "deps"] result = subprocess.run( - cmd, cwd=self.build_dir, capture_output=True, text=True, timeout=30 + cmd, cwd=self.build_dir, capture_output=True, text=True, timeout=120 ) - if result.returncode != 0: - return [] + if result.returncode != 0 and not result.stdout: + print(f"Warning: ninja -t deps returned code {result.returncode}") + if result.stderr: + print(f" stderr: {result.stderr.strip()}") + return - dependencies = [] - lines = result.stdout.strip().split("\n") + # Parse the combined output: each block starts with an object name line, + # followed by indented dependency lines + ws_root = getattr(self, "workspace_root", "..") + ws_prefix = ws_root.rstrip("/") + "/" - for line in lines[1:]: # Skip first line with metadata - line = line.strip() - if line and not line.startswith("#"): - # Convert absolute paths to relative paths from workspace root - dep_file = line - ws_root = getattr(self, "workspace_root", "..") - ws_prefix = ws_root.rstrip("/") + "/" - if dep_file.startswith(ws_prefix): - dep_file = dep_file[len(ws_prefix) :] - dependencies.append(dep_file) + current_obj = None + current_deps = [] + matched = 0 - return dependencies + for line in result.stdout.split("\n"): + if not line: + continue + if not line.startswith(" ") and not line.startswith("\t"): + # This is an object header line like: + # some/path/foo.cpp.o: #deps 42, deps mtime ... (VALID) + # Save the previous block if it was relevant + if current_obj and current_obj in object_files: + self.object_to_all_deps[current_obj] = current_deps + matched += 1 + if matched % 100 == 0: + print(f" Matched {matched} objects so far...") + # Parse the new object name (everything before the colon) + colon_pos = line.find(":") + if colon_pos > 0: + current_obj = line[:colon_pos].strip() + current_deps = [] + else: + current_obj = None + current_deps = [] + else: + # Indented dependency line + if current_obj is not None: + dep_file = line.strip() + if dep_file and not dep_file.startswith("#"): + # Strip workspace root prefix from absolute paths + if dep_file.startswith(ws_prefix): + dep_file = dep_file[len(ws_prefix):] + current_deps.append(dep_file) + # Don't forget the last block + if current_obj and current_obj in object_files: + self.object_to_all_deps[current_obj] = current_deps + matched += 1 + + except subprocess.TimeoutExpired: + print("Error: ninja -t deps timed out after 120 seconds") + return except Exception as e: - print(f"Error getting dependencies for {object_file}: {e}") - return [] + print(f"Error running ninja -t deps: {e}") + return + + print( + f"Completed dependency extraction for {len(self.object_to_all_deps)} " + f"of {len(object_files)} object files" + ) def _build_file_to_executable_mapping(self): """Build the final mapping from files to executables.""" print("Building file-to-executable mapping...") # For monorepo, truncate the path before and including projects/ + # This regex matches both absolute and relative monorepo paths self.project = None - rl_regex = rf"rocm-libraries[\\/]+projects[\\/]+([^\\/]+)[\\/]+(.*)" + rl_regex = rf"(?:^|.*[\\/])projects[\\/]+([^\\/]+)[\\/]+(.*)" for exe, object_files in self.executable_to_objects.items(): for obj_file in object_files: # Add all dependencies of this object file @@ -195,34 +208,51 @@ class EnhancedNinjaDependencyParser: print(f" {f}: {len(exes)} executables") def _is_project_file(self, file_path): - """Determine if a file is part of the project (not system files).""" - # Include files that are clearly part of the project - if any( - file_path.startswith(prefix) - for prefix in [ - "projects/composablekernel/include/", - "projects/composablekernel/library/", - "projects/composablekernel/test/", - "projects/composablekernel/example/", - "projects/composablekernel/src/", - "projects/composablekernel/profiler/", - "projects/composablekernel/build/include/", - "projects/composablekernel/build/_deps/gtest", - "projects/composablekernel/client_example", - "projects/composablekernel/codegen", - "projects/composablekernel/tile_engine", - ] - ): - return True + """Determine if a file is part of the project (not system files). - # Exclude system files + Handles both standalone-style paths (e.g., include/ck/...) and + monorepo-style paths (e.g., projects/composablekernel/include/ck/...). + """ + # Exclude system files first (absolute paths to system dirs) if any( file_path.startswith(prefix) for prefix in ["/usr/", "/opt/rocm", "/lib/", "/system/", "/local/"] ): return False - # Include files with common source/header extensions + # Project directory prefixes (without monorepo prefix). + # These match paths relative to the CK project root. + project_dirs = [ + "include/", + "library/", + "test/", + "example/", + "src/", + "profiler/", + "build/include/", + "build/_deps/gtest", + "client_example", + "codegen", + "tile_engine", + "dispatcher", + "experimental", + "tutorial", + ] + + # Check both stripped paths (relative to CK root) and + # monorepo-prefixed paths (relative to monorepo root) + if any(file_path.startswith(prefix) for prefix in project_dirs): + return True + + # Also check monorepo-style paths (projects/composablekernel/...) + if any( + file_path.startswith(f"projects/composablekernel/{prefix}") + for prefix in project_dirs + ): + return True + + # Include files with common source/header extensions that weren't + # excluded as system files above if file_path.endswith( (".cpp", ".hpp", ".h", ".c", ".cc", ".cxx", ".cu", ".hip", ".inc") ): @@ -344,7 +374,7 @@ def main(): sys.exit(1) parser = EnhancedNinjaDependencyParser(build_file, ninja_path) - parser.workspace_root = workspace_root # Attach for use in _get_object_dependencies + parser.workspace_root = workspace_root # Attach for use in _extract_object_dependencies parser.parse_dependencies() parser.print_summary() diff --git a/script/dependency-parser/src/selective_test_filter.py b/script/dependency-parser/src/selective_test_filter.py index 465db38615..9bae830351 100644 --- a/script/dependency-parser/src/selective_test_filter.py +++ b/script/dependency-parser/src/selective_test_filter.py @@ -34,15 +34,12 @@ import os def get_changed_files(ref1, ref2, project: str = None): """Return a set of files changed between two git refs.""" try: + cmd = ["git", "diff", "--name-only", ref1, ref2] + if project: + # Scope git diff to only this project's subtree for efficiency + cmd += ["--", f"projects/{project}/"] result = subprocess.run( - ["pwd"], - capture_output=True, - text=True, - check=True, - ) - print("cwd:", result.stdout) - result = subprocess.run( - ["git", "diff", "--name-only", ref1, ref2], + cmd, capture_output=True, text=True, check=True, @@ -105,7 +102,7 @@ def main(): if not os.path.exists(depmap_json): print(f"Dependency map JSON not found: {depmap_json}") sys.exit(1) - file_to_executables = load_depmap(depmap_json) + file_to_executables, _ = load_depmap(depmap_json) for f, exes in file_to_executables.items(): print(f"{f}: {', '.join(exes)}") print(f"Total files: {len(file_to_executables)}") @@ -122,7 +119,7 @@ def main(): if not os.path.exists(depmap_json): print(f"Dependency map JSON not found: {depmap_json}") sys.exit(1) - file_to_executables = load_depmap(depmap_json) + file_to_executables, _ = load_depmap(depmap_json) affected_executables = set() for f in changed_files: if f in file_to_executables: diff --git a/script/launch_tests.sh b/script/launch_tests.sh index 1911613023..466b2bbc43 100755 --- a/script/launch_tests.sh +++ b/script/launch_tests.sh @@ -2,26 +2,30 @@ # Copyright (c) Advanced Micro Devices, Inc., or its affiliates. # SPDX-License-Identifier: MIT +# Usage: launch_tests.sh [BUILD_DIR] +# BUILD_DIR: Path to the Ninja build directory (default: /build) + # Get the directory where the script is located -BUILD_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -# Go one level up to PACKAGE_HOME -PACKAGE_HOME="$(dirname "$BUILD_DIR")" +# Go one level up to PACKAGE_HOME (the CK project root) +PACKAGE_HOME="$(dirname "$SCRIPT_DIR")" -SCRIPT_DIR="$PACKAGE_HOME/script/" +# Discover the monorepo root (git toplevel), falling back to PACKAGE_HOME +GIT_ROOT="$(git -C "$PACKAGE_HOME" rev-parse --show-toplevel 2>/dev/null)" || GIT_ROOT="$PACKAGE_HOME" -# Search for build.ninja under PACKAGE_HOME -BUILD_NINJA_FILE="$PACKAGE_HOME/build/build.ninja" +# Accept an optional build directory argument; default to $PACKAGE_HOME/build +BUILD_DIR="${1:-$PACKAGE_HOME/build}" +BUILD_NINJA_FILE="$BUILD_DIR/build.ninja" -if [ -z "$BUILD_NINJA_FILE" ]; then - echo "Error: build.ninja not found under $PACKAGE_HOME" +if [ ! -f "$BUILD_NINJA_FILE" ]; then + echo "Error: build.ninja not found at $BUILD_NINJA_FILE" + echo "Usage: $0 [BUILD_DIR]" + echo "Please build the project first (e.g., cmake -G Ninja ... && ninja)" exit 1 fi -python3 "$SCRIPT_DIR/dependency-parser/main.py" parse "$BUILD_NINJA_FILE" --workspace-root "$PACKAGE_HOME" - -# Get the directory containing build.ninja -BUILD_DIR=$(dirname "$BUILD_NINJA_FILE") +python3 "$SCRIPT_DIR/dependency-parser/main.py" parse "$BUILD_NINJA_FILE" --workspace-root "$GIT_ROOT" # Path to enhanced_dependency_mapping.json in the same directory JSON_FILE="$BUILD_DIR/enhanced_dependency_mapping.json" @@ -32,10 +36,11 @@ if [ ! -f "$JSON_FILE" ]; then exit 1 fi -branch=$(git rev-parse --abbrev-ref HEAD) +branch=$(git -C "$GIT_ROOT" rev-parse --abbrev-ref HEAD) -# Run the command -python3 "$SCRIPT_DIR/dependency-parser/main.py" select "$JSON_FILE" origin/develop $branch +# Run the command from the git root so that git diff paths are correct +cd "$GIT_ROOT" +python3 "$SCRIPT_DIR/dependency-parser/main.py" select "$JSON_FILE" origin/develop "$branch" # Path to tests_to_run.json in the same directory TEST_FILE="tests_to_run.json"