ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-04-19 22:39:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

[rocm-libraries] ROCm/rocm-libraries#4525 (commit 7f34b22)

Browse Source 
[CK] Fix the launch_tests script.

## Motivation

Fix the script that filters the tests.

## Technical Details

There were several places where the paths had to be updated for the
launch_tests script to work correctly.

## Test Plan

<!-- Explain any relevant testing done to verify this PR. -->

## Test Result

<!-- Briefly summarize test outcomes. -->

## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
This commit is contained in:
Illia Silin
2026-02-12 04:43:27 +00:00
committed by assistant-librarian[bot]
parent e1e2f7ac2e
commit 47c7c034e9
5 changed files with 291 additions and 206 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
script/dependency-parser/README.md
View File

@@ -8,166 +8,200 @@ The parser:
- Identifies all executables in the Ninja build.
- Maps object files to their source and header dependencies using `ninja -t deps`.
- Constructs a reverse mapping from each file to all dependent executables.
- Handles multi-executable dependencies and supports parallel processing for scalability.
- Automatically detects monorepo structure (`projects/<name>/`) and scopes analysis accordingly.
- Exports results in CSV and JSON formats for integration with other tools.
## Features
- **Comprehensive Dependency Tracking**: Captures direct source file dependencies and, critically, all included header files via `ninja -t deps`.
- **Executable to Object Mapping**: Parses the `build.ninja` file to understand how executables are linked from object files.
- **Object to Source/Header Mapping**: Uses `ninja -t deps` for each object file to get a complete list of its dependencies.
- **Batch Dependency Extraction**: Runs a single `ninja -t deps` call (no arguments) to dump all dependency information at once, then filters in-memory. This avoids the massive overhead of per-object subprocess calls on large build files (e.g., a 246MB `build.ninja` with 29K+ objects completes in ~2 seconds instead of ~54 minutes).
- **Monorepo Awareness**: Automatically detects `projects/<project>/` paths, strips them to project-relative paths, and scopes `git diff` to only the relevant subtree.
- **File to Executable Inversion**: Inverts the dependency graph to map each file to the set of executables that depend on it.
- **Parallel Processing**: Utilizes a `ThreadPoolExecutor` to run `ninja -t deps` commands in parallel, significantly speeding up analysis for projects with many object files.
- **Filtering**: Option to filter out system files and focus on project-specific dependencies.
- **Filtering**: Filters out system files (`/usr/`, `/opt/rocm/`, etc.) and focuses on project-specific dependencies.
- **Multiple Output Formats**:
- **CSV**: `enhanced_file_executable_mapping.csv` - A comma-separated values file where each row lists a file and a semicolon-separated list of executables that depend on it.
- **JSON**: `enhanced_dependency_mapping.json` - A JSON file representing a dictionary where keys are file paths and values are lists of dependent executables.
- **CSV**: `enhanced_file_executable_mapping.csv` - Each row lists a file and a semicolon-separated list of dependent executables.
- **JSON**: `enhanced_dependency_mapping.json` - Includes file-to-executable mapping, executable-to-file mapping, repo metadata, and statistics.
- **Robust Error Handling**: Includes error handling for missing files and failed subprocess commands.
## Prerequisites
- **Python 3.7+**
- **Ninja build system**: The `ninja` executable must be in the system's PATH or its path provided as an argument.
- A **Ninja build directory** containing a `build.ninja` file and the compiled object files. The project should have been built at least once.
- A **Ninja build directory** containing a `build.ninja` file. The project should have been built at least once (even partially) so that `ninja -t deps` has dependency data.
## Quick Start with launch_tests.sh
The easiest way to use this tool is via the `launch_tests.sh` wrapper script:
```bash
# From the monorepo root (or anywhere):
script/launch_tests.sh /path/to/build-dir
# Uses default build dir (<CK_ROOT>/build) if no argument given:
script/launch_tests.sh
```
This script:
1. Discovers the git root (monorepo root) automatically.
2. Runs the dependency parser against `build.ninja`.
3. Runs `git diff` between `origin/develop` and the current branch (scoped to CK files only).
4. Maps changed files to affected tests/examples.
5. Runs the affected tests via `ctest` in chunks.
Environment variables:
- `CTEST_CHUNK_SIZE`: Number of tests per ctest invocation (default: 10).
- `CTEST_FAIL_FAST`: Set to `true` to stop on first failure (default: `false`).
## Using CMake with Ninja
To use this tool effectively, your C++ project should be configured with CMake to generate Ninja build files and dependency information. Follow these steps:
To use this tool effectively, your C++ project should be configured with CMake to generate Ninja build files:
1. **Configure CMake to use Ninja and generate dependencies:**
1. **Configure CMake to use Ninja:**
```bash
cmake -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release /path/to/your/source
cmake -G Ninja \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \
-DCMAKE_BUILD_TYPE=Release \
-DGPU_TARGETS="gfx942" \
/path/to/composablekernel
```
- The `-G Ninja` flag tells CMake to generate Ninja build files.
- `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` is optional but useful for other tooling.
- Ensure your CMakeLists.txt uses `target_include_directories` and proper dependency declarations for accurate results.
2. **Build your project with Ninja:**
2. **Build your project (full or partial):**
```bash
# Full build
ninja
```
- This step is required to generate all object files and dependency information (`.d` files) that the parser relies on.
3. **Run the dependency parser tool:**
# Or build specific targets
ninja example_gemm_xdl_fp16 example_gemm_xdl_fp16_v3
```
The parser only extracts dependencies for objects that were actually built.
3. **Run the dependency parser:**
```bash
python main.py parse /path/to/build.ninja --workspace-root /path/to/your/workspace
python main.py parse /path/to/build/build.ninja --workspace-root /path/to/monorepo-root
```
**Note:** Always run Ninja to ensure all dependencies are up to date before invoking the parser. If you change source files or headers, re-run Ninja first.
**Note:** `--workspace-root` should point to the **git root** (monorepo root) for correct monorepo detection. If omitted, it defaults to `..` relative to the build directory.
## Usage
All features are available via the unified main.py CLI:
All features are available via the unified `main.py` CLI:
```bash
# Dependency parsing (now supports --workspace-root)
python main.py parse examples/build-ninja/build.ninja --workspace-root /path/to/your/workspace
# Dependency parsing
python main.py parse /path/to/build.ninja --workspace-root /path/to/monorepo-root
# Selective test filtering
# Selective test filtering (between git refs)
python main.py select enhanced_dependency_mapping.json <ref1> <ref2> [--all | --test-prefix] [--output <output_json>]
# Code auditing
# Code auditing (list all files and their dependent executables)
python main.py audit enhanced_dependency_mapping.json
# Build optimization
# Build optimization (list affected executables for specific changed files)
python main.py optimize enhanced_dependency_mapping.json <changed_file1> [<changed_file2> ...]
```
**Arguments:**
### Parse arguments
1. `<path_to_build.ninja>`: (Required) The full path to the `build.ninja` file within your Ninja build directory.
2. `[--workspace-root <workspace_root>]`: (Optional, recommended) The root directory of your workspace.
3. `[path_to_ninja_executable]`: (Optional) The path to the `ninja` executable if it's not in your system's PATH. Defaults to `ninja`.
| Argument | Required | Description |
|----------|----------|-------------|
| `build_ninja` | Yes | Path to the `build.ninja` file |
| `--workspace-root` | No | Root of the workspace/monorepo (default: `..`) |
| `--ninja` | No | Path to the ninja executable (default: `ninja`) |
**Example:**
### Select arguments
```bash
# Assuming your build directory is 'build-ninja' and it contains 'build.ninja'
python src/enhanced_ninja_parser.py build-ninja/build.ninja
# With custom workspace root
python src/enhanced_ninja_parser.py build-ninja/build.ninja ninja /path/to/your/workspace
# If ninja is installed in a custom location
python src/enhanced_ninja_parser.py /path/to/project/build/build.ninja /usr/local/bin/ninja
```
| Argument | Required | Description |
|----------|----------|-------------|
| `depmap_json` | Yes | Path to `enhanced_dependency_mapping.json` |
| `ref1` | Yes | Source git ref (branch or commit SHA) |
| `ref2` | Yes | Target git ref (branch or commit SHA) |
| `--all` | No | Include all affected executables (default) |
| `--test-prefix` | No | Only include executables starting with `test_` |
| `--output` | No | Output JSON file (default: `tests_to_run.json`) |
## How It Works
1. **Initialization**:
* Takes the path to `build.ninja` and optionally the `ninja` executable.
* Sets up internal data structures to store mappings.
1. **Build File Parsing (`_parse_build_file`)**:
* Reads the `build.ninja` file (~246MB for the full CK monorepo build).
* Uses regular expressions to identify executable link rules and object compilation rules.
* Populates `executable_to_objects` and `object_to_source` mappings.
2. **Build File Parsing (`_parse_build_file`)**:
* Reads the `build.ninja` file.
* Uses regular expressions to identify rules for linking executables (e.g., `build my_exe: link main.o utils.o`) and compiling object files (e.g., `build main.o: cxx ../src/main.cpp`).
* Populates `executable_to_objects` (mapping an executable name to a list of its .o files) and `object_to_source` (mapping an object file to its primary source file).
2. **Batch Dependency Extraction (`_extract_object_dependencies`)**:
* Runs a single `ninja -t deps` command (no arguments) which dumps all dependency information for every built object file.
* Parses the output and filters to only the objects found in `object_to_source`.
* Strips the workspace root prefix from absolute paths to produce project-relative paths.
3. **Object Dependency Extraction (`_extract_all_object_dependencies`)**:
* Iterates through all unique object files identified in the previous step.
* For each object file, it calls `_get_object_dependencies`.
* This process is parallelized using `ThreadPoolExecutor` for efficiency. Each call to `ninja -t deps` runs in a separate thread.
3. **Monorepo Path Detection (`_build_file_to_executable_mapping`)**:
* Applies a regex to detect `projects/<project_name>/` in dependency paths.
* Strips the monorepo prefix so paths are relative to the CK project root (e.g., `include/ck/ck.hpp`).
* Records the detected project name for use by the selective test filter.
4. **Individual Object Dependencies (`_get_object_dependencies`)**:
* For a given object file (e.g., `main.o`), it runs the command: `ninja -t deps main.o` in the build directory.
* This command outputs a list of all files that `main.o` depends on, including its primary source (`main.cpp`) and all headers (`*.h`, `*.hpp`) it includes directly or indirectly.
* The output is parsed, cleaned, and returned as a list of file paths.
4. **File Filtering (`_is_project_file`)**:
* Excludes system files (`/usr/`, `/opt/rocm/`, etc.).
* Includes files in known CK directories (`include/`, `library/`, `test/`, `example/`, etc.).
* Also recognizes monorepo-prefixed paths (`projects/composablekernel/include/`, etc.).
5. **Building Final File-to-Executable Mapping (`_build_file_to_executable_mapping`)**:
* This is the core inversion step. It iterates through each executable and its associated object files.
* For each object file, it looks up the full list of its dependencies (source and headers) obtained in step 3 & 4.
* For every dependent file found, it adds the current executable to that file's entry in the `file_to_executables` dictionary.
* If `filter_project_files` is enabled, it checks each dependency against a list of common system paths (e.g., `/usr/include`, `_deps/`) and excludes them if they match.
6. **Filtering (`_is_project_file`)**:
* A helper function to determine if a given file path is likely a project file or a system/external library file. This helps in focusing the dependency map on the user's own codebase.
7. **Output Generation**:
* **`export_to_csv(csv_file)`**: Writes the `file_to_executables` mapping to a CSV file. Each row contains a file path and a semicolon-delimited string of executable names.
* **`export_to_json(json_file)`**: Dumps the `file_to_executables` mapping (where the set of executables is converted to a list) into a JSON file.
* **`print_summary()`**: Prints a summary of the findings, including the number of executables, object files, source files, and header files mapped.
5. **Selective Test Filtering (`selective_test_filter.py`)**:
* Loads the dependency mapping JSON.
* Runs `git diff --name-only` between two refs, scoped to `projects/<project>/` when in monorepo mode.
* Strips the monorepo prefix from changed file paths.
* Looks up each changed file in the dependency map to find affected executables.
* Exports the list of tests to run as JSON.
## Output Files
Running the script will generate two files in the same directory as the input `build.ninja` file:
Running the parser generates two files in the build directory:
- **`enhanced_file_executable_mapping.csv`**:
```csv
File,Executables
/path/to/project/src/main.cpp,my_exe_1;my_exe_2
/path/to/project/include/utils.h,my_exe_1;another_test
...
source_file,executables
"include/ck/ck.hpp","bin/example_gemm_xdl_fp16;bin/example_gemm_xdl_fp16_v3"
"example/01_gemm/gemm_xdl_fp16.cpp","bin/example_gemm_xdl_fp16"
```
- **`enhanced_dependency_mapping.json`**:
```json
{
"/path/to/project/src/main.cpp": ["my_exe_1", "my_exe_2"],
"/path/to/project/include/utils.h": ["my_exe_1", "another_test"],
...
"repo": {
"type": "monorepo",
"project": "composablekernel"
},
"file_to_executables": {
"include/ck/ck.hpp": ["bin/example_gemm_xdl_fp16", "bin/example_gemm_xdl_fp16_v3"],
"example/01_gemm/gemm_xdl_fp16.cpp": ["bin/example_gemm_xdl_fp16"]
},
"executable_to_files": {
"bin/example_gemm_xdl_fp16": ["include/ck/ck.hpp", "example/01_gemm/gemm_xdl_fp16.cpp"]
},
"statistics": {
"total_files": 180,
"total_executables": 20403,
"total_object_files": 29530,
"files_with_multiple_executables": 140
}
}
```
## Use Cases
- **Impact Analysis**: Determine which executables (especially tests) need to be rebuilt or re-run when a specific source or header file changes.
- **Build Optimization**: Understand the dependency structure to potentially optimize build times.
- **Selective CI/CD Testing**: Run only the tests affected by a PR's changes, cutting CI time dramatically.
- **Impact Analysis**: Determine which executables need to be rebuilt when a header changes.
- **Build Optimization**: Identify which targets are affected by a set of file changes.
- **Code Auditing**: Get a clear overview of how files are used across different executables.
- **Selective Testing**: Integrate with CI/CD systems to run only the tests affected by a given set of changes.
## Limitations
- Relies on the accuracy of Ninja's dependency information (`ninja -t deps`). If the build system doesn't correctly generate `.d` (dependency) files, the header information might be incomplete.
- The definition of "project file" vs. "system file" is based on a simple path-based heuristic and might need adjustment for specific project structures.
- Performance for extremely large projects (tens of thousands of object files) might still be a consideration, though parallelization helps significantly.
- Only objects that have been **actually built** will have dependency data. A partial build means partial coverage of the dependency map.
- The definition of "project file" vs. "system file" is based on a path-based heuristic and might need adjustment for other project structures.
## Troubleshooting
- **"ninja: command not found"**: Ensure `ninja` is installed and in your PATH, or provide the full path to the executable as the second argument.
- **"ninja: command not found"**: Ensure `ninja` is installed and in your PATH, or provide the full path via `--ninja`.
- **"build.ninja not found"**: Double-check the path to your `build.ninja` file.
- **Empty or Incomplete Output**:
* Make sure the project has been successfully built at least once. `ninja -t deps` relies on information generated during the build.
* Verify that your CMake (or other meta-build system) is configured to generate dependency files for Ninja.
- **Slow Performance**: For very large projects, the number of `ninja -t deps` calls can be substantial. While parallelized, it can still take time. Consider if all object files truly need to be analyzed or if a subset is sufficient for your needs.
This tool provides a powerful way to gain deep insights into your Ninja project's dependency structure, enabling more intelligent build and test workflows.
* Verify that your CMake is configured to generate dependency files for Ninja (`-G Ninja`).
- **JSON shows `"type": "component"` instead of `"monorepo"`**: Ensure `--workspace-root` points to the **git/monorepo root**, not the CK project root. The parser needs to see `projects/<name>/` in the dependency paths to detect monorepo mode.

49
script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py
View File

@@ -10,6 +10,7 @@
# Usage: python3 generate_list_of_files_not_referenced_in_tests -f /path/to/enhanced_dependency_mapping/json/file
import argparse
import os
import subprocess
import json
@@ -32,23 +33,38 @@ def main():
ref_files = json.load(file)
file_to_executables = ref_files["file_to_executables"]
all_files = (
subprocess.check_output(
'find ../../include/ ../../library/ ../../profiler/ -type f -iname "*.cpp" -o -iname "*.hpp"',
# Determine the CK project root: go up two levels from this script's location
# (script/dependency-parser/ -> script/ -> CK root)
script_dir = os.path.dirname(os.path.abspath(__file__))
ck_root = os.path.normpath(os.path.join(script_dir, "..", ".."))
search_dirs = ["include", "library", "profiler"]
all_files = []
for d in search_dirs:
search_path = os.path.join(ck_root, d)
if not os.path.isdir(search_path):
print(f"Warning: directory not found: {search_path}")
continue
result = subprocess.check_output(
f'find "{search_path}" -type f -iname "*.cpp" -o -iname "*.hpp"',
shell=True,
)
.decode("utf-8")
.split("\n")
)
all_files = all_files[:-1]
all_files[:] = [x[6:] for x in all_files]
).decode("utf-8").split("\n")
# Convert absolute paths to relative paths from CK root
ck_root_prefix = ck_root.rstrip("/") + "/"
for f in result:
f = f.strip()
if f:
if f.startswith(ck_root_prefix):
f = f[len(ck_root_prefix):]
all_files.append(f)
all_referenced_files = []
for v in file_to_executables:
if (
"composablekernel/include/" in v
or "composablekernel/library/" in v
or "composablekernel/profiler/" in v
# Match both standalone paths (include/, library/, profiler/) and
# monorepo paths (projects/composablekernel/include/, etc.)
if any(
d in v
for d in ["include/", "library/", "profiler/"]
):
exe_list = file_to_executables[v]
else:
@@ -60,10 +76,13 @@ def main():
not_referenced_files = {"include": [], "library": [], "profiler": []}
for f in all_files:
found = any(f in el for el in all_referenced_files)
# Check if this file appears in any referenced file path
# (handles both relative and monorepo-prefixed paths)
found = any(f in el or el.endswith(f) for el in all_referenced_files)
if not found:
pos = f.find("/")
not_referenced_files[f[:pos]].append(f)
if pos > 0 and f[:pos] in not_referenced_files:
not_referenced_files[f[:pos]].append(f)
print(json.dumps(not_referenced_files, indent="\t"))

190
script/dependency-parser/src/enhanced_ninja_parser.py
View File

@@ -16,8 +16,6 @@ import sys
import subprocess
from collections import defaultdict
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading
class EnhancedNinjaDependencyParser:
@@ -32,9 +30,6 @@ class EnhancedNinjaDependencyParser:
self.object_to_all_deps = {} # object -> [all_dependencies]
self.file_to_executables = defaultdict(set) # file -> {executables}
# Thread safety
self.lock = threading.Lock()
def parse_dependencies(self):
"""Main method to parse all dependencies."""
print(f"Parsing ninja dependencies from: {self.build_file_path}")
@@ -92,80 +87,98 @@ class EnhancedNinjaDependencyParser:
print(f"Found {len(self.object_to_source)} object-to-source mappings")
def _extract_object_dependencies(self):
"""Extract detailed dependencies for all object files using ninja -t deps."""
object_files = list(self.object_to_source.keys())
# Process object files in parallel for better performance
"""Extract detailed dependencies for all object files using a single ninja -t deps call.
Previous implementation spawned ninja -t deps per object file (29K+ subprocesses),
each re-parsing the full build.ninja. For large monorepo builds (246MB+ build.ninja),
each call takes 2-14 seconds, making the total ~54 minutes.
This implementation calls ninja -t deps once (no args) to dump ALL deps in ~2 seconds,
then parses the output and filters to only the objects we care about.
"""
object_files = set(self.object_to_source.keys())
if not object_files:
print("No object files found - skipping dependency extraction")
return
max_workers = min(128, len(object_files)) # Limit concurrent processes
print(f"Running single 'ninja -t deps' call for all built objects...")
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit all object files for processing
future_to_obj = {
executor.submit(self._get_object_dependencies, obj): obj
for obj in object_files
}
# Process completed futures
completed = 0
for future in as_completed(future_to_obj):
obj_file = future_to_obj[future]
try:
dependencies = future.result()
with self.lock:
self.object_to_all_deps[obj_file] = dependencies
completed += 1
if completed % 100 == 0:
print(
f"Processed {completed}/{len(object_files)} object files..."
)
except Exception as e:
print(f"Error processing {obj_file}: {e}")
print(
f"Completed dependency extraction for {len(self.object_to_all_deps)} object files"
)
def _get_object_dependencies(self, object_file):
"""Get all dependencies for a single object file using ninja -t deps."""
try:
# Run ninja -t deps for this object file
cmd = [self.ninja_executable, "-t", "deps", object_file]
cmd = [self.ninja_executable, "-t", "deps"]
result = subprocess.run(
cmd, cwd=self.build_dir, capture_output=True, text=True, timeout=30
cmd, cwd=self.build_dir, capture_output=True, text=True, timeout=120
)
if result.returncode != 0:
return []
if result.returncode != 0 and not result.stdout:
print(f"Warning: ninja -t deps returned code {result.returncode}")
if result.stderr:
print(f" stderr: {result.stderr.strip()}")
return
dependencies = []
lines = result.stdout.strip().split("\n")
# Parse the combined output: each block starts with an object name line,
# followed by indented dependency lines
ws_root = getattr(self, "workspace_root", "..")
ws_prefix = ws_root.rstrip("/") + "/"
for line in lines[1:]: # Skip first line with metadata
line = line.strip()
if line and not line.startswith("#"):
# Convert absolute paths to relative paths from workspace root
dep_file = line
ws_root = getattr(self, "workspace_root", "..")
ws_prefix = ws_root.rstrip("/") + "/"
if dep_file.startswith(ws_prefix):
dep_file = dep_file[len(ws_prefix) :]
dependencies.append(dep_file)
current_obj = None
current_deps = []
matched = 0
return dependencies
for line in result.stdout.split("\n"):
if not line:
continue
if not line.startswith(" ") and not line.startswith("\t"):
# This is an object header line like:
# some/path/foo.cpp.o: #deps 42, deps mtime ... (VALID)
# Save the previous block if it was relevant
if current_obj and current_obj in object_files:
self.object_to_all_deps[current_obj] = current_deps
matched += 1
if matched % 100 == 0:
print(f" Matched {matched} objects so far...")
# Parse the new object name (everything before the colon)
colon_pos = line.find(":")
if colon_pos > 0:
current_obj = line[:colon_pos].strip()
current_deps = []
else:
current_obj = None
current_deps = []
else:
# Indented dependency line
if current_obj is not None:
dep_file = line.strip()
if dep_file and not dep_file.startswith("#"):
# Strip workspace root prefix from absolute paths
if dep_file.startswith(ws_prefix):
dep_file = dep_file[len(ws_prefix):]
current_deps.append(dep_file)
# Don't forget the last block
if current_obj and current_obj in object_files:
self.object_to_all_deps[current_obj] = current_deps
matched += 1
except subprocess.TimeoutExpired:
print("Error: ninja -t deps timed out after 120 seconds")
return
except Exception as e:
print(f"Error getting dependencies for {object_file}: {e}")
return []
print(f"Error running ninja -t deps: {e}")
return
print(
f"Completed dependency extraction for {len(self.object_to_all_deps)} "
f"of {len(object_files)} object files"
)
def _build_file_to_executable_mapping(self):
"""Build the final mapping from files to executables."""
print("Building file-to-executable mapping...")
# For monorepo, truncate the path before and including projects/<project_name>
# This regex matches both absolute and relative monorepo paths
self.project = None
rl_regex = rf"rocm-libraries[\\/]+projects[\\/]+([^\\/]+)[\\/]+(.*)"
rl_regex = rf"(?:^|.*[\\/])projects[\\/]+([^\\/]+)[\\/]+(.*)"
for exe, object_files in self.executable_to_objects.items():
for obj_file in object_files:
# Add all dependencies of this object file
@@ -195,34 +208,51 @@ class EnhancedNinjaDependencyParser:
print(f" {f}: {len(exes)} executables")
def _is_project_file(self, file_path):
"""Determine if a file is part of the project (not system files)."""
# Include files that are clearly part of the project
if any(
file_path.startswith(prefix)
for prefix in [
"projects/composablekernel/include/",
"projects/composablekernel/library/",
"projects/composablekernel/test/",
"projects/composablekernel/example/",
"projects/composablekernel/src/",
"projects/composablekernel/profiler/",
"projects/composablekernel/build/include/",
"projects/composablekernel/build/_deps/gtest",
"projects/composablekernel/client_example",
"projects/composablekernel/codegen",
"projects/composablekernel/tile_engine",
]
):
return True
"""Determine if a file is part of the project (not system files).
# Exclude system files
Handles both standalone-style paths (e.g., include/ck/...) and
monorepo-style paths (e.g., projects/composablekernel/include/ck/...).
"""
# Exclude system files first (absolute paths to system dirs)
if any(
file_path.startswith(prefix)
for prefix in ["/usr/", "/opt/rocm", "/lib/", "/system/", "/local/"]
):
return False
# Include files with common source/header extensions
# Project directory prefixes (without monorepo prefix).
# These match paths relative to the CK project root.
project_dirs = [
"include/",
"library/",
"test/",
"example/",
"src/",
"profiler/",
"build/include/",
"build/_deps/gtest",
"client_example",
"codegen",
"tile_engine",
"dispatcher",
"experimental",
"tutorial",
]
# Check both stripped paths (relative to CK root) and
# monorepo-prefixed paths (relative to monorepo root)
if any(file_path.startswith(prefix) for prefix in project_dirs):
return True
# Also check monorepo-style paths (projects/composablekernel/...)
if any(
file_path.startswith(f"projects/composablekernel/{prefix}")
for prefix in project_dirs
):
return True
# Include files with common source/header extensions that weren't
# excluded as system files above
if file_path.endswith(
(".cpp", ".hpp", ".h", ".c", ".cc", ".cxx", ".cu", ".hip", ".inc")
):
@@ -344,7 +374,7 @@ def main():
sys.exit(1)
parser = EnhancedNinjaDependencyParser(build_file, ninja_path)
parser.workspace_root = workspace_root # Attach for use in _get_object_dependencies
parser.workspace_root = workspace_root # Attach for use in _extract_object_dependencies
parser.parse_dependencies()
parser.print_summary()

17
script/dependency-parser/src/selective_test_filter.py
View File

@@ -34,15 +34,12 @@ import os
def get_changed_files(ref1, ref2, project: str = None):
"""Return a set of files changed between two git refs."""
try:
cmd = ["git", "diff", "--name-only", ref1, ref2]
if project:
# Scope git diff to only this project's subtree for efficiency
cmd += ["--", f"projects/{project}/"]
result = subprocess.run(
["pwd"],
capture_output=True,
text=True,
check=True,
)
print("cwd:", result.stdout)
result = subprocess.run(
["git", "diff", "--name-only", ref1, ref2],
cmd,
capture_output=True,
text=True,
check=True,
@@ -105,7 +102,7 @@ def main():
if not os.path.exists(depmap_json):
print(f"Dependency map JSON not found: {depmap_json}")
sys.exit(1)
file_to_executables = load_depmap(depmap_json)
file_to_executables, _ = load_depmap(depmap_json)
for f, exes in file_to_executables.items():
print(f"{f}: {', '.join(exes)}")
print(f"Total files: {len(file_to_executables)}")
@@ -122,7 +119,7 @@ def main():
if not os.path.exists(depmap_json):
print(f"Dependency map JSON not found: {depmap_json}")
sys.exit(1)
file_to_executables = load_depmap(depmap_json)
file_to_executables, _ = load_depmap(depmap_json)
affected_executables = set()
for f in changed_files:
if f in file_to_executables: