ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-07-12 02:05:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Export ROCm/rocm-libraries@2d4a3223cb

Browse Source
This commit is contained in:
Ding, Yi
2026-03-11 23:03:20 -04:00
commit e6cd3f1e3f
6330 changed files with 1132789 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

207
script/dependency-parser/README.md Normal file
View File

@@ -0,0 +1,207 @@
# Dependency-based Selective Test Filtering using Static Analysis of Ninja Builds for C++ Projects
## Overview
This tool provides advanced dependency-based selective test filtering and build optimization for large C++ monorepos using static parsing of Ninja build files. By analyzing both source and header dependencies, it enables precise identification of which tests and executables are affected by code changes, allowing for efficient CI/CD workflows and faster incremental builds.
The parser:
- Identifies all executables in the Ninja build.
- Maps object files to their source and header dependencies using `ninja -t deps`.
- Constructs a reverse mapping from each file to all dependent executables.
- Automatically detects monorepo structure (`projects/<name>/`) and scopes analysis accordingly.
- Exports results in CSV and JSON formats for integration with other tools.
## Features
- **Comprehensive Dependency Tracking**: Captures direct source file dependencies and, critically, all included header files via `ninja -t deps`.
- **Executable to Object Mapping**: Parses the `build.ninja` file to understand how executables are linked from object files.
- **Batch Dependency Extraction**: Runs a single `ninja -t deps` call (no arguments) to dump all dependency information at once, then filters in-memory. This avoids the massive overhead of per-object subprocess calls on large build files (e.g., a 246MB `build.ninja` with 29K+ objects completes in ~2 seconds instead of ~54 minutes).
- **Monorepo Awareness**: Automatically detects `projects/<project>/` paths, strips them to project-relative paths, and scopes `git diff` to only the relevant subtree.
- **File to Executable Inversion**: Inverts the dependency graph to map each file to the set of executables that depend on it.
- **Filtering**: Filters out system files (`/usr/`, `/opt/rocm/`, etc.) and focuses on project-specific dependencies.
- **Multiple Output Formats**:
- **CSV**: `enhanced_file_executable_mapping.csv` - Each row lists a file and a semicolon-separated list of dependent executables.
- **JSON**: `enhanced_dependency_mapping.json` - Includes file-to-executable mapping, executable-to-file mapping, repo metadata, and statistics.
- **Robust Error Handling**: Includes error handling for missing files and failed subprocess commands.
## Prerequisites
- **Python 3.7+**
- **Ninja build system**: The `ninja` executable must be in the system's PATH or its path provided as an argument.
- A **Ninja build directory** containing a `build.ninja` file. The project should have been built at least once (even partially) so that `ninja -t deps` has dependency data.
## Quick Start with launch_tests.sh
The easiest way to use this tool is via the `launch_tests.sh` wrapper script:
```bash
# From the monorepo root (or anywhere):
script/launch_tests.sh /path/to/build-dir
# Uses default build dir (<CK_ROOT>/build) if no argument given:
script/launch_tests.sh
```
This script:
1. Discovers the git root (monorepo root) automatically.
2. Runs the dependency parser against `build.ninja`.
3. Runs `git diff` between `origin/develop` and the current branch (scoped to CK files only).
4. Maps changed files to affected tests/examples.
5. Runs the affected tests via `ctest` in chunks.
Environment variables:
- `CTEST_CHUNK_SIZE`: Number of tests per ctest invocation (default: 10).
- `CTEST_FAIL_FAST`: Set to `true` to stop on first failure (default: `false`).
## Using CMake with Ninja
To use this tool effectively, your C++ project should be configured with CMake to generate Ninja build files:
1. **Configure CMake to use Ninja:**
```bash
cmake -G Ninja \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \
-DCMAKE_BUILD_TYPE=Release \
-DGPU_TARGETS="gfx942" \
/path/to/composablekernel
```
2. **Build your project (full or partial):**
```bash
# Full build
ninja
# Or build specific targets
ninja example_gemm_xdl_fp16 example_gemm_xdl_fp16_v3
```
The parser only extracts dependencies for objects that were actually built.
3. **Run the dependency parser:**
```bash
python main.py parse /path/to/build/build.ninja --workspace-root /path/to/monorepo-root
```
**Note:** `--workspace-root` should point to the **git root** (monorepo root) for correct monorepo detection. If omitted, it defaults to `..` relative to the build directory.
## Usage
All features are available via the unified `main.py` CLI:
```bash
# Dependency parsing
python main.py parse /path/to/build.ninja --workspace-root /path/to/monorepo-root
# Selective test filtering (between git refs)
python main.py select enhanced_dependency_mapping.json <ref1> <ref2> [--all | --test-prefix] [--output <output_json>]
# Code auditing (list all files and their dependent executables)
python main.py audit enhanced_dependency_mapping.json
# Build optimization (list affected executables for specific changed files)
python main.py optimize enhanced_dependency_mapping.json <changed_file1> [<changed_file2> ...]
```
### Parse arguments
| Argument | Required | Description |
|----------|----------|-------------|
| `build_ninja` | Yes | Path to the `build.ninja` file |
| `--workspace-root` | No | Root of the workspace/monorepo (default: `..`) |
| `--ninja` | No | Path to the ninja executable (default: `ninja`) |
### Select arguments
| Argument | Required | Description |
|----------|----------|-------------|
| `depmap_json` | Yes | Path to `enhanced_dependency_mapping.json` |
| `ref1` | Yes | Source git ref (branch or commit SHA) |
| `ref2` | Yes | Target git ref (branch or commit SHA) |
| `--all` | No | Include all affected executables (default) |
| `--test-prefix` | No | Only include executables starting with `test_` |
| `--output` | No | Output JSON file (default: `tests_to_run.json`) |
## How It Works
1. **Build File Parsing (`_parse_build_file`)**:
* Reads the `build.ninja` file (~246MB for the full CK monorepo build).
* Uses regular expressions to identify executable link rules and object compilation rules.
* Populates `executable_to_objects` and `object_to_source` mappings.
2. **Batch Dependency Extraction (`_extract_object_dependencies`)**:
* Runs a single `ninja -t deps` command (no arguments) which dumps all dependency information for every built object file.
* Parses the output and filters to only the objects found in `object_to_source`.
* Strips the workspace root prefix from absolute paths to produce project-relative paths.
3. **Monorepo Path Detection (`_build_file_to_executable_mapping`)**:
* Applies a regex to detect `projects/<project_name>/` in dependency paths.
* Strips the monorepo prefix so paths are relative to the CK project root (e.g., `include/ck/ck.hpp`).
* Records the detected project name for use by the selective test filter.
4. **File Filtering (`_is_project_file`)**:
* Excludes system files (`/usr/`, `/opt/rocm/`, etc.).
* Includes files in known CK directories (`include/`, `library/`, `test/`, `example/`, etc.).
* Also recognizes monorepo-prefixed paths (`projects/composablekernel/include/`, etc.).
5. **Selective Test Filtering (`selective_test_filter.py`)**:
* Loads the dependency mapping JSON.
* Runs `git diff --name-only` between two refs, scoped to `projects/<project>/` when in monorepo mode.
* Strips the monorepo prefix from changed file paths.
* Looks up each changed file in the dependency map to find affected executables.
* Exports the list of tests to run as JSON.
## Output Files
Running the parser generates two files in the build directory:
- **`enhanced_file_executable_mapping.csv`**:
```csv
source_file,executables
"include/ck/ck.hpp","bin/example_gemm_xdl_fp16;bin/example_gemm_xdl_fp16_v3"
"example/01_gemm/gemm_xdl_fp16.cpp","bin/example_gemm_xdl_fp16"
```
- **`enhanced_dependency_mapping.json`**:
```json
{
"repo": {
"type": "monorepo",
"project": "composablekernel"
},
"file_to_executables": {
"include/ck/ck.hpp": ["bin/example_gemm_xdl_fp16", "bin/example_gemm_xdl_fp16_v3"],
"example/01_gemm/gemm_xdl_fp16.cpp": ["bin/example_gemm_xdl_fp16"]
},
"executable_to_files": {
"bin/example_gemm_xdl_fp16": ["include/ck/ck.hpp", "example/01_gemm/gemm_xdl_fp16.cpp"]
},
"statistics": {
"total_files": 180,
"total_executables": 20403,
"total_object_files": 29530,
"files_with_multiple_executables": 140
}
}
```
## Use Cases
- **Selective CI/CD Testing**: Run only the tests affected by a PR's changes, cutting CI time dramatically.
- **Impact Analysis**: Determine which executables need to be rebuilt when a header changes.
- **Build Optimization**: Identify which targets are affected by a set of file changes.
- **Code Auditing**: Get a clear overview of how files are used across different executables.
## Limitations
- Relies on the accuracy of Ninja's dependency information (`ninja -t deps`). If the build system doesn't correctly generate `.d` (dependency) files, the header information might be incomplete.
- Only objects that have been **actually built** will have dependency data. A partial build means partial coverage of the dependency map.
- The definition of "project file" vs. "system file" is based on a path-based heuristic and might need adjustment for other project structures.
## Troubleshooting
- **"ninja: command not found"**: Ensure `ninja` is installed and in your PATH, or provide the full path via `--ninja`.
- **"build.ninja not found"**: Double-check the path to your `build.ninja` file.
- **Empty or Incomplete Output**:
* Make sure the project has been successfully built at least once. `ninja -t deps` relies on information generated during the build.
* Verify that your CMake is configured to generate dependency files for Ninja (`-G Ninja`).
- **JSON shows `"type": "component"` instead of `"monorepo"`**: Ensure `--workspace-root` points to the **git/monorepo root**, not the CK project root. The parser needs to see `projects/<name>/` in the dependency paths to detect monorepo mode.

91
script/dependency-parser/generate_list_of_files_not_referenced_in_tests.py Normal file
View File

@@ -0,0 +1,91 @@
#!/usr/bin/env python3
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
# SPDX-License-Identifier: MIT
# This script generate list of files that are not referenced from any test (list in JSON format)
# Script only looks at not referenced files from three directories: include, library and profiler
# CK needs to be built with ability to use dependency parser and generate dependencies
# Usage: python3 generate_list_of_files_not_referenced_in_tests -f /path/to/enhanced_dependency_mapping/json/file
import argparse
import os
import subprocess
import json
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"-f",
required=True,
help="Path to enhanced_dependency_mapping.json file generated by dependency parser",
)
args = parser.parse_args()
return args
def main():
args = parse_args()
with open(args.f, "r") as file:
ref_files = json.load(file)
file_to_executables = ref_files["file_to_executables"]
# Determine the CK project root: go up two levels from this script's location
# (script/dependency-parser/ -> script/ -> CK root)
script_dir = os.path.dirname(os.path.abspath(__file__))
ck_root = os.path.normpath(os.path.join(script_dir, "..", ".."))
search_dirs = ["include", "library", "profiler"]
all_files = []
for d in search_dirs:
search_path = os.path.join(ck_root, d)
if not os.path.isdir(search_path):
print(f"Warning: directory not found: {search_path}")
continue
result = subprocess.check_output(
f'find "{search_path}" -type f -iname "*.cpp" -o -iname "*.hpp"',
shell=True,
).decode("utf-8").split("\n")
# Convert absolute paths to relative paths from CK root
ck_root_prefix = ck_root.rstrip("/") + "/"
for f in result:
f = f.strip()
if f:
if f.startswith(ck_root_prefix):
f = f[len(ck_root_prefix):]
all_files.append(f)
all_referenced_files = []
for v in file_to_executables:
# Match both standalone paths (include/, library/, profiler/) and
# monorepo paths (projects/composablekernel/include/, etc.)
if any(
d in v
for d in ["include/", "library/", "profiler/"]
):
exe_list = file_to_executables[v]
else:
continue
found = any("bin/test_" in el for el in exe_list)
if found:
all_referenced_files.append(v)
not_referenced_files = {"include": [], "library": [], "profiler": []}
for f in all_files:
# Check if this file appears in any referenced file path
# (handles both relative and monorepo-prefixed paths)
found = any(f in el or el.endswith(f) for el in all_referenced_files)
if not found:
pos = f.find("/")
if pos > 0 and f[:pos] in not_referenced_files:
not_referenced_files[f[:pos]].append(f)
print(json.dumps(not_referenced_files, indent="\t"))
if __name__ == "__main__":
main()

110
script/dependency-parser/main.py Normal file
View File

@@ -0,0 +1,110 @@
#!/usr/bin/env python3
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
# SPDX-License-Identifier: MIT
"""
Unified CLI for Ninja Dependency Analysis and Selective Testing
Features:
- Dependency parsing (from build.ninja)
- Selective test filtering (between git refs)
- Code auditing (--audit)
- Build optimization (--optimize-build)
"""
import argparse
import sys
def run_dependency_parser(args):
from src.enhanced_ninja_parser import main as ninja_main
sys.argv = ["enhanced_ninja_parser.py"] + args
ninja_main()
def run_selective_test_filter(args):
from src.selective_test_filter import main as filter_main
sys.argv = ["selective_test_filter.py"] + args
filter_main()
def main():
parser = argparse.ArgumentParser(
description="Unified Ninja Dependency & Selective Testing Tool"
)
subparsers = parser.add_subparsers(dest="command", required=True)
# Dependency parsing
parser_parse = subparsers.add_parser(
"parse", help="Parse build.ninja and generate dependency mapping"
)
parser_parse.add_argument("build_ninja", help="Path to build.ninja")
parser_parse.add_argument(
"--ninja", help="Path to ninja executable", default="ninja"
)
parser_parse.add_argument(
"--workspace-root", help="Path to workspace root", default=None
)
# Selective testing
parser_test = subparsers.add_parser(
"select", help="Selective test filtering between git refs"
)
parser_test.add_argument("depmap_json", help="Path to dependency mapping JSON")
parser_test.add_argument("ref1", help="Source git ref")
parser_test.add_argument("ref2", help="Target git ref")
parser_test.add_argument(
"--all", action="store_true", help="Include all executables"
)
parser_test.add_argument(
"--test-prefix",
action="store_true",
help="Only include executables starting with 'test_'",
)
parser_test.add_argument(
"--output", help="Output JSON file", default="tests_to_run.json"
)
# Code auditing
parser_audit = subparsers.add_parser(
"audit", help="List all files and their dependent executables"
)
parser_audit.add_argument("depmap_json", help="Path to dependency mapping JSON")
# Build optimization
parser_opt = subparsers.add_parser(
"optimize", help="List affected executables for changed files"
)
parser_opt.add_argument("depmap_json", help="Path to dependency mapping JSON")
parser_opt.add_argument("changed_files", nargs="+", help="List of changed files")
args = parser.parse_args()
if args.command == "parse":
parse_args = [args.build_ninja, args.ninja]
if args.workspace_root:
parse_args.append(args.workspace_root)
run_dependency_parser(parse_args)
elif args.command == "select":
filter_args = [args.depmap_json, args.ref1, args.ref2]
if args.test_prefix:
filter_args.append("--test-prefix")
if args.all:
filter_args.append("--all")
if args.output:
filter_args += ["--output", args.output]
run_selective_test_filter(filter_args)
elif args.command == "audit":
run_selective_test_filter([args.depmap_json, "--audit"])
elif args.command == "optimize":
run_selective_test_filter(
[args.depmap_json, "--optimize-build"] + args.changed_files
)
else:
parser.print_help()
if __name__ == "__main__":
main()

395
script/dependency-parser/src/enhanced_ninja_parser.py Normal file
View File

@@ -0,0 +1,395 @@
#!/usr/bin/env python3
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
# SPDX-License-Identifier: MIT
"""
Enhanced Ninja Dependency Parser
This script combines ninja build file parsing with ninja -t deps to create a comprehensive
mapping that includes both source files AND header files, and properly handles files
used by multiple executables.
"""
import re
import os
import sys
import subprocess
from collections import defaultdict
import json
class EnhancedNinjaDependencyParser:
def __init__(self, build_file_path, ninja_executable="ninja"):
self.build_file_path = build_file_path
self.build_dir = os.path.dirname(build_file_path)
self.ninja_executable = ninja_executable
# Core data structures
self.executable_to_objects = {} # exe -> [object_files]
self.object_to_source = {} # object -> primary_source
self.object_to_all_deps = {} # object -> [all_dependencies]
self.file_to_executables = defaultdict(set) # file -> {executables}
def parse_dependencies(self):
"""Main method to parse all dependencies."""
print(f"Parsing ninja dependencies from: {self.build_file_path}")
# Step 1: Parse build file for executable -> object mappings
self._parse_build_file()
# Step 2: Get all object files and their dependencies
print(f"Found {len(self.object_to_source)} object files")
print("Extracting detailed dependencies for all object files...")
self._extract_object_dependencies()
# Step 3: Build the final file -> executables mapping
self._build_file_to_executable_mapping()
def _parse_build_file(self):
"""Parse the ninja build file to extract executable -> object mappings."""
print("Parsing ninja build file...")
with open(self.build_file_path, "r") as f:
content = f.read()
# Parse executable build rules
exe_pattern = r"^build (bin/[^:]+):\s+\S+\s+([^|]+)"
obj_pattern = r"^build ([^:]+\.(?:cpp|cu|hip)\.o):\s+\S+\s+([^\s|]+)"
lines = content.split("\n")
for line in lines:
# Match executable rules
exe_match = re.match(exe_pattern, line)
if exe_match and (
"EXECUTABLE" in line
or "test_" in exe_match.group(1)
or "example_" in exe_match.group(1)
):
exe = exe_match.group(1)
deps_part = exe_match.group(2).strip()
object_files = []
for dep in deps_part.split():
if dep.endswith(".o") and not dep.startswith("/"):
object_files.append(dep)
self.executable_to_objects[exe] = object_files
continue
# Match object compilation rules
obj_match = re.match(obj_pattern, line)
if obj_match:
object_file = obj_match.group(1)
source_file = obj_match.group(2)
self.object_to_source[object_file] = source_file
print(f"Found {len(self.executable_to_objects)} executables")
print(f"Found {len(self.object_to_source)} object-to-source mappings")
def _extract_object_dependencies(self):
"""Extract detailed dependencies for all object files using a single ninja -t deps call.
Previous implementation spawned ninja -t deps per object file (29K+ subprocesses),
each re-parsing the full build.ninja. For large monorepo builds (246MB+ build.ninja),
each call takes 2-14 seconds, making the total ~54 minutes.
This implementation calls ninja -t deps once (no args) to dump ALL deps in ~2 seconds,
then parses the output and filters to only the objects we care about.
"""
object_files = set(self.object_to_source.keys())
if not object_files:
print("No object files found - skipping dependency extraction")
return
print(f"Running single 'ninja -t deps' call for all built objects...")
try:
cmd = [self.ninja_executable, "-t", "deps"]
result = subprocess.run(
cmd, cwd=self.build_dir, capture_output=True, text=True, timeout=120
)
if result.returncode != 0 and not result.stdout:
print(f"Warning: ninja -t deps returned code {result.returncode}")
if result.stderr:
print(f" stderr: {result.stderr.strip()}")
return
# Parse the combined output: each block starts with an object name line,
# followed by indented dependency lines
ws_root = getattr(self, "workspace_root", "..")
ws_prefix = ws_root.rstrip("/") + "/"
current_obj = None
current_deps = []
matched = 0
for line in result.stdout.split("\n"):
if not line:
continue
if not line.startswith(" ") and not line.startswith("\t"):
# This is an object header line like:
# some/path/foo.cpp.o: #deps 42, deps mtime ... (VALID)
# Save the previous block if it was relevant
if current_obj and current_obj in object_files:
self.object_to_all_deps[current_obj] = current_deps
matched += 1
if matched % 100 == 0:
print(f" Matched {matched} objects so far...")
# Parse the new object name (everything before the colon)
colon_pos = line.find(":")
if colon_pos > 0:
current_obj = line[:colon_pos].strip()
current_deps = []
else:
current_obj = None
current_deps = []
else:
# Indented dependency line
if current_obj is not None:
dep_file = line.strip()
if dep_file and not dep_file.startswith("#"):
# Strip workspace root prefix from absolute paths
if dep_file.startswith(ws_prefix):
dep_file = dep_file[len(ws_prefix):]
current_deps.append(dep_file)
# Don't forget the last block
if current_obj and current_obj in object_files:
self.object_to_all_deps[current_obj] = current_deps
matched += 1
except subprocess.TimeoutExpired:
print("Error: ninja -t deps timed out after 120 seconds")
return
except Exception as e:
print(f"Error running ninja -t deps: {e}")
return
print(
f"Completed dependency extraction for {len(self.object_to_all_deps)} "
f"of {len(object_files)} object files"
)
def _build_file_to_executable_mapping(self):
"""Build the final mapping from files to executables."""
print("Building file-to-executable mapping...")
# For monorepo, truncate the path before and including projects/<project_name>
# This regex matches both absolute and relative monorepo paths
self.project = None
rl_regex = rf"(?:^|.*[\\/])projects[\\/]+([^\\/]+)[\\/]+(.*)"
for exe, object_files in self.executable_to_objects.items():
for obj_file in object_files:
# Add all dependencies of this object file
if obj_file in self.object_to_all_deps:
for dep_file in self.object_to_all_deps[obj_file]:
match = re.search(rl_regex, dep_file, re.IGNORECASE)
if match:
dep_file = match.group(2)
if not self.project:
print(f"Found rocm-libraries project: '{match.group(1)}'")
self.project = match.group(1)
# Filter out system files and focus on project files
if self._is_project_file(dep_file):
self.file_to_executables[dep_file].add(exe)
print(f"Built mapping for {len(self.file_to_executables)} files")
# Show statistics
multi_exe_files = {
f: exes for f, exes in self.file_to_executables.items() if len(exes) > 1
}
print(f"Files used by multiple executables: {len(multi_exe_files)}")
if multi_exe_files:
print("Sample files with multiple dependencies:")
for f, exes in sorted(multi_exe_files.items())[:5]:
print(f" {f}: {len(exes)} executables")
def _is_project_file(self, file_path):
"""Determine if a file is part of the project (not system files).
Handles both standalone-style paths (e.g., include/ck/...) and
monorepo-style paths (e.g., projects/composablekernel/include/ck/...).
"""
# Exclude system files first (absolute paths to system dirs)
if any(
file_path.startswith(prefix)
for prefix in ["/usr/", "/opt/rocm", "/lib/", "/system/", "/local/"]
):
return False
# Project directory prefixes (without monorepo prefix).
# These match paths relative to the CK project root.
project_dirs = [
"include/",
"library/",
"test/",
"example/",
"src/",
"profiler/",
"build/include/",
"build/_deps/gtest",
"client_example",
"codegen",
"tile_engine",
"dispatcher",
"experimental",
"tutorial",
]
# Check both stripped paths (relative to CK root) and
# monorepo-prefixed paths (relative to monorepo root)
if any(file_path.startswith(prefix) for prefix in project_dirs):
return True
# Also check monorepo-style paths (projects/composablekernel/...)
if any(
file_path.startswith(f"projects/composablekernel/{prefix}")
for prefix in project_dirs
):
return True
# Include files with common source/header extensions that weren't
# excluded as system files above
if file_path.endswith(
(".cpp", ".hpp", ".h", ".c", ".cc", ".cxx", ".cu", ".hip", ".inc")
):
return True
return False
def export_to_csv(self, output_file):
"""Export the file-to-executable mapping to CSV with proper comma separation."""
print(f"Exporting mapping to {output_file}")
with open(output_file, "w") as f:
f.write("source_file,executables\n")
for file_path in sorted(self.file_to_executables.keys()):
executables = sorted(self.file_to_executables[file_path])
# Use semicolon to separate multiple executables within the field
exe_list = ";".join(executables)
f.write(f'"{file_path}","{exe_list}"\n')
def export_to_json(self, output_file):
"""Export the complete mapping to JSON."""
print(f"Exporting complete mapping to {output_file}")
# Build reverse mapping (executable -> files)
exe_to_files = defaultdict(set)
for file_path, exes in self.file_to_executables.items():
for exe in exes:
exe_to_files[exe].add(file_path)
mapping_data = {
"repo": {
"type": "monorepo" if self.project else "component",
"project": self.project
},
"file_to_executables": {
file_path: list(exes)
for file_path, exes in self.file_to_executables.items()
},
"executable_to_files": {
exe: sorted(files) for exe, files in exe_to_files.items()
},
"statistics": {
"total_files": len(self.file_to_executables),
"total_executables": len(self.executable_to_objects),
"total_object_files": len(self.object_to_source),
"files_with_multiple_executables": len(
[f for f, exes in self.file_to_executables.items() if len(exes) > 1]
),
},
}
with open(output_file, "w") as f:
json.dump(mapping_data, f, indent=2)
def print_summary(self):
"""Print a summary of the parsed dependencies."""
print("\n=== Enhanced Dependency Mapping Summary ===")
print(f"Total executables: {len(self.executable_to_objects)}")
print(f"Total files mapped: {len(self.file_to_executables)}")
print(f"Total object files processed: {len(self.object_to_all_deps)}")
# Files by type
cpp_files = sum(
1 for f in self.file_to_executables.keys() if f.endswith(".cpp")
)
hpp_files = sum(
1 for f in self.file_to_executables.keys() if f.endswith(".hpp")
)
h_files = sum(1 for f in self.file_to_executables.keys() if f.endswith(".h"))
print("\nFile types:")
print(f" .cpp files: {cpp_files}")
print(f" .hpp files: {hpp_files}")
print(f" .h files: {h_files}")
# Multi-executable files
multi_exe_files = {
f: exes for f, exes in self.file_to_executables.items() if len(exes) > 1
}
print(f"\nFiles used by multiple executables: {len(multi_exe_files)}")
if multi_exe_files:
print("\nTop files with most dependencies:")
sorted_multi = sorted(
multi_exe_files.items(), key=lambda x: len(x[1]), reverse=True
)
for file_path, exes in sorted_multi[:10]:
print(f" {file_path}: {len(exes)} executables")
def main():
# Accept: build_file, ninja_path, workspace_root
default_workspace_root = ".."
if len(sys.argv) > 3:
build_file = sys.argv[1]
ninja_path = sys.argv[2]
workspace_root = sys.argv[3]
elif len(sys.argv) > 2:
build_file = sys.argv[1]
ninja_path = sys.argv[2]
workspace_root = default_workspace_root
elif len(sys.argv) > 1:
build_file = sys.argv[1]
ninja_path = "ninja"
workspace_root = default_workspace_root
else:
build_file = f"{default_workspace_root}/build/build.ninja"
ninja_path = "ninja"
workspace_root = default_workspace_root
if not os.path.exists(build_file):
print(f"Error: Build file not found: {build_file}")
sys.exit(1)
try:
subprocess.run([ninja_path, "--version"], capture_output=True, check=True)
except (subprocess.CalledProcessError, FileNotFoundError):
print(f"Error: ninja executable not found: {ninja_path}")
sys.exit(1)
parser = EnhancedNinjaDependencyParser(build_file, ninja_path)
parser.workspace_root = workspace_root # Attach for use in _extract_object_dependencies
parser.parse_dependencies()
parser.print_summary()
# Export results
output_dir = os.path.dirname(build_file)
csv_file = os.path.join(output_dir, "enhanced_file_executable_mapping.csv")
json_file = os.path.join(output_dir, "enhanced_dependency_mapping.json")
parser.export_to_csv(csv_file)
parser.export_to_json(json_file)
print("\nResults exported to:")
print(f" CSV: {csv_file}")
print(f" JSON: {json_file}")
if __name__ == "__main__":
main()

175
script/dependency-parser/src/selective_test_filter.py Normal file
View File

@@ -0,0 +1,175 @@
#!/usr/bin/env python3
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
# SPDX-License-Identifier: MIT
"""
Selective Test Filter Tool
Given two git refs (branches or commit IDs), this tool:
- Identifies changed files between the refs
- Loads the enhanced dependency mapping JSON (from enhanced_ninja_parser.py)
- Maps changed files to affected test executables (optionally filtering for "test_" prefix)
- Exports the list of tests to run to tests_to_run.json
Usage:
python selective_test_filter.py <depmap_json> <ref1> <ref2> [--all | --test-prefix] [--output <output_json>]
Arguments:
<depmap_json> Path to enhanced_dependency_mapping.json
<ref1> Source git ref (branch or commit)
<ref2> Target git ref (branch or commit)
Options:
--all Include all executables (default)
--test-prefix Only include executables starting with "test_"
--output Output JSON file (default: tests_to_run.json)
"""
import sys
import subprocess
import json
import os
def get_changed_files(ref1, ref2, project: str = None):
"""Return a set of files changed between two git refs."""
try:
cmd = ["git", "diff", "--name-only", ref1, ref2]
if project:
# Scope git diff to only this project's subtree for efficiency
cmd += ["--", f"projects/{project}/"]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True,
)
raw_files = set(line.strip() for line in result.stdout.splitlines() if line.strip())
if project is None:
files = raw_files
print(f"Identified {len(files)} modified files")
else:
root = f"projects/{project}/"
root_len = len(root)
files = set()
for f in raw_files:
if f.startswith(root):
files.add(f[root_len:])
print(f"Identified {len(files)} files modified in project '{project}'")
return files
except subprocess.CalledProcessError as e:
print(f"Command '{e.cmd}' returned non-zero exit status {e.returncode}.")
print(f"Error output: {e.stderr}")
print(f"Standard output: {e.stdout}")
sys.exit(1)
def load_depmap(depmap_json):
"""Load the dependency mapping JSON."""
with open(depmap_json, "r") as f:
data = json.load(f)
# Support both old and new formats
json_project = None
if "repo" in data and data["repo"]["type"] == "monorepo":
json_project = data["repo"]["project"]
if "file_to_executables" in data:
return data["file_to_executables"], json_project
return data, json_project
def select_tests(file_to_executables, changed_files, filter_mode):
"""Return a set of test executables affected by changed files."""
affected = set()
for f in changed_files:
if f in file_to_executables:
for exe in file_to_executables[f]:
if filter_mode == "all":
affected.add(exe)
elif filter_mode == "test_prefix" and exe.startswith("test_"):
affected.add(exe)
return sorted(affected)
def main():
if "--audit" in sys.argv:
if len(sys.argv) < 2:
print("Usage: python selective_test_filter.py <depmap_json> --audit")
sys.exit(1)
depmap_json = sys.argv[1]
if not os.path.exists(depmap_json):
print(f"Dependency map JSON not found: {depmap_json}")
sys.exit(1)
file_to_executables, _ = load_depmap(depmap_json)
for f, exes in file_to_executables.items():
print(f"{f}: {', '.join(exes)}")
print(f"Total files: {len(file_to_executables)}")
sys.exit(0)
if "--optimize-build" in sys.argv:
if len(sys.argv) < 3:
print(
"Usage: python selective_test_filter.py <depmap_json> --optimize-build <changed_file1> [<changed_file2> ...]"
)
sys.exit(1)
depmap_json = sys.argv[1]
changed_files = set(sys.argv[sys.argv.index("--optimize-build") + 1 :])
if not os.path.exists(depmap_json):
print(f"Dependency map JSON not found: {depmap_json}")
sys.exit(1)
file_to_executables, _ = load_depmap(depmap_json)
affected_executables = set()
for f in changed_files:
if f in file_to_executables:
affected_executables.update(file_to_executables[f])
print("Affected executables:")
for exe in sorted(affected_executables):
print(exe)
print(f"Total affected executables: {len(affected_executables)}")
sys.exit(0)
if len(sys.argv) < 4:
print(
"Usage: python selective_test_filter.py <depmap_json> <ref1> <ref2> [--all | --test-prefix] [--output <output_json>]"
)
sys.exit(1)
depmap_json = sys.argv[1]
ref1 = sys.argv[2]
ref2 = sys.argv[3]
filter_mode = "all"
output_json = "tests_to_run.json"
if "--test-prefix" in sys.argv:
filter_mode = "test_prefix"
if "--all" in sys.argv:
filter_mode = "all"
if "--output" in sys.argv:
idx = sys.argv.index("--output")
if idx + 1 < len(sys.argv):
output_json = sys.argv[idx + 1]
if not os.path.exists(depmap_json):
print(f"Dependency map JSON not found: {depmap_json}")
sys.exit(1)
file_to_executables, json_project = load_depmap(depmap_json)
changed_files = get_changed_files(ref1, ref2, json_project)
if not changed_files:
print("No changed files detected.")
tests = []
else:
tests = select_tests(file_to_executables, changed_files, filter_mode)
with open(output_json, "w") as f:
json.dump(
{"tests_to_run": tests, "changed_files": sorted(changed_files)}, f, indent=2
)
print(f"Exported {len(tests)} tests to run to {output_json}")
if __name__ == "__main__":
main()