[rocm-libraries] ROCm/rocm-libraries#5249 (commit 2a114bb)

[CK] [CK_TILE] Improve build and test time of CI with smart
 dependency parser (#5249)

## Motivation

Existing dependency parser needs full build of tests to determine which
tests are affected by code changes in a PR. This still takes 2-4 hours
for building the tests which slows down the CI as the number of tests
grow. To resolve this issue we implemented a smart dependency parser
which uses CMake Configure to parse dependencies and build only the
affected test cases. We have ensured that two approaches are available
1) CMake pre-build analysis for each PR to ensure fast build and test.
2) Ninja post-build analysis to enable full build for nightly tests.

## Technical Details

```bash
### 1. Configure the project with CMake
cmake -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..

### 2. Analyze dependencies (no build required!)
python3 ../script/dependency-parser/main.py cmake-parse compile_commands.json build.ninja \
  --workspace-root .. --output cmake_dependency_mapping.json --parallel 8

### 3. Find tests affected by changes
python3 ../script/dependency-parser/main.py select cmake_dependency_mapping.json origin/develop \
  HEAD --test-prefix --output tests_to_run.json

### 4. Build only affected tests
ninja $(jq -r '.executables[]' tests_to_run.json | tr '\n' ' ')

### 5. Run affected tests
ctest -R "$(jq -r '.regex' tests_to_run.json)"
```

### Jenkins Integration
- Added `buildMode` to jenkinsfile to integrate both `selective` and
`full` build methods

### Known Limitations

### 1. Build-Time Generated Headers (HIGH RISK)

**Problem:** Files generated during the build process (e.g., via
`add_custom_command`) cannot be analyzed before building.

**Example:**
```cmake
add_custom_command(
  OUTPUT ${CMAKE_BINARY_DIR}/generated/config.hpp
  COMMAND generate_config.sh
  DEPENDS template.hpp.in
)
```

**Impact:** If a source file includes `generated/config.hpp`, the
dependency won't be detected until after building.

**Mitigation:**
- CK analysis shows **no generated headers** currently used
- If generated headers are added in the future, they must be built first
- Recommendation: Generate headers in CMake configure phase (not build
phase) when possible

## Test Plan
**1. Modified Files:**
```
include/ck_tile/ops/common.hpp
include/ck_tile/ops/gemm.hpp
include/ck_tile/ops/gemm/warp/warp_gemm.hpp
```
**2. Compare tests selected between `build.ninja` and `cmake-parse`
methods**

## Test Result
- 1. The test completed in 5-6 minutes finding about 8000+ executables
that should be built.
- 2. We selected a commit 5ccc1387ea which resulted in same 7 tests with
both legacy and new methods.
-

PR | Legacy tests | Smart tests | Notes
-- | -- | -- | --
5261 | 453 | 455 | Only 2 tests (test_amdgcn_mma and
test_amdgcn_sparse_mma)
5168 | 0 | 0 | Changes in   dispatcher only. No CK tests invoked.
5249 | 0 | 0 | Changes to   dependency parser. No CK tests invoked
5260 | 0 | 0 | Changes in   dispatcher only. No CK tests invoked.
5174 | 1 | 1 | One test from FMHA   affected by this PR in both cases
5383 | 0 | 0 | Changes are only in benchmark files. Did not trigger any
tests
5445 | 1 | 1 | Changes are only to tests/ck_tile/gemm_streamk. Only
triggered one streamk test in both cases.
5454 | 3 | 3 | Both methods identified same test_grouped_conv_bwd tests
5427 | 234 | 234 | Core infrastructure header changes. Detected exactly
same tests
5388 | 85 | 85 | modifies warp-level GEMM operations (warp_gemm.hpp,
warp_gemm_dispatcher.hpp). Correctly identified all the streamK gemm
tests

## Submission Checklist

- [x ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

script/dependency-parser/tests/test_cmake_dependency_analyzer.py

@@ -0,0 +1,512 @@
+#!/usr/bin/env python3
+# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
+# SPDX-License-Identifier: MIT
+
+"""
+Test-Driven Development tests for CMake Dependency Analyzer.
+
+This module tests the new pre-build dependency analysis approach that uses
+compile_commands.json and clang -MM instead of requiring a full ninja build.
+"""
+
+import json
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import Mock, patch, MagicMock
+import shutil
+import sys
+
+# Add parent directory to path for imports
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+
+class TestCompileCommandsParser(unittest.TestCase):
+    """Tests for parsing compile_commands.json."""
+
+    def setUp(self):
+        """Create temporary directory and sample compile_commands.json."""
+        self.temp_dir = tempfile.mkdtemp()
+        self.compile_commands_path = os.path.join(self.temp_dir, "compile_commands.json")
+
+    def tearDown(self):
+        """Clean up temporary directory."""
+        shutil.rmtree(self.temp_dir)
+
+    def test_parse_empty_compile_commands(self):
+        """Parser should handle empty compile_commands.json gracefully."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        with open(self.compile_commands_path, "w") as f:
+            json.dump([], f)
+
+        parser = CompileCommandsParser(self.compile_commands_path)
+        commands = parser.parse()
+
+        self.assertEqual(len(commands), 0)
+
+    def test_parse_single_command(self):
+        """Parser should correctly parse a single compile command."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        sample_commands = [
+            {
+                "directory": "/build",
+                "command": "/opt/rocm/bin/amdclang++ -DFOO=1 -I/include -c test.cpp -o test.o",
+                "file": "/src/test.cpp",
+            }
+        ]
+        with open(self.compile_commands_path, "w") as f:
+            json.dump(sample_commands, f)
+
+        parser = CompileCommandsParser(self.compile_commands_path)
+        commands = parser.parse()
+
+        self.assertEqual(len(commands), 1)
+        self.assertEqual(commands[0]["file"], "/src/test.cpp")
+        self.assertEqual(commands[0]["directory"], "/build")
+
+    def test_parse_multiple_commands(self):
+        """Parser should correctly parse multiple compile commands."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        sample_commands = [
+            {
+                "directory": "/build",
+                "command": "/opt/rocm/bin/amdclang++ -c test1.cpp -o test1.o",
+                "file": "/src/test1.cpp",
+            },
+            {
+                "directory": "/build",
+                "command": "/opt/rocm/bin/amdclang++ -c test2.cpp -o test2.o",
+                "file": "/src/test2.cpp",
+            },
+        ]
+        with open(self.compile_commands_path, "w") as f:
+            json.dump(sample_commands, f)
+
+        parser = CompileCommandsParser(self.compile_commands_path)
+        commands = parser.parse()
+
+        self.assertEqual(len(commands), 2)
+
+    def test_filter_by_extension(self):
+        """Parser should filter commands by file extension."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        sample_commands = [
+            {"directory": "/build", "command": "clang++ -c test.cpp -o test.o", "file": "/src/test.cpp"},
+            {"directory": "/build", "command": "clang++ -c test.cc -o test.o", "file": "/src/test.cc"},
+            {"directory": "/build", "command": "clang -c test.c -o test.o", "file": "/src/test.c"},
+        ]
+        with open(self.compile_commands_path, "w") as f:
+            json.dump(sample_commands, f)
+
+        parser = CompileCommandsParser(self.compile_commands_path)
+        commands = parser.parse(extensions=[".cpp"])
+
+        self.assertEqual(len(commands), 1)
+        self.assertEqual(commands[0]["file"], "/src/test.cpp")
+
+    def test_handles_arguments_format(self):
+        """Parser should handle both 'command' and 'arguments' formats."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        sample_commands = [
+            {
+                "directory": "/build",
+                "arguments": ["/opt/rocm/bin/amdclang++", "-c", "test.cpp", "-o", "test.o"],
+                "file": "/src/test.cpp",
+            }
+        ]
+        with open(self.compile_commands_path, "w") as f:
+            json.dump(sample_commands, f)
+
+        parser = CompileCommandsParser(self.compile_commands_path)
+        commands = parser.parse()
+
+        self.assertEqual(len(commands), 1)
+        self.assertIn("command", commands[0])
+
+
+class TestDependencyExtractor(unittest.TestCase):
+    """Tests for extracting dependencies using clang -MM."""
+
+    def setUp(self):
+        """Set up test fixtures."""
+        self.temp_dir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        """Clean up."""
+        shutil.rmtree(self.temp_dir)
+
+    def test_convert_compile_to_dependency_command(self):
+        """Should convert compile command to dependency extraction command."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        compile_cmd = "/opt/rocm/bin/amdclang++ -DFOO=1 -I/include -O3 -c /src/test.cpp -o /build/test.o"
+
+        extractor = DependencyExtractor()
+        dep_cmd = extractor.convert_to_dependency_command(compile_cmd, "/tmp/deps.d")
+
+        # Should have -MM flag
+        self.assertIn("-MM", dep_cmd)
+        # Should have -MF flag with output file
+        self.assertIn("-MF", dep_cmd)
+        self.assertIn("/tmp/deps.d", dep_cmd)
+        # Should NOT have -c flag
+        self.assertNotIn(" -c ", dep_cmd)
+        # Should NOT have -o flag with output
+        self.assertNotIn(" -o ", dep_cmd)
+        # Should preserve includes and defines
+        self.assertIn("-DFOO=1", dep_cmd)
+        self.assertIn("-I/include", dep_cmd)
+        # Should preserve source file
+        self.assertIn("/src/test.cpp", dep_cmd)
+
+    def test_parse_makefile_deps_simple(self):
+        """Should parse simple makefile-style dependency output."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        deps_content = "test.o: test.cpp header1.hpp header2.hpp\n"
+
+        extractor = DependencyExtractor()
+        deps = extractor.parse_makefile_deps(deps_content)
+
+        self.assertEqual(len(deps), 3)
+        self.assertIn("test.cpp", deps)
+        self.assertIn("header1.hpp", deps)
+        self.assertIn("header2.hpp", deps)
+
+    def test_parse_makefile_deps_multiline(self):
+        """Should parse multiline makefile-style dependency output."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        deps_content = """test.o: test.cpp \\
+  /include/header1.hpp \\
+  /include/header2.hpp \\
+  /include/header3.hpp
+"""
+
+        extractor = DependencyExtractor()
+        deps = extractor.parse_makefile_deps(deps_content)
+
+        self.assertEqual(len(deps), 4)
+        self.assertIn("test.cpp", deps)
+        self.assertIn("/include/header1.hpp", deps)
+        self.assertIn("/include/header2.hpp", deps)
+        self.assertIn("/include/header3.hpp", deps)
+
+    def test_parse_makefile_deps_empty(self):
+        """Should handle empty dependency output."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        extractor = DependencyExtractor()
+        deps = extractor.parse_makefile_deps("")
+
+        self.assertEqual(len(deps), 0)
+
+    @patch("subprocess.run")
+    def test_extract_dependencies_success(self, mock_run):
+        """Should successfully extract dependencies using clang -MM."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        # Mock successful clang -MM execution
+        mock_run.return_value = Mock(returncode=0, stdout="", stderr="")
+
+        extractor = DependencyExtractor()
+
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".d", delete=False) as f:
+            f.write("test.o: test.cpp header.hpp\n")
+            deps_file = f.name
+
+        with patch.object(extractor, "_get_deps_file", return_value=deps_file):
+            deps = extractor.extract("/build", "clang++ -c test.cpp -o test.o", "/src/test.cpp")
+
+        self.assertIn("test.cpp", deps)
+        self.assertIn("header.hpp", deps)
+        # Note: The implementation cleans up the temp file, so we don't need to
+
+    @patch("subprocess.run")
+    def test_extract_dependencies_compiler_error(self, mock_run):
+        """Should handle compiler errors gracefully."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        # Mock failed clang -MM execution
+        mock_run.return_value = Mock(returncode=1, stdout="", stderr="error: file not found")
+
+        extractor = DependencyExtractor()
+        deps = extractor.extract("/build", "clang++ -c test.cpp -o test.o", "/src/test.cpp")
+
+        # Should return empty list on error, not crash
+        self.assertEqual(deps, [])
+
+
+class TestNinjaTargetParser(unittest.TestCase):
+    """Tests for parsing ninja build files to get target mappings."""
+
+    def setUp(self):
+        """Set up test fixtures."""
+        self.temp_dir = tempfile.mkdtemp()
+        self.ninja_file = os.path.join(self.temp_dir, "build.ninja")
+
+    def tearDown(self):
+        """Clean up."""
+        shutil.rmtree(self.temp_dir)
+
+    def test_parse_executable_to_objects(self):
+        """Should parse executable -> object file mappings from build.ninja."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        ninja_content = """
+rule CXX_EXECUTABLE_LINKER__test_gemm
+  command = /opt/rocm/bin/amdclang++ $in -o $out
+
+build bin/test_gemm: CXX_EXECUTABLE_LINKER__test_gemm test/test_gemm.cpp.o library/gemm.cpp.o | lib.so
+"""
+        with open(self.ninja_file, "w") as f:
+            f.write(ninja_content)
+
+        parser = NinjaTargetParser(self.ninja_file)
+        exe_to_objects = parser.parse_executable_mappings()
+
+        self.assertIn("bin/test_gemm", exe_to_objects)
+        self.assertIn("test/test_gemm.cpp.o", exe_to_objects["bin/test_gemm"])
+        self.assertIn("library/gemm.cpp.o", exe_to_objects["bin/test_gemm"])
+
+    def test_parse_object_to_source(self):
+        """Should parse object -> source file mappings from build.ninja."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        ninja_content = """
+rule CXX_COMPILER__test_gemm
+  command = /opt/rocm/bin/amdclang++ -c $in -o $out
+
+build test/test_gemm.cpp.o: CXX_COMPILER__test_gemm /src/test/test_gemm.cpp
+build library/gemm.cpp.o: CXX_COMPILER__test_gemm /src/library/gemm.cpp
+"""
+        with open(self.ninja_file, "w") as f:
+            f.write(ninja_content)
+
+        parser = NinjaTargetParser(self.ninja_file)
+        obj_to_source = parser.parse_object_to_source()
+
+        self.assertIn("test/test_gemm.cpp.o", obj_to_source)
+        self.assertEqual(obj_to_source["test/test_gemm.cpp.o"], "/src/test/test_gemm.cpp")
+
+    def test_filter_test_executables(self):
+        """Should correctly filter test executables by prefix."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        ninja_content = """
+build bin/test_gemm: CXX_EXECUTABLE_LINKER__test_gemm test.o
+build bin/example_gemm: CXX_EXECUTABLE_LINKER__example_gemm example.o
+build bin/ckProfiler: CXX_EXECUTABLE_LINKER__ckProfiler profiler.o
+"""
+        with open(self.ninja_file, "w") as f:
+            f.write(ninja_content)
+
+        parser = NinjaTargetParser(self.ninja_file)
+        exe_to_objects = parser.parse_executable_mappings()
+
+        test_exes = [exe for exe in exe_to_objects if "test_" in exe]
+        self.assertEqual(len(test_exes), 1)
+        self.assertIn("bin/test_gemm", test_exes)
+
+
+class TestDependencyMapper(unittest.TestCase):
+    """Tests for building the file -> executable dependency mapping."""
+
+    def test_build_file_to_executable_mapping(self):
+        """Should build correct file -> executable mapping."""
+        from cmake_dependency_analyzer import DependencyMapper
+
+        # Simulated data
+        exe_to_objects = {
+            "bin/test_gemm": ["test/test_gemm.cpp.o", "lib/gemm.cpp.o"],
+            "bin/test_conv": ["test/test_conv.cpp.o", "lib/conv.cpp.o"],
+        }
+        obj_to_source = {
+            "test/test_gemm.cpp.o": "test/test_gemm.cpp",
+            "lib/gemm.cpp.o": "lib/gemm.cpp",
+            "test/test_conv.cpp.o": "test/test_conv.cpp",
+            "lib/conv.cpp.o": "lib/conv.cpp",
+        }
+        source_to_deps = {
+            "test/test_gemm.cpp": ["test/test_gemm.cpp", "include/gemm.hpp", "include/common.hpp"],
+            "lib/gemm.cpp": ["lib/gemm.cpp", "include/gemm.hpp"],
+            "test/test_conv.cpp": ["test/test_conv.cpp", "include/conv.hpp", "include/common.hpp"],
+            "lib/conv.cpp": ["lib/conv.cpp", "include/conv.hpp"],
+        }
+
+        mapper = DependencyMapper()
+        file_to_exes = mapper.build_mapping(exe_to_objects, obj_to_source, source_to_deps)
+
+        # common.hpp should map to both test executables
+        self.assertIn("include/common.hpp", file_to_exes)
+        self.assertIn("bin/test_gemm", file_to_exes["include/common.hpp"])
+        self.assertIn("bin/test_conv", file_to_exes["include/common.hpp"])
+
+        # gemm.hpp should only map to test_gemm
+        self.assertIn("include/gemm.hpp", file_to_exes)
+        self.assertIn("bin/test_gemm", file_to_exes["include/gemm.hpp"])
+        self.assertNotIn("bin/test_conv", file_to_exes["include/gemm.hpp"])
+
+    def test_normalize_paths(self):
+        """Should normalize paths relative to workspace root."""
+        from cmake_dependency_analyzer import DependencyMapper
+
+        mapper = DependencyMapper(workspace_root="/workspace/rocm-libraries/projects/composablekernel")
+
+        # Test monorepo-style path
+        normalized = mapper.normalize_path(
+            "/workspace/rocm-libraries/projects/composablekernel/include/ck/ck.hpp"
+        )
+        self.assertEqual(normalized, "include/ck/ck.hpp")
+
+        # Test already relative path
+        normalized = mapper.normalize_path("include/ck/ck.hpp")
+        self.assertEqual(normalized, "include/ck/ck.hpp")
+
+    def test_filter_system_files(self):
+        """Should filter out system files."""
+        from cmake_dependency_analyzer import DependencyMapper
+
+        mapper = DependencyMapper()
+
+        self.assertFalse(mapper.is_project_file("/usr/include/stdio.h"))
+        self.assertFalse(mapper.is_project_file("/opt/rocm/include/hip/hip_runtime.h"))
+        self.assertTrue(mapper.is_project_file("include/ck/ck.hpp"))
+        self.assertTrue(mapper.is_project_file("test/test_gemm.cpp"))
+
+
+class TestCMakeDependencyAnalyzer(unittest.TestCase):
+    """Integration tests for the full CMake dependency analyzer."""
+
+    def setUp(self):
+        """Set up test fixtures."""
+        self.temp_dir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        """Clean up."""
+        shutil.rmtree(self.temp_dir)
+
+    def test_output_format_compatibility(self):
+        """Output JSON should be compatible with selective_test_filter.py."""
+        from cmake_dependency_analyzer import CMakeDependencyAnalyzer
+
+        # Create minimal test data
+        analyzer = CMakeDependencyAnalyzer(
+            compile_commands_path=None,
+            ninja_path=None,
+            workspace_root=self.temp_dir,
+        )
+
+        # Manually set internal state for testing output format
+        analyzer.file_to_executables = {
+            "include/ck/ck.hpp": {"bin/test_gemm", "bin/test_conv"},
+            "test/test_gemm.cpp": {"bin/test_gemm"},
+        }
+        analyzer.executable_to_files = {
+            "bin/test_gemm": {"include/ck/ck.hpp", "test/test_gemm.cpp"},
+            "bin/test_conv": {"include/ck/ck.hpp"},
+        }
+
+        output_file = os.path.join(self.temp_dir, "output.json")
+        analyzer.export_to_json(output_file)
+
+        with open(output_file) as f:
+            data = json.load(f)
+
+        # Check required fields for selective_test_filter.py compatibility
+        self.assertIn("file_to_executables", data)
+        self.assertIn("executable_to_files", data)
+        self.assertIn("statistics", data)
+
+        # Check file_to_executables format (should be lists, not sets)
+        self.assertIsInstance(data["file_to_executables"]["include/ck/ck.hpp"], list)
+
+    def test_statistics_calculation(self):
+        """Should calculate correct statistics."""
+        from cmake_dependency_analyzer import CMakeDependencyAnalyzer
+
+        analyzer = CMakeDependencyAnalyzer(
+            compile_commands_path=None,
+            ninja_path=None,
+            workspace_root=self.temp_dir,
+        )
+
+        analyzer.file_to_executables = {
+            "include/common.hpp": {"bin/test1", "bin/test2", "bin/test3"},
+            "include/specific.hpp": {"bin/test1"},
+            "test/test1.cpp": {"bin/test1"},
+        }
+
+        stats = analyzer.calculate_statistics()
+
+        self.assertEqual(stats["total_files"], 3)
+        self.assertEqual(stats["files_with_multiple_executables"], 1)
+
+
+class TestParallelDependencyExtraction(unittest.TestCase):
+    """Tests for parallel dependency extraction."""
+
+    def test_batch_extraction_preserves_results(self):
+        """Parallel extraction should produce same results as serial."""
+        from cmake_dependency_analyzer import DependencyExtractor
+
+        extractor = DependencyExtractor(parallel_workers=4)
+
+        # This is more of an integration test placeholder
+        # Real parallel testing would require actual compiler invocations
+        self.assertIsNotNone(extractor)
+
+
+class TestEdgeCases(unittest.TestCase):
+    """Tests for edge cases and error handling."""
+
+    def test_handles_missing_compile_commands(self):
+        """Should raise appropriate error for missing compile_commands.json."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        with self.assertRaises(FileNotFoundError):
+            parser = CompileCommandsParser("/nonexistent/compile_commands.json")
+            parser.parse()
+
+    def test_handles_malformed_json(self):
+        """Should handle malformed JSON gracefully."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        temp_dir = tempfile.mkdtemp()
+        try:
+            path = os.path.join(temp_dir, "compile_commands.json")
+            with open(path, "w") as f:
+                f.write("not valid json {{{")
+
+            parser = CompileCommandsParser(path)
+            with self.assertRaises(json.JSONDecodeError):
+                parser.parse()
+        finally:
+            shutil.rmtree(temp_dir)
+
+    def test_handles_empty_ninja_file(self):
+        """Should handle empty ninja file gracefully."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        temp_dir = tempfile.mkdtemp()
+        try:
+            ninja_file = os.path.join(temp_dir, "build.ninja")
+            with open(ninja_file, "w") as f:
+                f.write("")
+
+            parser = NinjaTargetParser(ninja_file)
+            exe_to_objects = parser.parse_executable_mappings()
+
+            self.assertEqual(len(exe_to_objects), 0)
+        finally:
+            shutil.rmtree(temp_dir)
+
+
+if __name__ == "__main__":
+    unittest.main()

script/dependency-parser/tests/test_integration.py

@@ -0,0 +1,264 @@
+#!/usr/bin/env python3
+# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
+# SPDX-License-Identifier: MIT
+
+"""
+Integration tests for CMake Dependency Analyzer.
+
+These tests use real compile_commands.json and actual AMD clang compiler
+to verify the analyzer works correctly in production environment.
+"""
+
+import json
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+# Add parent directory to path for imports
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+# Skip all tests if compile_commands.json doesn't exist
+CK_ROOT = Path(__file__).parent.parent.parent.parent
+BUILD_DIR = CK_ROOT / "build"
+COMPILE_COMMANDS = BUILD_DIR / "compile_commands.json"
+BUILD_NINJA = BUILD_DIR / "build.ninja"
+
+SKIP_INTEGRATION = not COMPILE_COMMANDS.exists()
+SKIP_REASON = f"compile_commands.json not found at {COMPILE_COMMANDS}"
+
+
+@unittest.skipIf(SKIP_INTEGRATION, SKIP_REASON)
+class TestRealCompileCommands(unittest.TestCase):
+    """Tests using real compile_commands.json."""
+
+    def test_parse_real_compile_commands(self):
+        """Should parse real CK compile_commands.json."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        commands = parser.parse()
+
+        # CK has thousands of source files
+        self.assertGreater(len(commands), 100)
+
+        # Verify structure
+        for cmd in commands[:5]:
+            self.assertIn("file", cmd)
+            self.assertIn("directory", cmd)
+            self.assertIn("command", cmd)
+
+    def test_filter_cpp_files_only(self):
+        """Should correctly filter to only .cpp files."""
+        from cmake_dependency_analyzer import CompileCommandsParser
+
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        commands = parser.parse(extensions=[".cpp"])
+
+        for cmd in commands:
+            self.assertTrue(
+                cmd["file"].endswith(".cpp"),
+                f"Expected .cpp file, got {cmd['file']}",
+            )
+
+
+@unittest.skipIf(SKIP_INTEGRATION, SKIP_REASON)
+class TestRealDependencyExtraction(unittest.TestCase):
+    """Tests using real AMD clang for dependency extraction."""
+
+    def test_extract_real_dependencies(self):
+        """Should extract dependencies using real AMD clang."""
+        from cmake_dependency_analyzer import CompileCommandsParser, DependencyExtractor
+
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        commands = parser.parse(extensions=[".cpp"])
+
+        # Test with first command
+        if not commands:
+            self.skipTest("No compile commands found")
+
+        cmd = commands[0]
+        extractor = DependencyExtractor()
+        deps = extractor.extract(cmd["directory"], cmd["command"], cmd["file"])
+
+        # Should have at least the source file itself
+        self.assertGreater(len(deps), 0, f"No deps found for {cmd['file']}")
+
+        # Should include the source file
+        source_basename = os.path.basename(cmd["file"])
+        found_source = any(source_basename in d for d in deps)
+        self.assertTrue(found_source, f"Source file not in deps: {deps[:5]}")
+
+    def test_extract_header_dependencies(self):
+        """Should find CK header dependencies."""
+        from cmake_dependency_analyzer import CompileCommandsParser, DependencyExtractor
+
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        commands = parser.parse(extensions=[".cpp"])
+
+        # Find a test file that includes CK headers
+        test_cmd = None
+        for cmd in commands:
+            if "test_" in cmd["file"] or "example_" in cmd["file"]:
+                test_cmd = cmd
+                break
+
+        if not test_cmd:
+            self.skipTest("No test file found")
+
+        extractor = DependencyExtractor()
+        deps = extractor.extract(test_cmd["directory"], test_cmd["command"], test_cmd["file"])
+
+        # Should include CK headers
+        ck_headers = [d for d in deps if "include/ck" in d or "include/ck_tile" in d]
+        self.assertGreater(
+            len(ck_headers), 0,
+            f"No CK headers found in deps for {test_cmd['file']}"
+        )
+
+
+@unittest.skipIf(SKIP_INTEGRATION, SKIP_REASON)
+@unittest.skipIf(not BUILD_NINJA.exists(), "build.ninja not found")
+class TestRealNinjaParsing(unittest.TestCase):
+    """Tests using real build.ninja."""
+
+    def test_parse_real_executables(self):
+        """Should parse real executable mappings from build.ninja."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        parser = NinjaTargetParser(str(BUILD_NINJA))
+        exe_to_objects = parser.parse_executable_mappings()
+
+        # CK has many test executables
+        test_exes = [e for e in exe_to_objects if "test_" in e]
+        self.assertGreater(len(test_exes), 10, "Expected many test executables")
+
+        # Each executable should have at least one object file
+        for exe, objs in list(exe_to_objects.items())[:5]:
+            self.assertGreater(len(objs), 0, f"No objects for {exe}")
+
+    def test_parse_real_object_sources(self):
+        """Should parse real object -> source mappings."""
+        from cmake_dependency_analyzer import NinjaTargetParser
+
+        parser = NinjaTargetParser(str(BUILD_NINJA))
+        obj_to_source = parser.parse_object_to_source()
+
+        # Should have many object files
+        self.assertGreater(len(obj_to_source), 100)
+
+        # Each mapping should have valid source file
+        for obj, src in list(obj_to_source.items())[:5]:
+            self.assertTrue(
+                src.endswith((".cpp", ".cc", ".cu", ".hip")),
+                f"Invalid source for {obj}: {src}",
+            )
+
+
+@unittest.skipIf(SKIP_INTEGRATION, SKIP_REASON)
+@unittest.skipIf(not BUILD_NINJA.exists(), "build.ninja not found")
+class TestFullIntegration(unittest.TestCase):
+    """Full integration test of the analyzer."""
+
+    def test_small_batch_analysis(self):
+        """Should analyze a small batch of files correctly."""
+        from cmake_dependency_analyzer import (
+            CompileCommandsParser,
+            DependencyExtractor,
+            NinjaTargetParser,
+            DependencyMapper,
+        )
+
+        # Parse compile commands (limit to 10 for speed)
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        all_commands = parser.parse(extensions=[".cpp"])
+        commands = all_commands[:10]
+
+        # Extract dependencies
+        extractor = DependencyExtractor()
+        source_to_deps = extractor.extract_batch(commands)
+
+        self.assertEqual(len(source_to_deps), len(commands))
+
+        # Parse ninja
+        ninja_parser = NinjaTargetParser(str(BUILD_NINJA))
+        exe_to_objects = ninja_parser.parse_executable_mappings()
+        obj_to_source = ninja_parser.parse_object_to_source()
+
+        # Build mapping
+        mapper = DependencyMapper(workspace_root=str(CK_ROOT))
+        file_to_exes = mapper.build_mapping(exe_to_objects, obj_to_source, source_to_deps)
+
+        # Should have some mappings (depends on which files were analyzed)
+        # This test mainly verifies no crashes occur
+        self.assertIsInstance(file_to_exes, dict)
+
+    def test_output_json_format(self):
+        """Should produce JSON compatible with selective_test_filter.py."""
+        from cmake_dependency_analyzer import CMakeDependencyAnalyzer
+
+        # Create analyzer with limited scope
+        analyzer = CMakeDependencyAnalyzer(
+            compile_commands_path=str(COMPILE_COMMANDS),
+            ninja_path=str(BUILD_NINJA),
+            workspace_root=str(CK_ROOT),
+            parallel_workers=1,
+        )
+
+        # Manually set minimal data for output test
+        analyzer.file_to_executables = {
+            "include/ck/ck.hpp": {"bin/test_gemm"},
+        }
+        analyzer.executable_to_files = {
+            "bin/test_gemm": {"include/ck/ck.hpp"},
+        }
+
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
+            output_path = f.name
+
+        try:
+            analyzer.export_to_json(output_path)
+
+            with open(output_path) as f:
+                data = json.load(f)
+
+            # Verify format matches selective_test_filter.py expectations
+            self.assertIn("file_to_executables", data)
+            self.assertIn("statistics", data)
+
+            # Values should be lists, not sets
+            for key, value in data["file_to_executables"].items():
+                self.assertIsInstance(value, list)
+        finally:
+            os.unlink(output_path)
+
+
+@unittest.skipIf(SKIP_INTEGRATION, SKIP_REASON)
+class TestPerformance(unittest.TestCase):
+    """Performance tests."""
+
+    def test_extraction_speed(self):
+        """Single file extraction should be fast (<1s)."""
+        import time
+        from cmake_dependency_analyzer import CompileCommandsParser, DependencyExtractor
+
+        parser = CompileCommandsParser(str(COMPILE_COMMANDS))
+        commands = parser.parse(extensions=[".cpp"])
+
+        if not commands:
+            self.skipTest("No compile commands")
+
+        cmd = commands[0]
+        extractor = DependencyExtractor()
+
+        start = time.time()
+        deps = extractor.extract(cmd["directory"], cmd["command"], cmd["file"])
+        elapsed = time.time() - start
+
+        self.assertLess(elapsed, 1.0, f"Extraction took {elapsed:.2f}s, expected <1s")
+        self.assertGreater(len(deps), 0, "No dependencies extracted")
+
+
+if __name__ == "__main__":
+    unittest.main()