mirror of
https://github.com/ROCm/composable_kernel.git
synced 2026-05-04 21:51:28 +00:00
[CK] [CK_TILE] Improve build and test time of CI with smart dependency parser (#5249) ## Motivation Existing dependency parser needs full build of tests to determine which tests are affected by code changes in a PR. This still takes 2-4 hours for building the tests which slows down the CI as the number of tests grow. To resolve this issue we implemented a smart dependency parser which uses CMake Configure to parse dependencies and build only the affected test cases. We have ensured that two approaches are available 1) CMake pre-build analysis for each PR to ensure fast build and test. 2) Ninja post-build analysis to enable full build for nightly tests. ## Technical Details ```bash ### 1. Configure the project with CMake cmake -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON .. ### 2. Analyze dependencies (no build required!) python3 ../script/dependency-parser/main.py cmake-parse compile_commands.json build.ninja \ --workspace-root .. --output cmake_dependency_mapping.json --parallel 8 ### 3. Find tests affected by changes python3 ../script/dependency-parser/main.py select cmake_dependency_mapping.json origin/develop \ HEAD --test-prefix --output tests_to_run.json ### 4. Build only affected tests ninja $(jq -r '.executables[]' tests_to_run.json | tr '\n' ' ') ### 5. Run affected tests ctest -R "$(jq -r '.regex' tests_to_run.json)" ``` ### Jenkins Integration - Added `buildMode` to jenkinsfile to integrate both `selective` and `full` build methods ### Known Limitations ### 1. Build-Time Generated Headers (HIGH RISK) **Problem:** Files generated during the build process (e.g., via `add_custom_command`) cannot be analyzed before building. **Example:** ```cmake add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/generated/config.hpp COMMAND generate_config.sh DEPENDS template.hpp.in ) ``` **Impact:** If a source file includes `generated/config.hpp`, the dependency won't be detected until after building. **Mitigation:** - CK analysis shows **no generated headers** currently used - If generated headers are added in the future, they must be built first - Recommendation: Generate headers in CMake configure phase (not build phase) when possible ## Test Plan **1. Modified Files:** ``` include/ck_tile/ops/common.hpp include/ck_tile/ops/gemm.hpp include/ck_tile/ops/gemm/warp/warp_gemm.hpp ``` **2. Compare tests selected between `build.ninja` and `cmake-parse` methods** ## Test Result - 1. The test completed in 5-6 minutes finding about 8000+ executables that should be built. - 2. We selected a commit 5ccc1387ea which resulted in same 7 tests with both legacy and new methods. - PR | Legacy tests | Smart tests | Notes -- | -- | -- | -- 5261 | 453 | 455 | Only 2 tests (test_amdgcn_mma and test_amdgcn_sparse_mma) 5168 | 0 | 0 | Changes in dispatcher only. No CK tests invoked. 5249 | 0 | 0 | Changes to dependency parser. No CK tests invoked 5260 | 0 | 0 | Changes in dispatcher only. No CK tests invoked. 5174 | 1 | 1 | One test from FMHA affected by this PR in both cases 5383 | 0 | 0 | Changes are only in benchmark files. Did not trigger any tests 5445 | 1 | 1 | Changes are only to tests/ck_tile/gemm_streamk. Only triggered one streamk test in both cases. 5454 | 3 | 3 | Both methods identified same test_grouped_conv_bwd tests 5427 | 234 | 234 | Core infrastructure header changes. Detected exactly same tests 5388 | 85 | 85 | modifies warp-level GEMM operations (warp_gemm.hpp, warp_gemm_dispatcher.hpp). Correctly identified all the streamK gemm tests ## Submission Checklist - [x ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
137 lines
4.0 KiB
Bash
Executable File
137 lines
4.0 KiB
Bash
Executable File
#!/bin/bash
|
|
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
|
|
# SPDX-License-Identifier: MIT
|
|
|
|
# Smart Build CI Script
|
|
#
|
|
# This script orchestrates the smart-build process:
|
|
# 1. Runs ci_safety_check.sh to determine if selective build is safe
|
|
# 2. Generates dependency map using cmake-parse
|
|
# 3. Selects affected tests
|
|
# 4. Outputs build targets to a file for Jenkins to consume
|
|
#
|
|
# Exit codes:
|
|
# 0 = Success (selective build targets generated)
|
|
# 1 = Full build required (run ninja check)
|
|
#
|
|
# Output files:
|
|
# tests_to_run.json - Selected tests and executables
|
|
# build_targets.txt - Space-separated list of ninja targets to build
|
|
# build_mode.env - Environment variables (SMART_BUILD_MODE=selective|full)
|
|
|
|
set -e
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
BUILD_DIR="${BUILD_DIR:-$(pwd)}"
|
|
WORKSPACE_ROOT="${WORKSPACE_ROOT:-$(cd ${BUILD_DIR}/.. && pwd)}"
|
|
PARALLEL="${PARALLEL:-32}"
|
|
BASE_BRANCH="${BASE_BRANCH:-develop}"
|
|
|
|
echo "========================================="
|
|
echo "Smart Build CI"
|
|
echo "========================================="
|
|
echo "BUILD_DIR: ${BUILD_DIR}"
|
|
echo "WORKSPACE_ROOT: ${WORKSPACE_ROOT}"
|
|
echo "BASE_BRANCH: ${BASE_BRANCH}"
|
|
echo "PARALLEL: ${PARALLEL}"
|
|
echo "-----------------------------------------"
|
|
|
|
# Step 1: Run CI safety check
|
|
echo "Step 1: Running CI safety check..."
|
|
cd "${BUILD_DIR}"
|
|
|
|
if ! bash "${SCRIPT_DIR}/ci_safety_check.sh"; then
|
|
echo "CI safety check failed - full build required"
|
|
echo "full" > build_targets.txt
|
|
exit 1
|
|
fi
|
|
|
|
echo "✓ CI safety check passed - selective build enabled"
|
|
|
|
# Step 2: Generate dependency map
|
|
echo ""
|
|
echo "Step 2: Generating dependency map..."
|
|
if [ ! -f "compile_commands.json" ]; then
|
|
echo "Error: compile_commands.json not found in ${BUILD_DIR}"
|
|
echo "Make sure cmake configure has been run with -DCMAKE_EXPORT_COMPILE_COMMANDS=ON"
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -f "build.ninja" ]; then
|
|
echo "Error: build.ninja not found in ${BUILD_DIR}"
|
|
echo "Make sure cmake configure has been run with -G Ninja"
|
|
exit 1
|
|
fi
|
|
|
|
python3 "${SCRIPT_DIR}/main.py" cmake-parse \
|
|
compile_commands.json \
|
|
build.ninja \
|
|
--workspace-root "${WORKSPACE_ROOT}" \
|
|
--parallel ${PARALLEL} \
|
|
--output enhanced_dependency_mapping.json
|
|
|
|
if [ ! -f "enhanced_dependency_mapping.json" ]; then
|
|
echo "Error: Failed to generate enhanced_dependency_mapping.json"
|
|
exit 1
|
|
fi
|
|
|
|
echo "✓ Dependency map generated"
|
|
|
|
# Step 3: Select affected tests
|
|
echo ""
|
|
echo "Step 3: Selecting affected tests..."
|
|
python3 "${SCRIPT_DIR}/main.py" select \
|
|
enhanced_dependency_mapping.json \
|
|
origin/${BASE_BRANCH} \
|
|
HEAD \
|
|
--ctest-only \
|
|
--output tests_to_run.json
|
|
|
|
if [ ! -f "tests_to_run.json" ]; then
|
|
echo "Error: Failed to generate tests_to_run.json"
|
|
exit 1
|
|
fi
|
|
|
|
# Step 4: Check if any tests were selected
|
|
num_tests=$(jq -r '.tests_to_run | length' tests_to_run.json 2>/dev/null || echo "0")
|
|
echo "✓ Selected ${num_tests} tests"
|
|
|
|
if [ "${num_tests}" -eq 0 ]; then
|
|
echo ""
|
|
echo "========================================="
|
|
echo "Result: No tests affected by changes"
|
|
echo "========================================="
|
|
echo "none" > build_targets.txt
|
|
exit 0
|
|
fi
|
|
|
|
# Step 5: Extract build targets (executables)
|
|
echo ""
|
|
echo "Step 4: Extracting build targets..."
|
|
jq -r '.executables[]' tests_to_run.json | tr '\n' ' ' > build_targets.txt
|
|
|
|
num_targets=$(jq -r '.executables | length' tests_to_run.json)
|
|
echo "✓ Generated ${num_targets} build targets"
|
|
|
|
# Display summary
|
|
echo ""
|
|
echo "========================================="
|
|
echo "Smart Build Summary"
|
|
echo "========================================="
|
|
echo "Tests to run: ${num_tests}"
|
|
echo "Build targets: ${num_targets}"
|
|
echo "Output files:"
|
|
echo " - tests_to_run.json (test selection)"
|
|
echo " - build_targets.txt (ninja targets)"
|
|
echo " - build_mode.env (SMART_BUILD_MODE=selective)"
|
|
echo "========================================="
|
|
|
|
# Show first few targets for verification
|
|
echo ""
|
|
echo "Sample build targets (first 5):"
|
|
head -1 build_targets.txt | tr ' ' '\n' | head -5
|
|
|
|
echo ""
|
|
echo "✓ Smart build preparation complete"
|
|
exit 0
|