mirror of
https://github.com/ROCm/composable_kernel.git
synced 2026-05-01 12:11:19 +00:00
ec006bb8e0
* Improve random number generation * use different seed for each input (Q, K, V...); * use deterministic generation of: * seqstart_q/k (for group mode); * block_table (for paged-kvcahe); * cache_batch_idx (for kvcache); * Extract arg_parser-related code from run functions to use them as tests * Split examples into main programs and fmha runners, build instances separately * Add dummy tests that use instances and runners * Fix a missed corner case of f32->f8 conversion When value if < min f8 denormal but > min f8 denormal / 2, it must be rounded to min f8 denormal (i.e. 0b1), not to 0. * Fix incorrect fp8 scales for P and O in validation code DataTypeConfig was incorrectly compared with fp8_t. * Add host generation of dropout random values and use it for validation Previously host validation (reference_batched_dropout) used random numbers generated by BlockDropout of the kernel, meaning that incorrect generation on device (bad distribution, repeated numbers, too many zeros, etc.) would not trigger any validation errors. * Implement tests from smoke_test_bwd.sh * Return result as enum to distinguish failure and missing instance * Add tests for bwd features: bias, alibi, dropout * Implement tests from smoke_test_fwd.sh * Pass seqlen_q/k as vectors to fwd and bwd runners * Add tests for fwd features: bias, alibi, dropout * Add tests for pagedkv and splitkv * Fix conditions when to use splitkv and pagedkv kernels splitkv was executed only when use_kvcache which == (need_append_kvcache || use_cache_batch_idx || 0 < page_block_size). In the SplitKV tests: the regular fwd kernel was executed if use_cache_batch_idx was not requested even when num_splitkv > 1. In the AppendKV tests: the pagedkv kernel was executed but it often failed to find an instance. * Add tests for appendkv * Use is_v_rowmajor = true because there are no instances with column layout anymore * Split public and private compile options for instances Tests and examples need to know only about CK_TILE_FMHA_FWD_*_API. * Improve parsing validation in bias and mask * Pass bias as string for consistency with mask * Catch parsing and other exceptions * Add bwd test for deterministic flag * Initialize fp8 tensors (-init=ufq) similarly to uf * Fix splitkv/pagedkv invocation: use padded sk when seqlen_k_ptr is not null seqlen_k cannot be used to determine padding when seqlen_k_ptr is provided. The actual seqlen_k is taken from seqlen_k_ptr[b]. Even seqlen_k values (% bn0 == 0) use padded seqlen_k while seqlen_k_ptr may contain arbitrary values. In the example or tests this produces incorrect results with appendkv (for example, -d=32 -s=1 -s_k=64 -s_knew=7 -vlayout=c -b=8). * Fix use_pagedkv value when kvcache = true but page_block_size = 0 In this case block_table_ptr is nullptr which is accessed in the kernel. * Clean up bwd tests * Unify fwd tests for f16/bf16 and fp8 * Use better explicit instantiation declaration for fmha_bwd<2> * Use the same seed for all tests, allow to override it with env variable * Undo clang-format of one irrelevant file For some reason my local clang-format-18 and the one in CI work differently. * Do not build instances and tests on unsupported archs * Build instance libraries as OBJECT library * CI: Enable sccache for HIP There are source files with LANGUAGE HIP, they need -DCMAKE_HIP_COMPILER_LAUNCHER=sccache * Add tests to REGRESSION_TESTS * Fix OOB accesses in deterministic bwd due to incorrectly assumed kN0 The runner assumes kN0 = (hdim_q <= 128) ? 128 : 64 but there are smaller tiles (for tr_load or fp32). This can create too small dq_acc_buf. * Pass CK_TILE_FMHA_FWD_*_API as INTERFACE compile options The instances don't actually depend on them, only examples and tests do. Passing these definitions as INTERFACE allows to change FMHA_FWD_ENABLE_APIS without recompiling instances that are already in ccache. * Fix formatting and names
48 lines
2.3 KiB
C++
48 lines
2.3 KiB
C++
// SPDX-License-Identifier: MIT
|
|
// Copyright (c) 2018-2025, Advanced Micro Devices, Inc. All rights reserved.
|
|
|
|
#pragma once
|
|
|
|
#include "ck_tile/host/arg_parser.hpp"
|
|
#include "ck_tile/host/check_err.hpp"
|
|
#include "ck_tile/host/concat.hpp"
|
|
#include "ck_tile/host/convolution_host_tensor_descriptor_helper.hpp"
|
|
#include "ck_tile/host/convolution_parameter.hpp"
|
|
#include "ck_tile/host/device_memory.hpp"
|
|
#include "ck_tile/host/device_prop.hpp"
|
|
#include "ck_tile/host/fill.hpp"
|
|
#include "ck_tile/host/flush_icache.hpp"
|
|
#include "ck_tile/host/hip_check_error.hpp"
|
|
#include "ck_tile/host/host_tensor.hpp"
|
|
#include "ck_tile/host/joinable_thread.hpp"
|
|
#include "ck_tile/host/kernel_launch.hpp"
|
|
#include "ck_tile/host/ranges.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_dropout.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_dropout_randval.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_elementwise.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_gemm.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_masking.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_rotary_position_embedding.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_softmax.hpp"
|
|
#include "ck_tile/host/reference/reference_batched_transpose.hpp"
|
|
#include "ck_tile/host/reference/reference_elementwise.hpp"
|
|
#include "ck_tile/host/reference/reference_fused_moe.hpp"
|
|
#include "ck_tile/host/reference/reference_gemm.hpp"
|
|
#include "ck_tile/host/reference/reference_grouped_conv_bwd_data.hpp"
|
|
#include "ck_tile/host/reference/reference_grouped_conv_bwd_weight.hpp"
|
|
#include "ck_tile/host/reference/reference_grouped_conv_fwd.hpp"
|
|
#include "ck_tile/host/reference/reference_im2col.hpp"
|
|
#include "ck_tile/host/reference/reference_layernorm2d_fwd.hpp"
|
|
#include "ck_tile/host/reference/reference_moe_sorting.hpp"
|
|
#include "ck_tile/host/reference/reference_permute.hpp"
|
|
#include "ck_tile/host/reference/reference_reduce.hpp"
|
|
#include "ck_tile/host/reference/reference_rmsnorm2d_fwd.hpp"
|
|
#include "ck_tile/host/reference/reference_rowwise_quantization2d.hpp"
|
|
#include "ck_tile/host/reference/reference_softmax.hpp"
|
|
#include "ck_tile/host/reference/reference_topk.hpp"
|
|
#include "ck_tile/host/reference/reference_transpose.hpp"
|
|
#include "ck_tile/host/rotating_buffers.hpp"
|
|
#include "ck_tile/host/stream_config.hpp"
|
|
#include "ck_tile/host/stream_utils.hpp"
|
|
#include "ck_tile/host/timer.hpp"
|