Non-AVX512 variants of ZGEMM tiny code path do not support as many
problem sizes and transa/transb options. Skip unsuitable tests based
on the return value from from bli_zgemm_tiny call.
Also remove unused local variables.
AMD-Internal: [CPUPL-7303]
Update problem sizes and other parameters for general GEMM and
GEMMT tests, and dgemmt EVT tests, with the aim of reducing the
runtime and make tests more practical for use in pre-submit CI jobs.
AMD-Internal: [CPUPL-7386]
* Resolved memory-access issues in the SGEMM SUP kernels on AVX2 and AVX-512 by correcting instructions that could read invalid addresses in the C matrix.
* Removed k=0 kernel gtests for the native SGEMM and DGEMM paths as these tests caused spurious failures for kernels that are not intended to handle this case.
* Standardized all instruction macros to lowercase in the Zen4 kernel to improve readability and code consistency.
---------
AMD-Internal: CPUPL-8117
Co-authored-by: Rayan <rohrayan@amd.com>
* Bitexactness CRC verification and per-test JSON output
* Remove redundant BLIS_TEST_SEED random seed utilities
The random_seed_utils.h and BLIS_TEST_SEED environment variable are unnecessary since the codebase already ensures deterministic random number generation via RANDOM_POOL_SEED and SRAND_SEED constants hardcoded in testing_helpers.h.
* Add CRC support for integer/char computediff and cache env var checks
Add CRC calculation and binary output to the gtint_t and char specializations of computediff, matching the pattern used by all other overloads. Char values are widened to gtint_t for safe uint32_t-aligned CRC access.
Cache BLIS_ENABLE_CRC and BLIS_ENABLE_BINARY_OUTPUT env var lookups via static const bool in is_crc_enabled() and is_binary_output_enabled(). Guard all CRC/binary blocks in computediff with is_any_verification_enabled() so the common disabled path is a single static bool read with zero allocations.
* Address PR review comments and refactor computediff CRC blocks
Refactor: Extract duplicated CRC/binary-output blocks from all 8 computediff overloads into verify_vector_data and verify_matrix_data helpers in blis_test_utils namespace.
Bug fixes from PR review: add missing includes (cstdlib, utility), enforce MAX_OUTPUT_SIZE_BYTES limit with integer overflow guard, add buffer validation in all CRC generation functions, add default case to FLA_GET_DATATYPE_FACTOR macro, replace deprecated test_case_name() with test_suite_name(), add MAKE_DIRECTORY error checking in CMake, and update copyright years to 2026.
* Refactored crc_utils based on review comments.
* binary_output_utils.h cleanup.
* Address PR review comments: remove unused functions and fix copyright years.
Remove unused generate_crc_matrix, generate_crc_matrix_no_nb_diag,
generate_crc_matrix_no_nb_diag_with_storage, and
calculate_and_print_matrix_crc from crc_utils.h.
Remove unused calculate_and_print_matrix_hash from check_error.h.
Fix copyright year to 2026 only in crc_utils.h and binary_output_utils.h.
Remove (Performance) label from CRC heading in README.md.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix for review comments.
* Address review comments: rename verify to collect, consistent void returns, remove filename prefix
- Rename verify_vector_data/verify_matrix_data to collect_vector_data/
collect_matrix_data since these functions only collect CRC and binary
output data without performing comparison.
- Make return types consistent: change calculate_and_print_crc,
calculate_and_print_matrix_crc_with_storage, format_and_record_crc,
and write_comparison_outputs to return void since return values were
never used.
- Remove redundant test_output_ prefix from generate_binary_filename
to avoid duplication with the blis_test_outputs/ directory.
- Remove unused utility include from binary_output_utils.h.
- Update README wording from compiled out to disabled.
Made-with: Cursor
* Fix strict aliasing, use if constexpr, zero-pad CRC hex, separate feature guards
- Replace reinterpret_cast<uint32_t*> with memcpy-based read_uint32()
helper to avoid strict-aliasing UB on float/double/complex buffers.
Produces identical CRC values.
- Use if constexpr(CRC_ENABLED) instead of runtime if(!CRC_ENABLED) to
prevent CRC template instantiation when ENABLE_CRC is off.
- Zero-pad CRC hex output to 8 digits for stable downstream comparison.
- Separate ENABLE_CRC and ENABLE_BINARY_OUTPUT preprocessor guards in
verification_utils.h so each feature is compiled independently.
Made-with: Cursor
* Handle write_binary_output return values in write_comparison_outputs
Capture the bool return values from write_binary_output and, on failure, log a warning to stdout and record the error as a GTest property. This keeps binary output as a non-fatal diagnostic aid while ensuring return values are explicitly used.
Made-with: Cursor
---------
Co-authored-by: Anuraj <avettick@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* ZGEMM SUP: Add conjugate support for AVX-512 kernels on Zen4/Zen5/Zen6
- Add CONJA, CONJB and CONJA_CONJB variants to zgemm SUP micro-tiles
- Enable SUP path for conjugate cases when both are same type
- Unify RRC/CRC storage to use CV kernel variant
- Update SUP dispatch to handle conjugate flags correctly
Note: CONJ_NO_TRANSPOSE + CONJ_NO_TRANSPOSE and
CONJ_TRANSPOSE + CONJ_TRANSPOSE remain unsupported
---------
Co-authored-by: harsdave <harsdave@amd.com>
Various changes:
- Correct signature of get_random_matrix call in some level2 APIs.
- Move RNG seed to header to allow it to be used elsewhere in the code
- Remove unused variable in ref_gbmv.cpp
- Fix seed for all calls to rand()
- Correct arguments in calls to matrix setup and computediff calls,
especially for CBLAS row-major calls
- Add missing if statements in tests of input arguments
- Removed unused alpha argument from tbmv and tbsv
- Enable nan_inf check when testing input args like alpha and beta
- Also some corrections to testing input matrices and vectors
AMD-Internal: [CPUPL-7386]
Create gtestsuite programs for banded matrix APIs. Priority is to
create framework with a basic set of tests - refinement of problem
sizes can be investigated later.
AMD-Internal: [CPUPL-7386]
Create gtestsuite programs for subset of packed matrix APIs. Priority is to
create framework with a basic set of tests - refinement of problem sizes
can be investigated later.
AMD-Internal: [CPUPL-7386]
Create gtestsuite programs for subset of packed matrix APIs. Priority is to
create framework with a basic set of tests - refinement of problem sizes
can be investigated later.
AMD-Internal: [CPUPL-7386]
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Move asumv and nrm2 testinghelpers files from util to level1 (missed in
commit 0923d8ff56)
- Correct spelling mistakes and references to incorrect arguments in
comments in various files
- Correct comments listing invalid input tests in syr_IIT_ERS.cpp and
her_IIT_ERS.cpp
- Fix incorrect use of M in symv_IIT_ERS.cpp and hemv_IIT_ERS.cpp
AMD-Internal: [CPUPL-7386]
IIT_ERS tests were only implemented for {c,z}dotu and {c,z}geru. Add
the same tests for conjugate variants {c,z}dotc and {c,z}gerc.
AMD-Internal: [CPUPL-7386]
* Adding data pool for random number generator
* Adopting pool in axpy, gemv, and ukr.gemm
* Using templates in data pool getter function to generate different pools for different ranges
* Addressing review comments
* Using data pool in more APIs
* Using data pool in more APIs
* Updating testing infra for trsv and trsm
* Addressing review comments
* Fixing numerical issues in trsm testing
* More review comments
* Small change to get trsm tests to pass
* Scalling tiny data pool
* Addressing review comments
In earlier commits (e.g. a2beef3255) long running tests were broken
into separate executables for each data type. Standardize this approach
across all APIs, including separate executables for IIT_ERS tests, and
for each data type have separate executables for EVT tests.
AMD-Internal: [CPUPL-7386]
Co-authored-by: Vlachopoulou, Eleni <Eleni.Vlachopoulou@amd.com>
* GTestSuite: computediff improvements
- Using lazy evaluation to only create error strings when the comparison failed
- Check if increments are zero, then only check first element
- Change begin_asm and end_asm comments and unused code in files
kernels/haswell/3/sup/s6x16/bli_gemmsup_rv_haswell_asm_sMx6.c
kernels/zen4/3/sup/bli_gemmsup_cd_zen4_asm_z12x4m.c
to avoid problems in clobber checking script.
- Add missing clobbers in files
kernels/zen4/1m/bli_packm_zen4_asm_d24xk.c
kernels/zen4/1m/bli_packm_zen4_asm_z12xk.c
kernels/zen4/3/sup/bli_gemmsup_cv_zen4_asm_z12x4m.c
- Add missing newline at end of files.
- Update some copyright years for recent changes.
- Standardize license text formatting.
AMD-Internal: [CPUPL-6579]
- Added WORKING_DIRECTORY to try_run() calls to ensure execution occurs in ${BLIS_PATH}/lib.
- Prevents Windows error 0xc0000135 caused by missing DLLs during runtime of get_version.cpp.
- Ensures compatibility with both static and shared BLIS builds by aligning runtime context
with expected DLL locations.
AMD-Internal: [CPUPL-7187]
* Optimized avx512 ZGEMM kernel and edge-case handling
Edge kernel implementation:
- Refactored all of the zgemm kernels to process micro-tiles efficiently
- Specialized sub-kernels are added to handle leftover m dimention:12MASK,
8, 8MASK, 8, 4, 4MASK, 2.
- 12MASK edge kernel handles 11, 10, 9 m_left using 2 full zmm
load/store and 1 masked load/store.
- Similarly 8MASK handles 7, 6, 5 m_left using 1 full zmm load/store and
1 masked load/store.
- 4MASK handles 3, 1 m_left using 1 masked load/store.
- ZGEMM kernel now internally decomposes the m dimension into the following.
The main kernel is 12x4, which is having following edge kernels to
handle left-over m dimension:
edge kernels:
12MASKx4 (handles 11x4, 10x4, 9x4)
8x4 (handles 8x4)
8MASKx4 (handles 7x4, 6x4, 5x4)
4x4 (handles 4x4)
4MASKx4 (handles 3x4, 1x4)
2x4 (handles 2x4)
- similarly it decomposes for (12x3, 12x2 and 12x1) n_left kernels under
which the following edge kernels 12MASKxN_LEFT(3, 2, 1), 8XN_LEFT(3, 2, 1),
8MASKxN_LEFT(3, 2, 1), 4xN_LEFT(3, 2, 1), 4MASKxN_LEFT(3, 2, 1),
2xN_LEFT(3, 2, 1) handles leftover m dimension.
Threshold tuning:
- Enforced odd m dimension to avx512 kernels in tiny path, as avx2
kernels invokes gemv calls for m_left=1(odd m dimension of matrix)
The gemv function call adds overhead for very small sizes and results
in suboptimal performance.
- condition check "m%2 == 0" is added along with threshold checks to
force input with odd m dimension to use avx512 zgemm kernel.
- Threshold change to route all of the inputs to tiny path. Eliminating
dependency of avx2 zgemm_small path if A, B matrix storage is 'N'(not transpose) or
'T'(transpose).
- However tiny re-uses zgemm sup kernels which do not support
conjugate transpose storage of matrices. For such storage of
A, B matrix we still rely on avx2 zgemm_small kernel.
gtest changes:
- Removed zgemm edge kernel function(8x4, 4x4, 2x4 and fx4) and their
respective testing instaces from gtest.
AMD-Internal: [CPUPL-7203]
* Optimized avx512 ZGEMM kernel and edge-case handling
Edge kernel implementation:
- Refactored all of the zgemm kernels to process micro-tiles efficiently
- Specialized sub-kernels are added to handle leftover m dimention:12MASK,
8, 8MASK, 8, 4, 4MASK, 2.
- 12MASK edge kernel handles 11, 10, 9 m_left using 2 full zmm
load/store and 1 masked load/store.
- Similarly 8MASK handles 7, 6, 5 m_left using 1 full zmm load/store and
1 masked load/store.
- 4MASK handles 3, 1 m_left using 1 masked load/store.
- ZGEMM kernel now internally decomposes the m dimension into the following.
The main kernel is 12x4, which is having following edge kernels to
handle left-over m dimension:
edge kernels:
12MASKx4 (handles 11x4, 10x4, 9x4)
8x4 (handles 8x4)
8MASKx4 (handles 7x4, 6x4, 5x4)
4x4 (handles 4x4)
4MASKx4 (handles 3x4, 1x4)
2x4 (handles 2x4)
- similarly it decomposes for (12x3, 12x2 and 12x1) n_left kernels under
which the following edge kernels 12MASKxN_LEFT(3, 2, 1), 8XN_LEFT(3, 2, 1),
8MASKxN_LEFT(3, 2, 1), 4xN_LEFT(3, 2, 1), 4MASKxN_LEFT(3, 2, 1),
2xN_LEFT(3, 2, 1) handles leftover m dimension.
Threshold tuning:
- Enforced odd m dimension to avx512 kernels in tiny path, as avx2
kernels invokes gemv calls for m_left=1(odd m dimension of matrix)
The gemv function call adds overhead for very small sizes and results
in suboptimal performance.
- condition check "m%2 == 0" is added along with threshold checks to
force input with odd m dimension to use avx512 zgemm kernel.
- Threshold change to route all of the inputs to tiny path. Eliminating
dependency of avx2 zgemm_small path if A, B matrix storage is 'N'(not transpose) or
'T'(transpose).
- However tiny re-uses zgemm sup kernels which do not support
conjugate transpose storage of matrices. For such storage of
A, B matrix we still rely on avx2 zgemm_small kernel.
gtest changes:
- Removed zgemm edge kernel function(8x4, 4x4, 2x4 and fx4) and their
respective testing instaces from gtest.
AMD-Internal: [CPUPL-7203]
---------
Co-authored-by: harsdave <harsdave@amd.com>
New tests:
- Add IIT_ERS tests for TRMM, SYMM, SYRK, SYR2K, HEMM, HERK, HER2K
Corrections and improvements:
- GEMM: Use local definitions of input size, trans, etc arguments
to allow finer control of choices, especially for testing invalid
leading dimensions.
- GEMM, GEMMT, GEMM_COMPUTE, TRMM, TRSM: In alpha=beta=zero
test, initialize C to extreme value to test that C is set rather than scaled
- GEMM: Use correct M x N dimensions for C in calls to computediff
- GEMM: Declare info variable in disabled tests in gemm_IIT_ERS.cpp
AMD-Internal: [CPUPL-6725]
- GEMV transpose kernels lack ability to compute directly on non-unit stride inputs.
- This limitation is stopping libflame to use blis kernel directly instead of going through framework.
- Added ability to handle non-unit incx in the kernel by packing x into a temporary buffer.
AMD-Internal: [CPUPL-6903]
New tests:
- Add IIT_ERS tests for HEMV, HER, HER2, SYMV, SYR, SYR2
Corrections and improvements:
- GEMV: Correct matrix sizes for transpose cases when checking
input matrix A has not been modified.
- GEMV: Initialize y to extreme value for alpha=beta=zero case
to check y is set rather than scaled.
AMD-Internal: [CPUPL-6725]
Naming of Zen kernels and associated files was inconsistent with BLIS
conventions for other sub-configurations and between different Zen
generations. Other anomalies existed, e.g. dgemmsup 24x column
preferred kernels names with _rv_ instead of _cv_. This patch renames
kernels and file names to address these issues.
AMD-Internal: [CPUPL-6579]
* Added DGEMV no transpose multithreaded Implementations
- Added new avx512 M and N kernels for DGEMV.
- Added multiple MT implementations for same kernels.
- Added AOCL_dynamic logic for L2 apis.
- Tuned AOCL_dynamic and code path selection for DGEMV on ZEN5.
- Added same kernels for SGEMV, but these kernels are not enabled yet.
- Added SGEMV reference kernel.
AMD-Internal: [SWLCSG-3408]
Co-authored-by: Varaganti, Kiran <Kiran.Varaganti@amd.com>
- Temperory fix for regression in DGEMV for non-unit stride y inputs. The code
section responsible for handling non-unit stride y has been removed from the
frame.
- The kernel code is extended with if condition to handle both unit and non-unit
stride y.
AMD-Internal: [CPUPL-6869]
- Implemented multithreading framework for the DGEMV API on Zen architectures. Architecture specific AOCL-dynamic logic determines the optimal number of threads for improved performance.
- The condition check for the value of beta is optimized by utilizing masked operations. The mask value is set based on value of beta, and the masked operations are applied when the vector y is loaded or scaled with beta.
AMD-Internal: [CPUPL-6746]
- Introducted new assembly kernel that copies data from source
to destination from the front and back of the vector at the
same time. This kernel provides better performance for larger
input sizes.
- Added a wrapper function responsible for selecting the kernel
used by DCOPYV API to handle the given input for zen5
architecture.
- Updated AOCL-dynamic threshold for DCOPYV API in zen4 and
zen5 architectures.
- New unit-tests were included in the grestsuite for the new
kernel.
AMD-Internal: [CPUPL-6650]
Change-Id: Ie2af88b8e97196b6aa02c089e59247742002f568
- Added a set of AVX512 fringe kernels(using masked loads and
stores) in order to avoid rerouting to the GEMV typed API
interface(when m = 1). This ensures uniformity in performance
across the main and fringe cases, when the calls are multithreaded.
- Further tuned the thresholds to decide between ZGEMM Tiny, Small
SUP and Native paths for ZEN4 and ZEN5 architectures(in case
of parallel execution). This would account for additional
combinations of the input dimensions.
- Moved the call to Tiny-ZGEMM before the BLIS object creation,
since this code-path operates on raw buffers.
- Added the necessary test-cases for functional and memory testing
of the newly added kernels.
AMD-Internal: [CPUPL-6378][CPUPL-6661]
Change-Id: I9af73d1b6ef82b26503d4fc373111132aee3afd6
- Implemented an AVX512 rank-1 kernel that is
expected to handle column-major storage schemes
of A, B and C(without transposition) when k = 1.
- This kernel is single-threaded, and acts as a direct
call from the BLAS layer for its compatible inputs.
- Defined custom BLAS and BLIS_IMPLI layers for CGEMM
(instead of using the macro definition), in order to
integrate the call to this kernel at runtime(based on
the corresponding architecture and input constraints).
- Added unit-tests for functional and memory testing of the
kernel.
- Updated the ZEN5 context to include the AVX512 CGEMM
SUP kernels, with its cache-blocking parameters.
AMD-Internal: [CPUPL-6498]
Change-Id: I42a66c424325bd117ceb38970726a05e2896a46b
- Implemented the following AVX512 SUP
column-preferential kernels(m-variant) for CGEMM :
Main kernel : 24x4m
Fringe kernels : 24x3m, 24x2m, 24x1m,
16x4, 16x3, 16x2, 16x1,
8x4, 8x3, 8x2, 8x1,
fx4, fx3, fx2, fx1(where 0<f<8).
- Utlized the packing kernel to pack A when
handling inputs with CRC storage scheme. This
would in turn handle RRC with operation transpose
in the framework layer.
- Further adding C prefetching to the main kernel,
and updated the cache-blocking parameters for
ZEN4 and ZEN5 contexts.
- Added a set of decision logics to choose between
SUP and Native AVX512 code-paths for ZEN4 and ZEN5
architectures.
- Updated the testing interface for complex GEMMSUP
to accept the kernel dimension(MR) as a parameter, in
order to set the appropriate panel stride for functional
and memory testing. Also updated the existing instantiators
to send their kernel dimensions as a parameter.
- Added unit tests for functional and memory testing of these
newly added kernels.
AMD-Internal: [CPUPL-6498]
Change-Id: Ie79d3d0dc7eed7edf30d8d4f74b888135f31d6b4
- Included a new code section to handle input having non-unit strided y
vector for dgemv transpose case. Removed the same from the respective
kernels to avoid repeated branching caused by condition checks within
the 'for' loop.
- The condition check for beta is equal to zero in the primary kernels
are moved outside the for loop to avoid repeated branching.
- The '_mm512_reduce_pd' operations in the primary kernel is replaced by
a series of operations to reduce the number of instructions required
to reduce the 8 registers.
- Changing naming convention for DGEMV transpose kernels.
- Modified unit kernel test to avoid y increment for dgemv tranpose
kernels during the test.
AMD-Internal: [CPUPL-6565]
Change-Id: I1ac516d6b8f156ac53ac9f6eb18badd50e152e05
- Implemented the following AVX512 native
computational kernels for CGEMM :
Row-preferential : 4x24
Column-preferential : 24x4
- The implementations use a common set of macros,
defined in a separate header. This is due to the
fact that the implementations differ solely on
the matrix chosen for load/broadcast operations.
- Added the associated AVX512 based packing kernels,
packing 24xk and 4xk panels of input.
- Registered the column-preferential kernel(24x4) in
ZEN4 and ZEN5 contexts. Further updated the cache-blocking
parameters.
- Removed redundant BLIS object creation and its contingencies
in the native micro-kernel testing interface(for complex types).
Added the required unit-tests for memory and functionality
checks of the new kernels.
AMD-Interal: [CPUPL-6498]
Change-Id: I520ff17dba4c2f9bc277bf33ba9ab4384408ffe1
Optionally enable parallelism inside gtestsuite so that we can
check BLIS functions perform correctly when nested parallelism
is in operation. Enable with:
cmake ... -DOPENMP_NESTED={0,1,2,1diff}
where in gtestsuite
- 0 is the default choice with no parallelism.
- 1 and 2 are simple nested parallelism.
- 1diff has one level of parallelism setting different numbers
of threads to be used by BLIS and reference library calls
from each gtestsuite thread.
Note: OMP_NUM_THREADS must be set appropriately to enable or
disable parallelism at each level in the test programs
as desired.
OMP_NUM_THREADS will also define the parallelism used
within the BLIS library (if it is multithreaded), unless
BLIS-specific ways of specifying parallelism have been
used. If a BLIS-specific parallelism option has been set,
the same mechanism will be used in the 1diff option to
vary the number of threads in BLIS per application thread.
AMD-Internal: [CPUPL-3902]
Change-Id: I89f9edb4125c64ef03e025a9f6ccb84960ba8771
More changes to standardize copyright formatting and correct years
for some files modified in recent commits.
AMD-Internal: [CPUPL-5895]
Change-Id: Ie95d599710c1e0605f14bbf71467ca5f5352af12
- Added 32x3n n-biased kernels to directly handle the cases where n=3
which were earlier being handled by the primary n-biased, 32x8n,
kernel.
- Modified the n-biased fringe kernels to further handle the smaller
m-fringe cases. Thus, now the kernels handle the following range of m
for any value of n:
- 16x8n : m = [16, 31)
- 8x8n : m = [8, 15)
- m_leftx8n : m = [1, 7]
- Updated the function pointer map for n-biased kernels with added
granularity to invoke the smaller fringe cases directly on the basis
of m-dimension.
- Added micro-kernel unit tests for all the dgemv_n kernels.
AMD-Internal: [CPUPL-6231]
Change-Id: Ibe88848c2c1bbb65b3e79fbc90a2800dc15f5119
- Added new ZTRSM kernels for right and left variants.
- Kernel dimensions are 12x4.
- 12x4 ZGEMM SUP kernels are used internally
for solving GEMM subproblem.
- These kernels do not support conjugate transpose.
- Only column major inputs are supported.
- Tuned thresholds to pick efficent code path for ZEN5.
AMD-Internal: [CPUPL-6356]
Change-Id: I33ba3d337b0fcd972ca9cfe4668cb23d2b279b6e
- Reduced the blocking size of 'bli_ddotv_zen_int10'
kernel from 40 elements to 20 elements for better
utilization of vector registers
- Replaced redundant 'for' loops in 'bli_ddotv_zen_int10'
kernel with 'if' conditions to handle reminder
iterations. As only a single iteration is used when
reminder is less than the primary unroll factor.
- Added a conditional check to invoke the vectorized
DDOTV kernels directly(fast-path), without incurring
any additional framework overhead.
- The fast-path is taken when the input size is ideal
for single-threaded execution. Thus, we avoid the
call to bli_nthreads_l1() function to set the ideal
number of threads.
- Updated getestsuite ukr tests for 'bli_ddotv_zen_int10'
kernel.
AMD-Internal: [CPUPL-4877]
Change-Id: If43f0fcff1c5b1563ad233005717398b5b6fb8f2
In case the executable to obtain the BLIS library version fails,
catch and report common errors to help with debugging.
Also correct the test for bli_info_get_info() support to mark
that it is not available in any AOCL version <= 4.1
AMD-Internal: [CPUPL-4500]
Change-Id: Ie8f728b49faa60e0469562dbf77d67f86b415cd8
- As part of AOCL-BLAS, there exists a set of vectorized
SUP kernels for GEMM, that are performant when invoked
in a bare-metal fashion.
- Designed a macro-based interface for handling tiny
sizes in GEMM, that would utilize there kernels. This
is currently instantiated for 'Z' datatype(double-precision
complex).
- Design breakdown :
- Tiny path requires the usage of AVX2 and/or AVX512
SUP kernels, based on the micro-architecture. The
decision logic for invoking tiny-path is specific
to the micro-architecture. These thresholds are defined
in their respective configuration directories(header files).
- List of AVX2/AVX512 SUP kernels(lookup table), and their
lookup functions are defined in the base-architecture from
which the support starts. Since we need to support backward
compatibility when defining the lookup table/functions, they
are present in the kernels folder(base-architecture).
- Defined a new type to be used to create the lookup table and its
entries. This type holds the kernel pointer, blocking dimensions
and the storage preference.
- This design would only require the appropriate thresholds and
the associated lookup table to be defined for the other datatypes
and micro-architecture support. Thus, is it extensible.
- NOTE : The SUP kernels that are listed for Tiny GEMM are m-var
kernels. Thus, the blocking in framework is done accordingly.
In case of adding the support for n-var, the variant
information could be encoded in the object definition.
- Added test-cases to validate the interface for functionality(API
level tests). Also added exception value tests, which have been
disabled due to the SUP kernel optimizations.
AMD-Internal: [CPUPL-6040][CPUPL-6018][CPUPL-5319][CPUPL-3799]
Change-Id: I84f734f8e683c90efa63f2fa79d2c03484e07956
- Added support for gemv kernels unit test in gtestsuite.
- Added micro-kernel tests and memory tests for DGEMV
transpose case kernels.
AMD-Internal: [CPUPL-5835]
Change-Id: I7d2d3cdbfea436f6c9b2cce9f2e85bfc5c51f201
- Updated the existing AVX2 {S/D}AMAXV kernels to comply
to the standard when having exception values. This makes
it exhibit the same behaviour as it AVX512 variants.
Provided additional optimizations with loop unrolling.
- Removed redundant early return checks inside the kernels,
since they have been abstracted to a higher layer.
- Updated the unit-tests(micro-kernel) and exception value
tests for appropriate code-coverage. Also re-enabled the
exception value tests.
AMD-Internal: [CPUPL-4745]
Change-Id: I36c793220bd4977a00281af9737c51cd1e5c60d9
- Restore test for old bli_dgemm_zen4_asm_8x24 kernel, so that
we can test this if linking with older AOCL versions.
- Move K_bli_dgemm_avx512_asm_8x24 definition from AOCL_42 list
to AOCL_50 list.
AMD-Internal: [CPUPL-4500]
Change-Id: Id522f4bc5b89e86f77c4e1d26c75e261736ab450
- Bug : The current {S/D}AMAXV AVX512 kernels produced an
incorrect functionality with multiple absolute maximums.
They returned the last index when having multiple occurences,
instead of the first one.
- Implemented a bug-fix to handle this issue on these AVX512
kernels. Also ensured that the kernels are compliant with
the standard when handling exception values.
- Further optimized the code by decoupling the logic to find
the maximum element and its search space for index. This way,
we use lesser latency instructions to compute the maximum
first.
- Updated the unit-tests, exception value tests and early return
tests for the API to ensure code-coverage.
AMD-Internal: [CPUPL-4745]
Change-Id: I2f44d33dbaf89fe19e255af1f934877816940c6f
This patch introduces comprehensive optimizations to the DGEMM kernel, focusing on loop
efficiency and edge kernel performance. The following technical improvements have been implemented:
1. **IR Loop Optimization:**
- The IR loop has been re-implemented in hand-written assembly to eliminate the overhead associated
with `begin_asm` and `end_asm` calls, resulting in more efficient execution.
2. **JR Loop Integration:**
- The JR loop is now incorporated into the micro kernel. This integration avoids the repetitive overhead
of stack frame management for each JR iteration, thereby enhancing loop performance.
3. **Kernel Decomposition Strategy:**
- The m dimension is decomposed into specific sizes: 20, 18, 17, 16, 12, 11, 10, 9, 8, 4, 2, and 1.
- For remaining cases, masked variants of edge kernels are utilized to handle the decomposition efficiently.
1. **Interleaved Scaling by Alpha:**
- Scaling by the alpha factor is interleaved with load instructions to optimize the instruction pipeline
and reduce latency.
2. **Efficient Mask Preparation:**
- Masks are prepared within inline assembly code only at points where masked load-store operations are necessary,
minimizing unnecessary overhead.
3. **Broadcast Instruction Optimization:**
- In edge kernels where each FMA (Fused Multiply-Add) operation requires a broadcast without subsequent reuse,
the broadcast instruction is replaced with `mem_1to8`.
- This allows the compiler to optimize by assigning separate vector registers for broadcasting, thus avoiding
dependency chains and improving execution efficiency.
4. **C Matrix Update Optimization:**
- During the update of the C matrix in edge kernels, columns are pre-loaded into multiple vector registers.
This approach breaks dependency chains during FMA operations following the scaling by alpha, thereby mitigating
performance bottlenecks and enhancing throughput.
These optimizations collectively improve the performance of the DGEMM kernel, particularly in handling edge cases and
reducing overhead in critical loops. The changes are expected to yield significant performance gains in matrix multiplication
operations.
This patch also involves changes for tiny gemm interface. A light
interface for calling kernels and removing calls to avx2 dgemm kernels
as we use avx512 dgemm kernels for all the sizes for zen4 and zen5.
For zen4 and zen5 when A matrix transposed(CRC, RRC), tiny kernel does not have
the support to handle such inputs and thus such inputs are routed to
gemm_small path.
AMD-Internal: [CPUPL-6054]
Change-Id: I57b430f9969ca39aa111b54fa169e4225b900c4a
- AVX512 specific DGEMV native kernels are added for Zen4/5
architectures to handle the NO_TRANSPOSE cases and are independent of
the AXPYF fused kernels.
- The following set of kernels biased towards the n-dimension perform
beta scaling of y vector within the kernel itself and handle cases
where n is less than 5:
- bli_dgemv_n_zen_int_32x8n_avx512( ... )
- bli_dgemv_n_zen_int_32x4n_avx512( ... )
- bli_dgemv_n_zen_int_32x2n_avx512( ... )
- bli_dgemv_n_zen_int_32x1n_avx512( ... )
- The bli_dgemv_n_zen_int_16mx8_avx512( ... ) is biased towards the
m-dimension and for this kernel beta scaling is handled beforehand
within the framework.
- Added unit-tests for the new kernels.
- AVX2 path for Zen/2/3 architectures still follows the old approach of
using fused kernel, namely AXPYF, to perform the GEMV operation.
AMD-Internal: [CPUPL-5560]
Change-Id: I22bc2a865cd28b9cdcb383e17d1ff38bdd28de79
