Commit Graph

3645 Commits

Author SHA1 Message Date
Deepak Negi
182a6373b5 Added support to specify bias data type in u8s8s32/s8s8s32 API's
Description:
1. The bias type was supported only based on output data type.
2. The option is added in the pre-ops structure to select the bias data
   type(s8/s32/bf16) irrespective of the storage data type in
   u8s8s32/s8s8s32 API's.

AMD-Internal: SWLCSG-3302

Change-Id: I3c465fe428672d2d58c1c60115c46d2d5b11f0f4
2025-01-15 05:56:26 -05:00
Arnav Sharma
2f2741f4ab Fixed Bug in test_scalv
- Since bli_obj_length(&a) is being used to get the length of A vector,
  we need to initialize A as a vector of M x 1 dimension as the
  bli_obj_length(...) will return the M dimension of the object.
- The vector was being initialized with a dimension of 1 x N resulting
  in bli_obj_length(&a) always returning 1 as the length of the vector.

AMD-Internal: [CPUPL-6297]
Change-Id: Id0e79752f9b81c1573deda3dd32ef0fef10df50c
2025-01-14 23:53:53 -05:00
Vignesh Balasubramanian
a80436ab21 Standardizing the EVT compliance of {S/D}AMAXV API
- Updated the existing AVX2 {S/D}AMAXV kernels to comply
  to the standard when having exception values. This makes
  it exhibit the same behaviour as it AVX512 variants.
  Provided additional optimizations with loop unrolling.

- Removed redundant early return checks inside the kernels,
  since they have been abstracted to a higher layer.

- Updated the unit-tests(micro-kernel) and exception value
  tests for appropriate code-coverage. Also re-enabled the
  exception value tests.

AMD-Internal: [CPUPL-4745]
Change-Id: I36c793220bd4977a00281af9737c51cd1e5c60d9
2025-01-13 06:56:31 -05:00
harsh dave
7510e27007 DGEMM Optimizations
Refined thresholds to decide between native and sup DGEMM code-paths for both zen4 and zen5 processors.

AMD-Internal: [CPUPL-6300]
Change-Id: Ib32a256dba99a0a92b7ecaa7684443a66c459566
2025-01-13 01:09:39 -05:00
Edward Smyth
0ae5a0492f GTestSuite: fix to ukr tests for dgemm avx512 8x24 kernels
- Restore test for old bli_dgemm_zen4_asm_8x24 kernel, so that
  we can test this if linking with older AOCL versions.
- Move K_bli_dgemm_avx512_asm_8x24 definition from AOCL_42 list
  to AOCL_50 list.

AMD-Internal: [CPUPL-4500]
Change-Id: Id522f4bc5b89e86f77c4e1d26c75e261736ab450
2025-01-10 12:33:15 -05:00
Vignesh Balasubramanian
8e660215c3 Introduced fast-path in DAXPYV API
- Added a conditional check to invoke the vectorized
  DAXPYV kernels directly(fast-path), without incurring
  any additional framework overhead.

- The fast-path is taken when the input size is ideal for
  single-threaded execution. Thus, we avoid the call to
  bli_nthreads_l1() function to set the ideal number of threads.

AMD-Internal: [CPUPL-4878]
Change-Id: I001fd1b8bbd2d691ecb3e2423ec7998e130850bb
2025-01-10 09:19:38 -05:00
Vignesh Balasubramanian
345204d69b Additional updates to the thresholds for ZGEMM small path
- Further updated the thresholds for entry to ZGEMM small
  path(AVX2), when the execution is mulithreaded. The newer
  thresholds account for more skinnier inputs, compatible with
  single-threaded small path, as opposed to multithreaded
  SUP path.

AMD-Internal: [CPUPL-6040][CPUPL-5930]
Change-Id: I333f97d8af49733310e4ae48b12baba15ef828d6
2025-01-10 08:29:31 -05:00
Edward Smyth
97ede96ed4 Correct duplicate object file names
Some kernel file names were the same for different sub-configurations,
which could result in duplicate copies of the same object being archived
depending upon the order of (re-)compiling the source files. Rename the
files to be specific to each sub-configuration to avoid this problem.

AMD-Internal: [CPUPL-5895]
Change-Id: I182ac706e04a364f1df20fd0fb5b633eb10eeafb
2025-01-10 06:03:36 -05:00
Mithun Mohan
7a25505f5c Simulation of spread like pattern in worker thread to core binding in LPGEMM.
-In multi-threaded cases if a packed/close pattern thread to core
binding is used (e.g.: OMP_PROC_BIND=close and OMP_PLACES=core|threads),
LPGEMM (OMP framework) launches threads such that threads with adjacent
id's are bound to nearby (even adjacent) cores. Depending on the
processor architecture, multiple threads with adjacent id's can be bound
to cores sharing the same last level cache. However it was observed that
when these threads (with adjacent id's) access the B reorder buffer, the
last level cache access was suboptimal. This can be attributed to the
per thread reorder buffer block accesses and how it maps to the last
level cache.
-In these cases, m is small (<= 4 * MR) and n value is such that number
of NR blocks (n/NR) is less than available threads nt (like < 0.5 * nt).
In such cases, id's of the threads can be modified such that the number
of threads with adjacent id's bound to the last level cache can be
reduced. This looks similar to the spread pattern used in thread to core
binding. This reduces the load on last level cache due to reorder buffer
access and improves performance in these cases. A heuristic method is
used to detect whether thread to core binding follows close pattern
before applying the thread id modifications.

AMD-Internal: [SWLCSG-3185]
Change-Id: Ie3c87d56e0f7b59161a381f382cf4e2d5d02a591
2025-01-10 06:02:06 -05:00
Edward Smyth
567039a7fe Fortran interfaces for bli_thread_get APIs
Create and export Fortran interfaces for bli_thread_get_num_threads()
and bli_thread_get_{jc,pc,ic,jr,ir}_nt() APIs.

bli_thread_get_is_parallel() is intended for internal BLIS usage, so
not adding a Fortran interfaces for it at this time.

AMD-Internal: [CPUPL-6168]
Change-Id: Ieba2537e5455cc289536aec3de5d4b5866e607f1
2025-01-10 05:07:33 -05:00
Kiran Varaganti
eb8efd742d Added guard for including omp.h
aoclos.c:20 is #include <omp.h> but this needs to be guarded by e.g. #ifdef BLIS_ENABLE_OPENMP
otherwise it leads to compilation failure if the environment does not have OpenMP available.
(affected platform: clang 20 on ubuntu 24.04LTS)
Issue reported in https://github.com/amd/blis/issues/25, thanks to maychiew1988.

Change-Id: I4ea0b85f8194345f0534e17229acb4827193dfe6
2025-01-10 04:54:34 -05:00
Meghana Vankadari
852cdc6a9a Implemented batch_matmul for f32 & int8 datatypes
Details:
- The batch matmul performs a series of matmuls, processing
  more than one GEMM problem at once.
- Introduced a new parameter called batch_size for the user
  to indicate number of GEMM problems in a batch/group.
- This operation supports processing GEMM problems with
  different parameters including dims,post-ops,stor-schemes etc.,
- This operation is optimized for problems where all the
  GEMMs in a batch are of same size and shape.
- For now, the threads are distributed among different GEMM
  problems equally irrespective of their dimensions which
  leads to better performance for batches with identical GEMMs
  but performs sub-optimally for batches with non-identical GEMMs.
- Optimizations for batches with non-identical GEMMs is in progress.
- Added bench and input files for batch_matmul.
- Added logger functionality for batch_matmul APIs.

AMD-Internal: [SWLCSG-2944]
Change-Id: I83e26c1f30a5dd5a31139f6706ac74be0aa6bd9a
2025-01-10 04:10:53 -05:00
Mithun Mohan
ef4286a97e Multi-data type buffer and scale support for matrix add|mul post-ops in s32 API.
-As it stands the buffer type in matrix add|mul post-ops is expected to
be the same as that of the output C matrix type. This limitation is now
removed and user can specify the buffer type by setting the stor_type
attribute in add|mul post-op struct. As of now int8, int32, bfloat16 and
float types are supported for the buffer in s32 micro-kernels. The same
support is also added for bf16 micro-kernels, with bfloat16 and float
supported for now.
-Additionally the values (from buffer) are added/multiplied as is to the
output registers while performing the matrix add|mul post-ops. Support
is added for scaling these values before using them in the post-ops.
Both scalar and vector scale_factors are supported.
-The bias_stor_type attribute is renamed to stor_type in bias post-ops.

AMD-Internal: [SWLCSG-3319]
Change-Id: I4046ab84481b02c55a71ebb7038e38aec840c0fa
2025-01-10 02:11:12 -05:00
Meghana Vankadari
051c9ac7a2 Bug fixes in F32 and INT8 APIs
Details:
- Fixed few bugs in downscale post-op for f32 datatype.
- Fixed a bug in setting strides of packB buffer in
  int8 APIs.

Change-Id: Idb3019cc4593eace3bd5475dd1463dea32dbe75c
2025-01-09 04:07:26 -05:00
varshav
7b9d29f9b3 Adding post-ops for JIT kernels
- Added Downscale, tanh and sigmoid post-op support to the JIT kernels
 - Mask bf16s4 kernel call while JIT kernels are enabled to avoid compile-time error.
 - Added the optional support for B-prefetch in the JIT kernels
 - Resolved the visibility issues in global variable jit_krnels_generated
 - Modified the array generation for scale and zp values in the bench

Change-Id: I09b8afc843f51ac23645e02f210a2c13d3af804d
2025-01-08 12:55:27 +00:00
Mithun Mohan
4a95f44d39 Buffer scale support for matrix add and matrix mul post-ops in bf16 API.
-Currently the values (from buffer) are added/multiplied as is to the
output registers while performing the matrix add/mul post-ops. Support
is added for scaling these values before using them in the post-ops.
Both scalar and vector scale_factors are supported.

AMD-Internal: [SWLCSG-3181]
Change-Id: Ifdb7160a1ea4f5ecccfa3ef31ecfed432898c14d
2025-01-08 10:35:50 +00:00
harsh dave
aeda581539 Fix: Rename bli_tiny_gemm.c to bli_tiny_gemm_amd.c
When compiling with config generic (or any non-zen build),
the bli_dgemm_tiny_6x8 kernel is not defined. Since bli_dgemm_tiny()
is only used within amd specific file, bli_tiny_gemm.c has been renamed
to bli_tiny_gemm_amd.c to reflect its specific usage.

Thanks to Smyth, Edward<edward.smyth@amd.com> for identifying and helping to fix the issue.

Change-Id: If5d134aeba6d30d0a51e6d7d6fa9b3c4450a3307
2025-01-07 23:29:32 -05:00
Vignesh Balasubramanian
cdaa2ac7fd Bugfix and optimizations for AVX512 AMAXV micro-kernels
- Bug : The current {S/D}AMAXV AVX512 kernels produced an
  incorrect functionality with multiple absolute maximums.
  They returned the last index when having multiple occurences,
  instead of the first one.

- Implemented a bug-fix to handle this issue on these AVX512
  kernels. Also ensured that the kernels are compliant with
  the standard when handling exception values.

- Further optimized the code by decoupling the logic to find
  the maximum element and its search space for index. This way,
  we use lesser latency instructions to compute the maximum
  first.

- Updated the unit-tests, exception value tests and early return
  tests for the API to ensure code-coverage.

AMD-Internal: [CPUPL-4745]
Change-Id: I2f44d33dbaf89fe19e255af1f934877816940c6f
2025-01-07 22:56:20 +05:30
Vignesh Balasubramanian
f548f42607 Fixing compiler warnings on ZGEMM
- Scoped some of the variables used in zgemm_blis_impl()
  when determining the thresholds to small path. These
  variables will be used only when the architecture is
  ZEN5 or ZEN4.

AMD-Internal: [CPUPL-5895]
Change-Id: I6f90856f34454423ac777e33c74fe5ec6bb94e13
2025-01-07 10:59:43 +05:30
Meghana Vankadari
c9f0240679 Returning early for col-major inputs in u8s8s32os32|s8 APIs
Details:
- For u8s8s32os32|s8 APIs, A & B matrices are of different
  datatypes. Hence col-major inputs cannot be supported by
  swapping the matrices internally. Added a check to return
  early in such cases.

Change-Id: I99fbebe811c3d05310f30f7fc978f5084b5a51ba
AOCL-Jan2025-b1
2025-01-05 23:46:06 +05:30
harsh dave
ea4212c550 Increase buffer size to prevent segmentation fault
Since the threshold for tiny path was large but the buffer size was
not enough to store the complete packed matrix. That is leading to
segmentation faults.

This commit fix the buffer size as per the threshold of tiny gemm path.
With the corrected buffer size, the matrix is packed correctly.

AMD-Internal: [CPUPL-6201]
Change-Id: I0292a07f6146e7f1ccd8c1010b4c41c218fd9b47
2025-01-03 06:39:50 -05:00
Mithun Mohan
8d8a8e2f19 Light-weight logging framewok for LPGEMM.
-A light-weight mechanism/framework to log input details and a
stringified version of the post-ops structure is added to LPGEMM.
Additionally the runtime of the API is also logged.
The logging framework logs to a file with filename following the format
aocl_gemm_log_<PID>_<TID>.txt.
-To enable this feature, the AOCL_LPGEMM_LOGGER_SUPPORT=1 macro needs to
be defined when compiling BLIS (with aocl_gemm addon enabled) by passing
CFLAGS="-DAOCL_LPGEMM_LOGGER_SUPPORT=1" to ./configure. Additionally
AOCL_ENABLE_LPGEMM_LOGGER=1 has to be exported in the environment during
LPGEMM runtime.

AMD-Internal: [SWLCSG-3280]
Change-Id: I30bfb35b2dc412df70044601b335938fc9f49cfb
2025-01-03 11:28:57 +00:00
Nallani Bhaskar
6cb1acf3c3 Fixed out-of-memory read access in bf16 reorder reference
Description:

Loop count was taken as 16 instead of n0_partial_rem in packb_nrlt16_bf16bf16f32of32_col_major_ref function.

Updated comments on reference reorder functionality.

AMD Internal: SWLCSG-3279

Change-Id: Idfc3b92906bc2b24651c7923e395fe10db56166b
2025-01-03 04:09:08 -05:00
Meghana Vankadari
bfc512d3e1 Implemented batch_gemm for bf16bf16f32of32|bf16
Details:
- The batch matmul performs a series of matmuls, processing
  more than one GEMM problem at once.
- Introduced a new parameter called batch_size for the user
  to indicate number of GEMM problems in a batch/group.
- This operation supports processing GEMM problems with
  different parameters including dims,post-ops,stor-schemes etc.,
- This operation is optimized for problems where all the
  GEMMs in a batch are of same size and shape.
- For now, the threads are distributed among different GEMM
  problems equally irrespective of their dimensions which
  leads to better performance for batches with identical GEMMs
  but performs sub-optimally for batches with non-identical GEMMs.
- Optimizations for batches with non-identical GEMMs is in progress.
- Added bench and input files for batch_matmul.

AMD-Internal: [SWLCSG-2944]
Change-Id: Idc59db5b8c5794bf19f6f86bcb8455cd2599c155
2025-01-03 03:28:32 -05:00
Shubham Sharma
8f99d8a5bb Fixed warnings and compilation issues with GCC in TRSM
- Current implementation uses macros to expand the code at
  compile time, but this is causing some false warning in GCC12 and 14.
- Added switch case in trsm right variants for n_remainder.
- This ensures that n_rem is compile time constant, therefore
   warnings related to array subscript out of bounds are fixed.
- mtune=znver3 flag is causing compilation issue in GCC 9.1,
  therefore this flag is removed.
- Remaned the file bli_trsm_small to bli_trsm_small_zen5 in order
  to avoid possibily of missing symbols.

AMD-Internal: [CPUPL-6199]
Change-Id: Ib8e90196ce0a41d38c2b29226df5ab6c2d8ba996
2024-12-18 06:22:05 -05:00
Edward Smyth
4ce708c316 Move some BLAS extension APIs to extra subdirectories
In preparation for merging next group of changes from upstream BLIS,
move some BLAS extension APIs to new extra subdirectories in
frame/compat and frame/compat/cblas/src. Other extension APIs will
be moved in later commits.

Some tidying up to better match upstream BLIS code has also been done.

AMD-Internal: [CPUPL-2698]
Change-Id: I0780a775d37242fba562c3f13666da0ad2b2cdfb
2024-12-17 04:54:39 -05:00
Shubham Sharma
050e5a382f Fixed warning for GCC 12+
- Warnings in DTRSM  kernel caused by uninitialized registers
   and extra loop unroll is fixed.
- Warning in DGEMM kernel caused by extra space is fixed.

Change-Id: I1d9cfaa0b2847f5fdbe8b343a462d67a3aca0819
2024-12-17 01:44:41 -05:00
Nallani Bhaskar
40719e0438 Implemented reference function for bf16 reorder function
Description:

Implemented a reference version for
aocl_gemm_reorder_bf16bf16f32of32 function
to make the code cpu architecture independent.

AMD-Internal: [ SWLCSG-3279 ]

Change-Id: I0c715864c0ab3e5afea2ee6ee9546b75c3dbf9ec
2024-12-17 05:46:39 +00:00
Hari Govind S
a028108cbb Optimisation of AOCL-dynamic for dotv API
-  Adding AOCL-dynamic logic for dotv on zen4
   and zen5 architecture.

AMD-Internal: [CPUPL-4877]
Change-Id: I421ad5e48cf001d1fc5c13e9f1cbbf4db0bf5f37
2024-12-16 22:09:59 -05:00
Edward Smyth
0c6d006225 Changes to rntm to reduce mutex operations
Change usage of global_rntm and tl_rntm to elimate need
for mutex operations when accessing global_rntm. Usage of
these data structures is now as follows:
* global_rntm is set once during bli_init_apis and includes
  all getenv calls to check BLIS threading and error printing
  environment variables. global_rntm is then read-only.
* tl_rntm is intialized once from global_rntm on each
  application thread. Any calls to BLIS set threading/ways
  APIs will update tl_rntm for that application thread only
  (Previously they updated global_rntm for all application threads).
* Re-initialize info_value in tl_rntm in every call to bli_init APIs.
* In bli_rntm_init_from_global() we initialize the local (per API
  call) rntm as a copy of tl_rntm and then update threading values
  in bli_thread_update_rntm_from_env() to reflect the current status
  of OpenMP runtime ICVs.

AMD-Internal: [CPUPL-6168][SWLCSG-3143]
Change-Id: Ib9387ee2b51f507ed08cc38267057109acea14a6
2024-12-16 04:45:26 -05:00
Shubham Sharma
beaea1b88f Added new DTRSM small code path for ZEN5
- Added new DTRSM kernels for right  and left variants.
- Kernel dimensions are 24x8.
- 24x8 DGEMM SUP kernels are used internally
  for solving GEMM subproblem.
- Tuned thresholds to pick efficent code path for ZEN5.

AMD-Internal: [CPUPL-6016]
Change-Id: I743d6dc47717952c2913085c0db3454ae9d046db
2024-12-16 10:38:45 +05:30
Vignesh Balasubramanian
609af9bfe2 Threshold tuning for ZGEMM small path
- Updated the threshold check for ZGEMM small path to include
  runtime checks for redirection, specific to the micro-architecture.

- The current ZGEMM small path has only its AVX2 variant available.
  Post implementing an AVX512(same/different algorithm), the thresholds
  will further be fine-tuned.

- Included the dot-product based AVX512 ZGEMM kernels in the ZEN5
  context. It will be used as part of handling RRC and CRC storage
  schemes of C, A and B matrices in both single-thread and multi-thread
  runs.

AMD-Internal: [CPUPL-5949]
Change-Id: Ic8b7cf0e00b7c477f748669f160c4b01df995c75
2024-12-13 12:51:22 -05:00
harsdave
7813938f70 Support DGEMM Computation for Transposed A Matrix with CRC and RRC Storage Scheme
- This patch introduces changes to support DGEMM computation when the input matrix A is transposed.

- The changes accommodate CRC (Column-Row-Column) and RRC (Row-Row-Column) storage schemes for matrices
  C, A, and B. The primary goal is to pack the A matrix in a column-stored scheme, enabling the re-use
  of the DGEMM SUP kernel for efficient computation.

- Performance is better when BLIS_PACK_BUFFER macro is set to 0.
  By default, it is set to 1[enabled].

AMD-Internal: [CPUPL-6054]
Change-Id: I543a84b05c9e6380bc03017ab6da685e7006a64e
2024-12-13 05:19:40 -05:00
harsdave
54b46ec1ed Enhance 24x8 DGEMM SUP/Tiny Kernel Performance with Optimized Loops and Edge Kernels
This patch introduces comprehensive optimizations to the DGEMM kernel, focusing on loop
efficiency and edge kernel performance. The following technical improvements have been implemented:

1. **IR Loop Optimization:**
   - The IR loop has been re-implemented in hand-written assembly to eliminate the overhead associated
     with `begin_asm` and `end_asm` calls, resulting in more efficient execution.

2. **JR Loop Integration:**
   - The JR loop is now incorporated into the micro kernel. This integration avoids the repetitive overhead
     of stack frame management for each JR iteration, thereby enhancing loop performance.

3. **Kernel Decomposition Strategy:**
   - The m dimension is decomposed into specific sizes: 20, 18, 17, 16, 12, 11, 10, 9, 8, 4, 2, and 1.
   - For remaining cases, masked variants of edge kernels are utilized to handle the decomposition efficiently.

1. **Interleaved Scaling by Alpha:**
   - Scaling by the alpha factor is interleaved with load instructions to optimize the instruction pipeline
     and reduce latency.

2. **Efficient Mask Preparation:**
   - Masks are prepared within inline assembly code only at points where masked load-store operations are necessary,
     minimizing unnecessary overhead.

3. **Broadcast Instruction Optimization:**
   - In edge kernels where each FMA (Fused Multiply-Add) operation requires a broadcast without subsequent reuse,
     the broadcast instruction is replaced with `mem_1to8`.
   - This allows the compiler to optimize by assigning separate vector registers for broadcasting, thus avoiding
     dependency chains and improving execution efficiency.

4. **C Matrix Update Optimization:**
   - During the update of the C matrix in edge kernels, columns are pre-loaded into multiple vector registers.
     This approach breaks dependency chains during FMA operations following the scaling by alpha, thereby mitigating
     performance bottlenecks and enhancing throughput.

These optimizations collectively improve the performance of the DGEMM kernel, particularly in handling edge cases and
reducing overhead in critical loops. The changes are expected to yield significant performance gains in matrix multiplication
operations.

This patch also involves changes for tiny gemm interface. A light
interface for calling kernels and removing calls to avx2 dgemm kernels
as we use avx512 dgemm kernels for all the sizes for zen4 and zen5.

For zen4 and zen5 when A matrix transposed(CRC, RRC), tiny kernel does not have
the support to handle such inputs and thus such inputs are routed to
gemm_small path.

AMD-Internal: [CPUPL-6054]
Change-Id: I57b430f9969ca39aa111b54fa169e4225b900c4a
2024-12-13 00:03:00 -05:00
Edward Smyth
5c946e10dc Always define function bli_nthreads_optimum
bli_nthreads_optimum is exported and called directly by AOCL libFLAME,
however it was only defined if building a multithreaded BLIS library
with AOCL_DYNAMIC enabled. Change to always define this function. If
BLIS is serial or if AOCL_DYNAMIC is disabled, this function returns
without modifying the supplied rntm.

Change-Id: Ie65690e9e6ec2a8ea77b3778f96676a68e6260be
2024-12-12 12:43:43 -05:00
Arnav Sharma
25e59fcbb9 DGEMV Optimizations for NO_TRANSPOSE Cases
- AVX512 specific DGEMV native kernels are added for Zen4/5
  architectures to handle the NO_TRANSPOSE cases and are independent of
  the AXPYF fused kernels.
- The following set of kernels biased towards the n-dimension perform
  beta scaling of y vector within the kernel itself and handle cases
  where n is less than 5:
    - bli_dgemv_n_zen_int_32x8n_avx512( ... )
    - bli_dgemv_n_zen_int_32x4n_avx512( ... )
    - bli_dgemv_n_zen_int_32x2n_avx512( ... )
    - bli_dgemv_n_zen_int_32x1n_avx512( ... )
- The bli_dgemv_n_zen_int_16mx8_avx512( ... ) is biased towards the
  m-dimension and for this kernel beta scaling is handled beforehand
  within the framework.
- Added unit-tests for the new kernels.
- AVX2 path for Zen/2/3 architectures still follows the old approach of
  using fused kernel, namely AXPYF, to perform the GEMV operation.

AMD-Internal: [CPUPL-5560]
Change-Id: I22bc2a865cd28b9cdcb383e17d1ff38bdd28de79
2024-12-12 10:26:50 -05:00
Deepak Negi
615789e196 Fixed compilation issue with clang 18 on windows
Description
-In enum AOCL_PARAMS_STORAGE_TYPES the member FLOAT was declared and the
 clang 18 compiler in msvc throwing issue with multiple definition. We
 replace FLOAT and BFLOAT16 to AOCL_GEMM_<F32/BF16>.

AMD-Internal: CPUPL-6174

Change-Id: Ic061af068854d51629b82b495efd0eb54543f329
2024-12-12 06:37:06 -05:00
Vignesh Balasubramanian
da6e9defcb Dynamic selection of AVX2 or AVX512 DNRM2 kernels
- Added a kernel selection logic based on the input
  dimension(runtime parameter), to choose between
  deploying AVX2 or AVX512 computational kernel for
  single-thread execution.

- An empirical analysis was conducted to arrive at the
  thresholds, for ZEN4 and ZEN5 architectures.

- Updated the fast-path threshold for ZEN4 to be in hand
  with the tipping points of its dynamic thread-setter(used
  when AOCL_DYNAMIC is enabled).

AMD-Internal: [CPUPL-5937]
Change-Id: I96d7f167658c9e25a0098c4c67e12e4ba673e228
2024-12-10 10:53:54 +05:30
Deepak Negi
baeebe75c9 Support for standard AutoAWQ storage format.
Description:
1. AutoAWQ use a int32 buffer to store 8 elements each of 4 bits in this
   format [0, 2, 4, 6, 1, 3, 5, 7].
2. Support is added to convert above format back to the original
   sequential order [0, 1, 2, 3, 4, 5, 6, 7] before reordering
   in the AWQ API.

AMD-Internal: SWLCSG-3169

Change-Id: I5395766060c200ab81d0b8be94356678a169ac13
2024-12-02 04:02:27 -05:00
Meghana Vankadari
fbb72d047f Added group quantization and zero-point support for WOQ kernels
Description:

1. Added group quantization and zero-point (zp) in
   aocl_gemm_bf16s4f32o<bf16|f32> API.

2. Group quantization is technique to improve accuracy
   where scale factors to dequantize weights varies at group
   level instead of per channel and per tensor level.

3. Added zp and scaling in woq packb kernels so that for
   large M values zp and scaling are performed at pack-b
   stage and bf16 kernels are called

4. Adding zp support and scaling to default path in WoQ kernels
   created some performance overhead when M value is very small.

5. Added string group_size to lpgemm bench to read
   group size from bench_input.txt and tested for
   various combinations of matrix dimensions.

6. The scalefactors could be of type float or bf16
   and the  zeropoint values are expected to be
   in int8 format.

AMD-Internal: [SWLCSG-3168, SWLCSG-3172]

Change-Id: Iff07b54d76edc7408eb2ea0b29ce8b4a04a38f57
2024-12-02 06:46:13 +00:00
Shubham Sharma.
be6fbadd95 BlockSize Tuning for ZEN4 and ZEN5
- Enabled dynamic blocksizes for DGEMM in ZEN4 and ZEN5 systems.
- MC, KC and NC are dynamically selected at runtime for DGEMM native.
- A local copy of cntx is created and blocksizes are updated in the local cntx.
- Updated threshold for picking DGEMM SUP kernel for ZEN4.

AMD-Internal: [CPUPL-5912]
Change-Id: Ic12a1a48bfa59af26cc17ccfa47a2a33fadde1f6
2024-11-29 03:19:16 -05:00
Shubham Sharma
f2320a1fef Enabled DGEMM row major kernel for ZEN4
- Merged ZEN4 and ZEN5 DGEMM 8x24 kernel.
- Replaced 32x6 kernel with 8x24. Now same
  kernel is used for ZEN4 and ZEN5.
- Blocksizes have been tuned for genoa only.
- DGEMM kernel for DTRSM native code path
  is replaced with 8x24 kernel.
- Enabled alpha scaling during packing for ZEN4.
- ZEN4 8x24 kernel has been removed.

AMD-Internal: [CPUPL-5912]
Change-Id: I89a16a7e3355af037d21d453aabf53c5ecccb754
2024-11-29 08:18:48 +00:00
Shubham Sharma
082081658f BugFix: Fixed extreme value handling in AVX512 DGEMM kernel
- Extreme values are not handled correctly when beta == 0 and C is
  column major stored.
- For checking if beta is zero, VCOMISD(XMM(1), XMM(2)) is used,
  beta(XMM1) is compared with zero(XMM2),
  for column major C, setting of xmm2 to zero was missed.
- XMM2 is set to zero after the jump to column major stored C code
  is made, this skips the setting of XMM2 to zero for column major
  C.
- This is fixed by setting XMM2 to zero before the column major jump.

AMD-Internal: [CPUPL-5851]
Change-Id: Ic511071fbc82a082fa48a1543c0c7325eaf75cb8
2024-11-29 08:13:57 +00:00
Shubham Sharma.
bc3238e21e BugFixes in ZEN5 DGEMM kernel
- Changed fringe cases to use ZEN5 DGEMM kernel instead
  of ZEN4 kernel.
- ASAN reporting error when RBP is used even when
  -fno-stack-pointer flag is used, therefore replaced RBP
   register with R11 register.
- Added missing RDX register in clobber list which is causing
  failures with AOCC compiler.

Thanks to harsh.dave@amd.com for debugging some of the issues.

AMD-Internal: [CPUPL-5851]
Change-Id: I0ee412c97c9dbfb3e7a736a10bfd93d775779b5b
2024-11-29 00:22:41 -05:00
Shubham Sharma
266bd32dea Enable fringe case handling in DGEMM ZEN5 macro kernel
- Generic kernel is used if N is not multiple of NR
  or M is not multiple of MR.
- This limit the maximum values of NR that can be used.
- Support for fringe case handling is added in DGEMM
  macro kernel so that macro kernel can be used for
  all problem sizes.

AMD-Internal: [CPUPL-5912]
Change-Id: I85c17e91d7511bb35ffed0f346d6ff0376baf62f
2024-11-29 00:22:33 -05:00
Kiran Varaganti
3e2795f406 OpenMP barrier overhead bug fix
In the function bli_thread_update_rntm_from_env()mutex is used for reading global_rntm
"bli_pthread_mutex_lock( &global_rntm_mutex );" This causes regression when application is
Multithreaded. The cause of this regression is due to these mutexes, Imagine a scenario
two threads launched, one thread acquires this mutex, second thread stalls till mutex is
freed by first thread, as a result second thread will be slower to arrive at openmp barrier
in application thereby increasing the openmp barrier overhead.
Things get worst when more number of threads are launched.
Thanks to rocHPL for sharing standalone panelfact application to reproduce this issue.
Thanks to @Edward Symth (edward.smyth@amd.com) for finding this bug.

[SWLCSG-3143]
2024-11-22 15:36:30 +05:30
Deepak Negi
04ae01aeab Added support to specify bias data type in bf16 API's
Description:
1. The bias type was supported only based on output data type.
2. The option is added in the pre-ops structure to select the bias data
   type irrespective of the storage data type in bf16 and WoQ API's


AMD-Internal: SWLCSG-3171


Change-Id: Iac10b946c2d4a5c405b2dc857362be0058615abf
2024-11-19 05:30:02 -05:00
Edward Smyth
971c890fc6 GTestSuite: Select ukr tests by BLIS version
Add definitions in gtestsuite header to list available kernel
by AOCL BLIS version. Check these definitions in ukr test
programs to avoid missing symbol errors when testing with an
older version of BLIS.

Currently AOCL_41, AOCL_42, AOCL_50 and AOCL_DEV are supported,
with AOCL_DEV inferred from the version being later than the
value of AOCL_BLAS_LATEST_VERSION set in CMakeLists.txt. Thanks
to Eleni Vlachopoulou for the cmake functionality to automatically
detect the version from the library.

AMD-Internal: [CPUPL-4500]
Change-Id: I40ffd3d3789324fbb1dabfbf5e1dd4e0c94d54d9
2024-11-15 10:07:29 -05:00
Deepak Negi
60a8c71a1a Sigmoid and Tanh post-operation support for int8 API's.
Description:

Implemented sigmoid, tanh as fused post-ops in
aocl_gemm_<s8|u8>s8<s32|s16>o<s8|u8|s32> API's

Sigmoid(x) = 1/1+e^(-x)
Tanh(x) = (1-e^(-2x))/(1+e^(2x))

Updated bench_lpgemm to recognize sigmod, tanh
as options for post-ops from bench_input and verified.

AMD-Internal: [SWLCSG-3178]

Change-Id: I9df3aab02222f728ff9d1f292c7bc549f30176f0
AOCL-Nov2024-b2
2024-11-15 05:36:31 -05:00
Deepak Negi
146f3b2eb2 Sigmoid and Tanh post-operation support for f32 API.
Description:

Implemented sigmoid, tanh as fused post-ops in
aocl_gemm_f32f32f32of32 API's

Sigmoid(x) = 1/1+e^(-x)
Tanh(x) = (1-e^(-2x))/(1+e^(2x))

Updated bench_lpgemm to recognize sigmod, tanh
as options for post-ops from bench_input and verified.

AMD-Internal: [SWLCSG-3178]

Change-Id: Iac0a907f6dea1d9cb82d9fd8716bfdbf1c33921d
2024-11-15 04:20:20 -04:00