Changes to fix errors and warnings when using gcc 16.1.0:
- Copy changes from 5c2b22da81 in upstream BLIS to extend disabling of
tree-vectorization in affected kernels to gcc 16 and later.
- Remove unused variables in bli_packm_blk_var1_md.c and bli_util_unb_var1.c
to fix warning messages.
Background
bp (base pointer) is the %rbp/%ebp register on x86/x86-64. Inline assembly
kernels in BLIS use asm volatile blocks where they manually manage registers
- including saving and restoring bp themselves to use it as a general-purpose
register for holding loop counters or matrix pointers.
When GCC's tree-vectorizer (specifically the superword-level parallelism (SLP)
pass) runs on a translation unit containing inline asm, it can generate code
that itself needs bp as a frame pointer or in the vectorized prologue/epilogue.
At that point GCC internally marks bp as unavailable and then, when it tries to
compile the inline asm block that also references bp, it throws an error.
As a workaround, disabling tree vectorization for the entire file removes the
conflict - with no vectorizer-generated code, bp stays free for the inline asm.
* Pack B matrix for zgemm conjugate input in SUP path
- B matrix is packed for zgemm input where B matrix is conjugate transpose.
AMD-Internal: CPUPL-8274
* Pack B matrix for zgemm conjugate input in SUP path
- B matrix is packed for zgemm input where B matrix is conjugate transpose.
AMD-Internal: CPUPL-8274
---------
Co-authored-by: harsdave <harsdave@amd.com>
nn, tn, tt, nt, cn, ct, nc, tc.
Previously, the error-handling logic only explicitly checked
for and rejected cc and hh cases. This patch extends that
check to include ch and hc configurations, ensuring they
correctly return a failure/fallback instead of proceeding
with unsupported kernels.
AMD-Internal: [CPUPL-8151]
Co-authored-by: harsdave <harsdave@amd.com>
* ZGEMM SUP: Add conjugate support for AVX-512 kernels on Zen4/Zen5/Zen6
- Add CONJA, CONJB and CONJA_CONJB variants to zgemm SUP micro-tiles
- Enable SUP path for conjugate cases when both are same type
- Unify RRC/CRC storage to use CV kernel variant
- Update SUP dispatch to handle conjugate flags correctly
Note: CONJ_NO_TRANSPOSE + CONJ_NO_TRANSPOSE and
CONJ_TRANSPOSE + CONJ_TRANSPOSE remain unsupported
---------
Co-authored-by: harsdave <harsdave@amd.com>
Add compatibility for OpenMP implementations (e.g., MSVC, older GCC)
that lack functions introduced in OpenMP 3.0 i.e. omp_get_active_level()
and omp_get_max_active_levels(). On these compilers, the tests instead
are based on the older omp_get_nested() functionality.
Thanks to @tony-davis for highlighting this issue.
AMD-Internal: [CPUPL-7303]
CPUPL-7578: New thread control API with global and thread-local variants
Summary: Add new BLIS thread control APIs that provide fine-grained control over threading with proper global and thread-local (TLS) semantics. Fix several correctness issues where set_num_threads() and set_ways() did not properly override each other's state.
New/Modified APIs:
bli_thread_set_num_threads() — Sets thread count globally (updates both global_rntm and tl_rntm)
bli_thread_set_num_threads_local() — Sets thread count for calling thread only (tl_rntm)
bli_thread_get_num_threads() — Returns effective thread count, deriving from ways if set
bli_thread_reset() — Resyncs tl_rntm from global_rntm
bli_thread_set_ways() — Sets loop factorization (jc, pc, ic, jr, ir)
bli_thread_get_is_parallel() — Returns whether parallelism is enabled
bli_thread_get_jc_nt/ic_nt/pc_nt/jr_nt/ir_nt() — Returns individual way values
b77_thread_set_num_threads_local_() — Fortran-compatible wrapper
Bug fixes:
bli_thread_set_num_threads() now clears ways (-1) and sets auto_factor=TRUE on both global_rntm and tl_rntm, so it properly overrides prior BLIS_JC_NT/BLIS_IC_NT environment settings
bli_thread_set_ways() now propagates to global_rntm (inside mutex) and clears stale num_threads on both global_rntm and tl_rntm, so get_num_threads() returns the product of ways instead of a stale value
Fix data race in bli_thread_init_rntm_from_global_rntm() — copy global_rntm under mutex before debug printing
Fix data race in set_num_threads_local() debug print
Test suite (43 tests, 106 assertions):
test_thread_control.c (OpenMP, 23 tests): environment inheritance, global propagation, thread-local isolation, local precedence, per-thread local, reset, nested parallel, edge cases, set_ways, is_parallel, concurrent updates, DGEMM with threads, interleaved settings, persistence, parallel DGEMM, thread pool, reset-to-sync, env ways vs set_num_threads, ways→set_nt→reset, ways→local→reset, round-trip, set_nt→set_ways override, set_ways propagation to new threads
test_thread_control_pthread.c (pthread, 20 tests): equivalent coverage plus concurrent set/reset race condition test, set_nt→set_ways override, set_ways propagation via pthread_create
Files changed (9 files, +2630/-29 lines):
bli_thread.c — Core API implementations and fixes
bli_thread.h — New function declarations
b77_thread.c — Fortran wrapper
test_thread_control.c — OpenMP test suite (23 tests)
test_thread_control_pthread.c — pthread test suite (20 tests)
TEST_THREAD_CONTROL_README.md — Documentation
AMD-Internal: CPUPL-7578
Commit 8310b2d5d3 added new functions and global variables in
blis.h intended only for internal use. These were causing
missing symbol problems when blis.h is included in C
applications as they are not exported from the shared library.
Use BLIS_IS_BUILDING_LIBRARY and BLIS_CONFIGURETIME_CPUID
preprocessor definitions to only expose these when compiling BLIS
and not when using it.
AMD-Internal: [CPUPL-8091]
Implement zen6 cpuid and arch changes, and add zen6 as a
separate BLIS sub-configuration and code path within amdzen
configuration family. Currently all optimization choices are
copies of zen5 sub-configuration.
AMD-Internal: [CPUPL-7162]
Changes to simplify AMD CPUID functionality:
- Variable "features" is limited in size as each bit represents a
specific hardware function. Move detection of FP datapath
width to a separate variable. Also mask the FP datapath bits
explicitly for a more reliable test.
- Add detection of facility to downgrade FP512 datapath to FP256.
- bli_cpuid_is_avx512_fallback function does not exist, so remove header
definition.
AMD-Internal: [CPUPL-7303]
The bli_thread_barrier(thread) call before bli_l3_sup_thrinfo_free() in
bli_l3_sup_thread_decorator() was added by analogy with the conventional
path's PR #702 fix, but is not needed in the sup (small/unpacked) path.
In the conventional path, pack buffers are cached in the control tree
(cntl_t->pack_mem) and freed in the decorator after func() returns. A
barrier is required there to prevent a fast chief from releasing a pack
buffer back to the PBA pool while slower peers in a different sub-group
still read from it.
The sup path does not have this problem because:
1. Pack buffers are stack-local variables (mem_t in var2m), freed inside
func() by packm_sup_finalize_mem() after internal loop barriers.
They are never freed in this decorator.
2. The global communicator (gl_comm) is freed outside the parallel
region, protected by the implicit OpenMP barrier at the closing
brace of the parallel construct.
3. Sub-group communicators (created when packa/packb is enabled) are
freed only by the ochief thread in bli_thrinfo_free(). Non-chief
threads never dereference the shared communicator — they only read
their own ocomm_id and free_comm fields. When neither matrix is
packed, no sub-communicators exist (ocomm=NULL, free_comm=FALSE).
The custom spin-wait barrier (bli_thread_barrier) is significantly
slower than the OpenMP runtime barrier at high thread counts, causing
a ~10% DGEMM performance regression at 96 threads on AMD EPYC Turin
(e.g. 11000x300x200 DGEMM).
Ref: https://github.com/flame/blis/pull/702
Resolves: [CPUPL-7979] [SWLCSG-3951] [LWPHPCENGG-622]
Copy of similar change in upstream BLIS (843a5e8) to fix issues
https://github.com/flame/blis/issues/873 and
https://github.com/amd/blis/issues/50
Details:
- Previously, `<omp.h>` was included in `bli_thrcomm_openmp.h` so that the
framework could access the necessary OpenMP functions.
- As @melven reported (#873), this causes issues when `blis.h` is included
in C++ code since the `<omp.h>` include happens with `extern "C"`.
- Move the include from the header to the necessary .c files so that it
does not "pollute" `blis.h`.
Thanks to @DaAwesomeP and @bartoldeman for reporting this issue in
AOCL BLIS
AMD-Internal: [CPUPL-7303]
bli_arch_query_id() is used to select kernels in optimized BLAS APIs. Previous
implementation incurred the overhead of multiple function calls. This has
been reduced by:
- Changing the function to be defined in a header file so it can be inlined.
- Avoiding call to bli_arch_check_id_once that was a wrapper for a call to
bli_pthread_once. Instead bli_pthread_once is called directly.
- For builds with a single BLIS sub-configuration, correct arch_id is taken
directly from a header file in the corresponding config subdirectory,
avoiding the bli_pthread_once call and making the value explicit at
compile time, which may enable additional optimizations.
To enable these changes, the variables arch_id and model_id defined in
frame/base/bli_arch.c are no longer static, as they must be accessed in multiple
files (i.e. they are now global variables). Rename to g_arch_id and g_model_id
to distinguish from any locally defined arch_id or model_id variables.
Error handing code for invalid diag argument in Col major path was
incorrect in cblas_ctrmm compared to other invalid argument checks
and other data type variants.
AMD-Internal: [CPUPL-7303]
* Adding a model to determine which matrices enter the SGEMM tiny path
* This extends the sizes of matrices that enter the tiny path, which was constrained to the L1 cache size previously
* Now matrices that fit in L2 are also allowed into the tiny path, provided they are determined to be faster than the SUP path
* Adding thresholds based on the SUP path sizes
* Added for Zen4 and Zen5
---------
AMD-Internal: CPUPL-7555
Co-authored-by: Rohan Rayan <rohrayan@amd.com>
* thread: free global communicator after parallel region completes in pthreads decorator
Avoid potential data race by deferring free until all threads have joined. Previously, chief thread could free inside while non-chief threads still held pointers. Now, frees after the parallel region, following barrier and joins.
Files:
- frame/thread/bli_l3_sup_decor_pthreads.c
- frame/thread/bli_l3_decor_pthreads.c
* AMD-Internal: [CPUPL-7694]
* Global Communicator is now freed outside the parallel region
Description:
// Root threads don't "own" the global communicator
thrinfo_t* root = bli_thrinfo_create_root(comm, id, pool, pba);
// Setting free_comm=FALSE makes it clear: "This thread doesn't own this resource"
The thread that creates the communicator should be responsible for freeing it:
// Framework creates global communicator
thrcomm_t* gl_comm = bli_thrcomm_create(n_threads);
// Framework should clean it up, not individual threads
// (Even though only chief would actually do the cleanup)
Global communicators: Created by framework → free_comm=FALSE
Local communicators: Created by threads → free_comm=TRUE
Setting free_comm=FALSE provides an extra safety layer - if the chief thread logic ever changes, root threads won't accidentally try to free global communicators.
The current implementation has the framework handle global communicator cleanup:
// We shouldn't free the global communicator since it was already freed
// by the global communicator's chief thread in bli_l3_thrinfo_free()
** Technically root threads could have free_comm=TRUE and still be safe due to the chief thread protection, but the current change uses FALSE for better semantic clarity and architectural consistency.**
Made all changes to align with this design.
[CPUPL-7577]
* Removed old comments
* Applied similar changes to sequential code path
* For single thread we use global BLIS_SINGLE_COMM variable instead of allocating memory from sba pool
* Fixed comments
* Cleanup comments
- When alpha == 0, we are expected to only scale y vector with beta and not read A or X at all.
- This scenario is not handled properly in all code paths which causes NAN and INF from A and X being wrongly propagated. For example, for non-zen architecture (default block in switch case) no such check is present, similarly some of the avx512 kernels are also missing these checks.
- When beta == 0, we are not expected to read Y at all, this also is not handled correctly in one of the avx512 kernel.
- To fix these, early return condition for alpha == 0 is added to bla layer itself so that each kernel does not have to implement the logic.
- DGEMV AVX512 transpose kernel has been fixed to load vector Y only when beta != 0.
AMD-Internal: [CPUPL-7585]
```c
_Pragma( "omp parallel num_threads(n_threads)" )
{
// ... thread work ...
// Free the current thread's thrinfo_t structure.
bli_l3_thrinfo_free( rntm_p, thread ); // Line 183
}
// *** MISSING BARRIER HERE! ***
// Check the array_t back into the small block allocator...
bli_sba_checkin_array( array ); // Line 200
```
```c
// DANGEROUS execution timeline:
Thread 0 (chief):
completes func()
calls bli_l3_cntl_free()
calls bli_l3_thrinfo_free() → frees gl_comm ✓
exits OpenMP parallel region
calls bli_sba_checkin_array(array) → frees array ✗
Thread 1,2,3 (still executing):
still in func() or bli_l3_cntl_free()
trying to access freed gl_comm → CRASH!
trying to access freed array pools → CRASH!
```
This is **exactly the same issue** that PR #702 fixed in other files! The function needs a barrier before threads exit the parallel region to ensure:
1. **All threads complete their work** before any cleanup starts
2. **Global communicator isn't freed** while other threads are using it
3. **Array pools aren't freed** while other threads are accessing them
* Replaced omp barrier with bli_thread_barrier and added similar fix for pack compute routine
## **The Sequence is Critical:**
1. **All threads execute `func(...)`** - may access shared resources including communicators
2. **Barrier ** - ensures ALL threads finish their work
3. **Then each thread calls `bli_l3_sup_thrinfo_free()`** - only chief actually frees gl_comm
4. **Safe cleanup** - no use-after-free because all threads are done using gl_comm
## **What Would Happen Without the Barrier:**
```c
// Thread execution timeline WITHOUT barrier:
Time 1: Thread 0 finishes func() early → immediately calls bli_l3_sup_thrinfo_free()
Time 2: Thread 0 (chief) frees gl_comm
Time 3: Thread 1,2,3... still in func() → try to use freed gl_comm → CRASH!
```
## **With the Barrier (Current Safe Code):**
```c
// Thread execution timeline WITH barrier:
Time 1: All threads finish func() at different times
Time 2: ALL threads reach barrier → wait for slowest thread
Time 3: ALL threads proceed past barrier together
Time 4: Chief frees gl_comm → safe because no one using it anymore
```
- Remove redundant AOCL_DTL_LOG_NUM_THREADS calls from early return paths
- Update thread count logging to use AOCL_get_requested_threads_count() for early exits
- Clean up duplicate DTL logging in gemv_unf_var1 and gemv_unf_var2 implementations
- Remove thread count logging from bli_dgemv_n_zen4_int kernel variants
- Simplify aocldtl_blis.c AOCL_DTL_log_gemv_sizes by removing redundant conditional
- Standardize DTL trace exit patterns across axpy, scal, and gemv operations
- Remove commented-out DTL logging code in zen4 gemv kernel
This patch ensures thread count is logged only once per operation and uses
the correct API (AOCL_get_requested_threads_count) for early exit scenarios
where the actual execution thread count may differ from requested threads.
- Added `#pragma omp barrier` just before threads start releasing their threadinfo / global
communicator.
- This ensures all threads reach this sync point, preventing interleaved cleanup.
Co-authored-by: harsdave <harsdave@amd.com>
Adding SGEMM tiny path for Zen architectures.
Needed to cover some performance gaps seen wrt MKL
Only allowing matrices that all fit into the L1 cache to the tiny path
Only tuned for single threaded operation at the moment
Todo: Tune cases where AVX2 performs better than AVX512 on Zen4
Todo: The current ranges are very conservative, there may be scope to increase the matrix sizes that go into the tiny path
AMD-Internal: CPUPL-7555
Co-authored-by: Rohan Rayan rohrayan@amd.com
Recent changes meant nt was defined but unused for serial builds
when DTL was not enabled. Restructure code to avoid this issue and
make the code more readable.
AMD-Internal: [CPUPL-6579]
* DTL Log update
Updates logs with nt and AOCL Dynamic selected nt for axpy, scal and dgemv
Modified bench_gemv.c to able to process modified dtl logs.
* Updated DTL log for copy routine with actual nt and dynamic nt
* Refactor OpenMP pragmas and clean up code
Removed unnecessary nested OpenMP pragma and cleaned up function end comment.
* Fixed DTL log for sequential build
* Added thread logging in bla_gemv_check for invalid inputs
---------
Co-authored-by: Smyth, Edward <Edward.Smyth@amd.com>
- In the TPSV APIs, operations such as (*n - 1) * *incx and *n * (*n + 1) may lead to overflow when using bla_integer. To mitigate this risk, these operations have been updated to use dim_t variables instead of bla_integer.
- Explicitly initialize all fields of mem_t structures in bli_znormfv_unb_var1 and bli_dnormfv_unb_var1 to prevent undefined behavior when memory is not allocated.
- Add a NULL check after bli_thread_broadcast() in bli_thrinfo_sup_create_for_cntl to ensure that the communicator is valid, and call bli_abort() if broadcast fails.
- AMD specific BLAS1 and BLAS2 franework: changes to make variants
more consistent with each other
- Initialize kernel pointers to NULL where not immediately set
- Fix code indentation and other other whitespace changes in DTL
code and addon/aocl_gemm/frame/s8s8s32/lpgemm_s8s8s32_sym_quant.c
- Fix typos in DTL comments
- Add missing newline at end of test/CMakeLists.txt
- Standardize on using arch_id variable name
AMD-Internal: [CPUPL-6579]
Fix compiler warnings for unused variables. Situations where they
occurred were:
- bla_gemm_amd.c: Builds not including AMD zen4 or zen5 codepaths.
- bla_trsm_amd.c: Builds where {M,N,K]=1 optimizations are disabled.
- Avoid warnings when DTL is not enabled.
AMD-Internal: [CPUPL-6579]
* Check for single thread or multi-thread execution
- Check for single or multi-threaded call to dgemm.
- If it is single threaded, dgemm is computed in tiny path
if inputs under threashold.
- If it is multi-threaded dgemm input, tiny path will evaluate
whether it is performant compute the given input with single
thread or not. Based on that input will be routed.
* Disable gemm_small on zen4/5 & single thread execution check for tiny path
- Check for single or multi-threaded call to dgemm.
- If it is single threaded, dgemm is computed in tiny path
if inputs under threashold.
- If it is multi-threaded dgemm input, tiny path will evaluate
whether it is performant compute the given input with single
thread or not. Based on that input will be routed.
- Architecture checks for zen3, zen2 and zen are placed
for gemm_small path.
---------
Co-authored-by: harsdave <harsdave@amd.com>
- Change begin_asm and end_asm comments and unused code in files
kernels/haswell/3/sup/s6x16/bli_gemmsup_rv_haswell_asm_sMx6.c
kernels/zen4/3/sup/bli_gemmsup_cd_zen4_asm_z12x4m.c
to avoid problems in clobber checking script.
- Add missing clobbers in files
kernels/zen4/1m/bli_packm_zen4_asm_d24xk.c
kernels/zen4/1m/bli_packm_zen4_asm_z12xk.c
kernels/zen4/3/sup/bli_gemmsup_cv_zen4_asm_z12x4m.c
- Add missing newline at end of files.
- Update some copyright years for recent changes.
- Standardize license text formatting.
AMD-Internal: [CPUPL-6579]
Change handling of AOCL_VERBOSE environment variable and DTL APIs
AOCL_DTL_Enable_Logs() and AOCL_DTL_Disable_Logs() to better handle
threading within the calling application or library. Previously
this was a global setting, which was not thread-safe if these APIs
were called by different application threads. Recent changes made
them thread-safe but resulted in initialization issues.
Details of changes in this commit:
- Move DTL initialization and finalization out of bli_init_apis()
and handle pthread_once protection separately within DTL code.
- Check AOCL_VERBOSE environment variable and update global ICV
gbIsLoggingEnabled. Each application thread then initializes
from this to a thread local variable tlIsLoggingEnabled.
- tlIsLoggingEnabled is then used by AOCL_DTL_Enable_Logs() and
AOCL_DTL_Disable_Logs() to enable each application thread to
control logging separately.
- DTL_Initialize() no longer required the desired tracing level
(AOCL_DTL_TRACE_LEVEL) to be passed as an argument, as this
is available from the blis.h header file.
AMD-Internal: [CPUPL-7010]
Bug Fix in tpsv and tpmv - integer overflow
When BLAS integer size is 32 bits which is the case for LP64 binaries
the computation of "n * (n+1) / 2" will cause overflow due to overflow
of the operation "n * (n + 1)". This overflow is now fixed.
Fix: Replace bla_integer with dim_t for variables used for indexing
packed matrix (AP) in TPSV and TPMV functions. This prevents
overflow when computing kk = n*(n+1)/2 for large matrices.
In addition we added new file under bench called bench_getlibraryInfo.c which prints
all the information related to blis binary.
- Fixes ctpsv_, dtpsv_, stpsv_, ztpsv_
- Fixes ctpmv_, dtpmv_, stpmv_, ztpmv_
- Maintains BLAS compatibility
- Adjust the DCOPY aocl_dynamic threshold on Zen4 for optimal
fast-path selection.
- Extending the Zen5 dynamic-scheduling logic beyond the previous
8-thread limit.
- Update corresponding fast_path_thresh values in the frame such
that it matches the new dynamic logic.
- Added the initial implementation of the dgemm_thread_decision()
function to decide between single-threaded and multi-threaded
execution for DGEMM inputs.
- The function models per-thread tile work, core GFLOP/s, and
thread overhead (T_over=15 µs), and computes a K-threshold
that determines when multi-threading becomes beneficial.
- Returns true for ST and false for MT.
AMD-Internal: [SWLCSG-3418]
- Logic to determine if small code path should be taken or not does not take into account if matrix A is too large.
- Added a condition to use native code path if matrix A is very large.
AMD-Internal: [CPUPL-7201]
More improvements to DTL coverage and coding:
- Expand logging and tracing coverage to IxAMIN and GEMM_BATCH APIs
- Expand logging and performance states to GEMM3M APIs
- Expand logging coverage to matrix copy, transpose and add APIs
- Misc tidying of code
AMD-Internal: [CPUPL-7010]
Naming of Zen kernels and associated files was inconsistent with BLIS
conventions for other sub-configurations and between different Zen
generations. Other anomalies existed, e.g. dgemmsup 24x column
preferred kernels names with _rv_ instead of _cv_. This patch renames
kernels and file names to address these issues.
AMD-Internal: [CPUPL-6579]
- For Conjugate inputs, ZTRSM small code path is less accurate than native codepath.
- Redirected the conjugate inputs to native code path on ZEN4 if TRSM preinversion is disabled.
- Tuned AOCL_DYNAMIC to handle the new inputs redirected to ZTRSM native.
More improvements to DTL coverage and coding:
- Removed some DTL overheads from performance stats timing for all APIs
where it is currently implemented (i.e. gemm, gemmt, trsm, nrm2)
- Expand logging coverage to gemm pack and compute APIs, including
performance stats for gemm_compute
- Expand logging coverage to rot, rotg, rotm and rotmg APIs
- Tidied order of function prototypes in aocl_dtl/aocldtl_blis.h
AMD-Internal: [CPUPL-7010]
Commit eaa76dfe28 added LAPACK 3.12 GEMMTR
interfaces as aliases to existing BLIS GEMMT. Here we add full set of
Fortran upper case and no underscore API aliases and _blis_impl variants.
AMD-Internal: [CPUPL-6581]
More improvements to DTL coverage and coding:
- Expand logging coverage to banded matrix APIs in frame/compat/f2c
- Expand logging coverage to packed matrix APIs in frame/compat/f2c
- Commit b8aa5c2894 was wrong to
remove calls to AOCL_DTL_INITIALIZE for APIs where bli_init_auto()
is not called. AOCL_DTL_INITIALIZE is essential when logging is
enabled but tracing is not, otherwise the ICV gbIsLoggingEnabled
will not be initialized based on logging status and remain as
the default FALSE value.
AMD-Internal: [CPUPL-7010]
* Added DGEMV no transpose multithreaded Implementations
- Added new avx512 M and N kernels for DGEMV.
- Added multiple MT implementations for same kernels.
- Added AOCL_dynamic logic for L2 apis.
- Tuned AOCL_dynamic and code path selection for DGEMV on ZEN5.
- Added same kernels for SGEMV, but these kernels are not enabled yet.
- Added SGEMV reference kernel.
AMD-Internal: [SWLCSG-3408]
Co-authored-by: Varaganti, Kiran <Kiran.Varaganti@amd.com>
