Replace include of non-standard header malloc.h by
stdlib.h to fix issue reported on upstream BLIS
github.
https://github.com/flame/blis/issues/785
AMD-Internal: [CPUPL-4307]
Change-Id: I4ac5cb3164fe7050bba6579b08cc2d3ff412ccba
Non aligned rntm_t struct can potentially have its first/last cache
line shared with other objects in memory. This could affect performance
depending on how much the shared cache lines are used. rntm_t struct is
aligned to 64 bytes to workaround this issue.
Change-Id: Id0956fca771be062ada9f81e8cd75ac1f290fd8e
In Gtestsuite CMakeLists.txt, find_library() will search
user-mentioned library in default system paths first then
in user specified paths. To avoid this CMake is updated
to search the user mentioned library in user specified
path and ignore searching in default path.
AMD-Internal: [CPUPL-4284]
Change-Id: Ia99cf59eb39deac4110d3d733f17548d432dde64
- The bli_snormfv_unb_var1( ... ) and bli_cnormfv_unb_var1( ... )
functions posed an uninitialized pointer read coverity issue,
due to the local rntm_t object being declared as part of the
function scope, but initialized only on a need basis(i.e, when
attempting to pack x vector if incx != 1).
- The fix was to have the declaration and initialization inside
the case where incx != 1, thereby making the scope of the rntm_t
and mem_t objects more stringent.
- This required an additional condition to call the kernel in case
of unit stride.
AMD-Internal: [CPUPL-4278]
Change-Id: I763b1d4920532557749d8943f12b6df626aa5372
- Changed the threshold for using ZTRSM small
code path when multithreading is enabled.
- Very skinny matrices are not taken into
consideration in existing threshold tuning.
AMD-Internal: [CPUPL-4267]
Change-Id: I4294ec58a8535af7a9d618ae8f0d86407b66f341
Include bli_config.h before bli_system.h in
./frame/compat/cblas/src/cblas.h so that BLIS_ENABLE_SYSTEM is
defined correctly before it is needed. This copies the change
to ./frame/include/blis.h made in 1f527a93b9 (via merge
c6f3340125). Also standardize some comments and formatting
between blis.h and cblas.h
AMD-Internal: [CPUPL-4251]
Change-Id: Ie5cab646367f15003c25fa126344b02640d9106e
Updated sgemm testcase to handle multiple values of alpha, beta for different input size
Added sgemm testcase to cover m,n,k dimension till 20 size atleast instepsize of 1
Change-Id: Id10ba3d7a05154b171511ef11ea76297494672cd
Description:
1. Updated the thread partition logic for aocl_gemm_f32f32f32of32
for m<MR, n<NR cases and also balanced thread in m, n directions
such that each thread gets equal amount of work and not to span
thread without any work.
2. Disabled dynamic enabling of packing of a and b matrixes for
smaller sizes for genoa architecture.
AMD-Internal: [SWLCSG-2353 , SWLCSG-2391]
Change-Id: I03b2c50e592c2e9d336ea84c0e0394af63a34cec
Some AVX512 intrinsics(eg: _mm_loadu_epi8) were introduced in later
versions of gcc (11+) in addition to already existing masked intrinsic
(eg: _mm_mask_loadu_epi8). In order to support compilation using gcc
10.2, either the masked intrinsic or other gcc 10.2 compatible intrinsic
needs to be used (eg: _mm_loadu_si128) in LPGEMM <u|s>8s8os32 kernels.
AMD-Internal: [SWLCSG-2542]
Change-Id: I6cfedfdcb28711b19df63d162ab267f5eea8d2ef
- Instead of putting the built libraries in blis/bin directory, build them in the chosen build-cmake directory.
- Install headers in <prefix>/include instead of <prefix>/include/blis.
- Fix on some targets to match configure/make system.
- Update documentation.
AMD-Internal: [CPUPL-2748]
Change-Id: I15553948209345dbee350e89965b6a3c72a4e340
Some text files were missing a newline at the end of the file.
One has been added.
AMD-Internal: [CPUPL-3519]
Change-Id: I4b00876b1230b036723d6b56755c6ca844a7ffce
User control over code path using AOCL_ENABLE_INSTRUCTIONS
or BLIS_ARCH_TYPE only makes sense for fat binary builds.
Thus this functionality is now disabled by default for
single architecture builds. User can still override the default
selections by using configure options --enable-blis-arch-type
or --disable-blis-arch-type.
Other changes:
- include x86_64 family as using zen codepaths in cmake build system.
- Update help and error messages to include AOCL_ENABLE_INSTRUCTIONS.
AMD-Internal: [CPUPL-4202]
Change-Id: I7aa5fcf89df8675bcc12d81f81781de647e0fcf8
1. Prefetch only MR rows or rows required for fringe cases
2. Specify prefetching offset - the least column address supported
by masked functions
3. Removed unnecessary prefetches in fringe case for mx4 kernels
Updated gtestuite for sgemm calls
AMD_Internal: [CPUPL-4221]
Change-Id: I1e2e7d3ebce37dc54a2f0a5c1c70ce0a6d4c8d6c
- This commit uses avx2 and avx512 masked load instructions
for handling edge case where vector size is not exact multiple
of avx2/avx512 vector register size.
- Thanks to Shubham, Sharma <shubham.sharma3@amd.com> for
avx512 ddotv kernel changes
Change-Id: I998651eeb1083caf3308f1b45bd7d55b7974bcb4
- For the inputs where either m or n is 1, based on right or
left side, it invokes c/z scalv kernel and post that it scales the
matrix post checking whether the input is blis conjugate transpose
or not.
- Previously the check condition was case sensitive *diaga = 'n',
and as a result, it is always executing the "else" code-part.
- Fixed the condition check.
AMD-Internal: [CPUPL-4204]
Change-Id: Iae2514c742ab17ac6c6e43036da095a74ad131c5
Segfault was reported through nightly jenkins job.
Issue was observed when running in MT mode.
Issue was due to extra broadcast being used.
Extra broadcast would access out of bound memory on input buffer
Cleaned up cobbler list by removing unused registers.
AMD_Internal: [CPUPL-4180]
Change-Id: I1c8715b2850ef855328f2ef12f215987299bdb2b
Changes in commit 64a1f786d5 (via merge c6f3340125) included in
./frame/include/bli_type_defs.h a prototype that uses the C
restrict keyword. When using C++ we need to provide a definition
for this C language keyword. This is done in bli_lang_defs.h which
was included in blis.h but not in cblas.h.
AMD-Internal: [CPUPL-4188]
Change-Id: I75d5f32599d18794331ff452e562eb42afb5ae93
- In case the build directory doesn't get cleaned between different configurations this should re-generate the headers correctly.
AMD-Internal: [CPUPL-2748]
Change-Id: I57cd03a9ae87d8ddfee64fe8b1a1ee9ea1b7ad3c
* commit '5013a6cb':
More edits and fixes to docs/FAQ.md.
Fixed newly broken link to CREDITS in FAQ.md.
More minor fixes to FAQ.md and Sandboxes.md.
Updates to FAQ.md, Sandboxes.md, and README.md.
Safelist 'master', 'dev', 'amd' branches.
Re-enable and fix fb93d24.
Reverted fb93d24.
Re-enable and fix 8e0c425 (BLIS_ENABLE_SYSTEM).
Removed last vestige of #define BLIS_NUM_ARCHS.
Added new packm var3 to 'gemmlike'.
Fix problem where uninitialized registers are included in vhaddpd in the Mx1 gemmsup kernels for haswell.
Fix more copy-paste errors in the haswell gemmsup code.
Do a fast test on OSX. [ci skip]
Fix AArch64 tests and consolidate some other tests.
Use C++ cross-compiler for ARM tests.
Attempt to fix cxx-test for OOT builds.
Updated travis-ci.org link in README.md to .com.
Disabled (at least temporarily) commit 8e0c425.
Define BLIS_OS_NONE when using --disable-system.
Updated stale calls to malloc_intl() in gemmlike.
Blacklist clang10/gcc9 and older for 'armsve'.
Add test to Travis using C++ compiler to make sure blis.h is C++-compatible.
Moved lang defs from _macro_def.h to _lang_defs.h.
Minor tweaks to gemmlike sandbox.
Added local _check() code to gemmlike sandbox.
README.md citation updates (e.g. BLIS7 bibtex).
Tweaks to gemmlike to facilitate 3rd party mods.
Whitespace tweaks.
Add row- and column-strides for A/B in obj_ukr_fn_t.
Clean up some warnings that show up on clang/OSX.
Remove schema field on obj_t (redundant) and add new API functions.
Add dependency on the "flat" blis.h file for the BLIS and BLAS testsuite objects.
Disabled sanity check in bli_pool_finalize().
Implement proposed new function pointer fields for obj_t.
AMD-Internal: [CPUPL-2698]
Change-Id: I6fc33351fa824580cf4f25b63f0370383cd9422d
Memory allocated for pointer chars_for_dt was not freed at the
end of function in testsuite.
Freeing up of the buffer fixed the issue.
AMD-Internal: [CPUPL-3932]
Change-Id: I432c3ff95d289159f02a871b6d4fff5ab252ea9e
Updating cmake files to place include folder under
blis directory in new cmake system on windows.
AMD-Internal: [CPUPL-2748]
Change-Id: I650cca95193f7c89b39648ac1bda1fa1093b1560
Added all fringe kernels with mask load store support
Fringe kernels cover m direction from 5 to 1 and
n direction from 15 to 1 for row storage format
- New edge kernels that uses masked load-store
instructions for handling corner cases.
- Mask load-store instruction macros are added.
vmaskmovps, VMASKMOVPS for masked load-store.
- It improves performance by reducing branching overhead
and by being more cache friendly.
- Mask load-store is added only for row storage format
AMD-Internal: [CPUPL-4041]
Change-Id: I563c036c79bf8e476a8ebde37f8f6db751fb3456
- Following optimizations are included for dgemm 6x8 native kernel.
1) Reorganized the C update and store to reduce register dependencies.
2) moved the C prefetch to part-way through the kernel for efficiently
prefetching C matrix at appropriate distance.
3) Offsetting A matrix, so that kernel can use a smaller instruction
encoding saving, saving i-cache space.
4) Aligned the K iteration loop.
- Thanks to Moore, Branden <Branden.Moore@amd.com> for these design
changes of DGEMM 6x8 native kernels.
- Additional change, reorganization of C update and store for
beta zero case to facilitate out of order execution of storing of C
matrix.
Change-Id: I9d1ec8d39f1154b0f38b136bd6a04b05d7d1e6ba
- Added a conditional check to see if the vectorized kernels
for DNRM2_ and DZNRM2_ can be called directly, without
incurring any framework overhead.
- The condition to satisfy this fast-path is for the size to be
such that the ideal threads required is 1, with the vector having
unit stride( so that packing at the framework-level can be avoided ).
AMD-Internal: [CPUPL-4045]
Change-Id: Ie37e86f802ada0e226dff88e74f0341e97ebfe28
1. Added input parameter checking for the extension APIs
1. gemm_pack_get_size API
2. gemm_pack API
2. Additionally added early returns for these APIs when
m or n dimensions are 0.
3. Routines for input parameter check for all the 3
BLAS extension APIs - gemm_pack_get_size, gemm_pack and
gemm_compute are defined in:
frame/compat/check/bla_gemm_pack_compute_check.h
4. Added AOCL DTL TRACE for all the functions of
1. gemm_pack_get_size
2. gemm_pack
3. gemm_compute
AMD-Internal: [CPUPL-3560]
Change-Id: I4351b8494d888eae7e7431a7e1e23e442ffc8631
Details:
- Moved the downscale & postop options from commmandline to
input file.
- Now the format of the input file is as follows:
dt_in dt_out stor transa transb op_a op_b m n k lda ldb ldc postops
- In case of no-postops, 'none' has to be passed in the place of
postops.
- Removed duplication of mat_mul_bench_main function for bf16 APIs.
- Added a function called print_matrix for each datatype which can
help in printing matrices while debugging.
- Added printing of ref, computed and diff values while reporting
failure.
- Added new functions for memory allocation and freeing. Different
types of memory allocation is chosen based on mode bench is
running(performance or accuracy mode).
Change-Id: Ia7d740c53035bc76e578a03869590c9f04396b72
The LPGEMM micro-kernel operates on blocks of dimension MRxKC and
KCxNR. Current LPGEMM design involves using all the available threads
for computing the output. If the number of threads assigned along ic
or jc direction is more than M/MR or N/NR blocks respectively, it
could results in threads sleeping due to the lack of MR or NR blocks.
This scenario is now handled by reducing the number of threads if
there are threads without any work (MR or NR blocks).
AMD-Internal: [SWLCSG-2354, SWLCSG-2389, SWLCSG-2267]
Change-Id: I74819337c7a0d3ab05ea0e18bb42780f977ea8f6
We are using pthread_self to get a thread id for use in the DTL
tracing functionality to name individual output files per thread.
This is not an appropriate use of pthread_self as its return type
(pthread_t) is an opaque type that can vary between implementations.
On linux we haven't had a problem, as pthread_t is an unsigned long
int. However on freeBSD it is a pointer to an empty struct. The
difference between this and the int type we used for its value within
the BLIS code was causing a compile error.
The best long term solution would be for pthread builds to maintain
their own internal thread id. A mechanism to implement this has not
yet been identifie. In the meantime, we make the following changes as
a stopgap:
- Explicitly cast from pthread_t return value to our BLIS internal
data type AOCL_TID.
- Make AOCL_TID a long int rather than pid_t (i.e. an int) in pthread
builds to match the sizes expected on both Linux and FreeBSD.
AMD-Internal: [CPUPL-4167]
Change-Id: Ia07ee8f97273cc3bab46f6bca1eeb7954320415b
- This commit helps improving performance for very small input
by reducing framework check and routing all such inputs to
bli_dgemm_tiny_6x8_kernel. It forces single threaded computation
for such sizes.
- It invokes bli_dgemm_tiny_6x8_kernel for ZEN, ZEN2, ZEN3 and ZEN4
code path. Except for the case AOCL_ENABLE_INSTRUCTIONS environment
variable is set to avx512. In that case, such a small inputs are
routed to bli_dgemm_tiny_24x8_kernel avx512 kernel.
AMD-Internal: [CPUPL-1701]
Change-Id: Idf59f4a8ee76ee8f2514a33be2b618e3ce02383e
- According to BLAS Standards, SCALV should return when incx .le. 0.
- To make SCALV compliant to this, added an early return inside the BLAS
layer, for the cases where incx <= 0.
- Also, added early return for the case where alpha is a unit scalar.
AMD-Internal: [CPUPL-3562]
Change-Id: Id474fdd6ed9232226f5c5381d0398f43384e4a49
- Abstracted packing from the vectorized kernels for SNRM2 and SCNRM2 to
a layer higher.
- Added a scalar loop to handle compute in case of non-unit strides.
This loop ensures functionality in case packing fails at the
framework level.
AMD-Internal: [CPUPL-3633]
Change-Id: I555aea519d7434d43c541bb0f661f81105135b98
- When configured for haswell config "Warning unused variable 'zero'"
was throwed during compilation.
- Removed zero variable which is not being used
AMD-Internal: [CPUPL-3973]
Change-Id: I45a1f16b4c50307b07148bba63ca5332c48648b8
- Updated the final reduction of partial sums( AVX-2 code section )
to use scalar accumulation entirely, instead of using the
_mm256_hadd_pd( ... ) intrinsic. This will in turn change the
associativity in the reduction step.
- Reverted to using scalar code on the fringe cases in AVX-2 kernel
for DNRM2 and DZNRM2, for improving functional correctness.
AMD-Internal: [CPUPL-4049]
Change-Id: I9d320b39d23a0cbcc77fb24d951fced778ea5ea5
- This commit implements avx512 dgemm kernel for k=1 cases.
which gets called for zen4 codepath.
- Added architecture check for k=1 kernel in dgemm code path
to pick correct kernel based on cpu arhcitecture since now
blis is having avx2 and avx512 dgemm kernels for k=1 case.
- Previously in dgemm path bli_dgemm_8x6_avx2_k1_nn kernel was
being called irrespective of architecture type.
- Added architecture check before calling the kernel for case where
k=1, so only for respective architectures this kernel is invoked.
AMD-Internal: [CPUPL-4017]
Change-Id: I418bbc933b41db41d323b331c6d89893868a6971
- AOCL dynamic logic that determines the number of threads to be
launched has been modified.
AMD-Internal: [CPUPL-3956]
Change-Id: Ia6c052515bd24e93660f020a7d0894fc75a229fc
- Pack and compute are now compared against GEMM operation of reference
library when MKL is not used as a reference.
- For the case where both A and B are unpacked, the reference GEMM is
invoked with a unit-alpha scalar.
- If MKL is used as reference, then these APIs are compared against pack
and compute operations of MKL.
- Updated description in ref_gemm_compute.cpp to reflect this behavior.
AMD-Internal: [CPUPL-4084]
Change-Id: Id0521c9cad8743a7ae471a7f3c547ceb67191f86
- Added 4x12 ZGEMM row-preferred kernel.
- Added 4x12 ZTRSM row-preferred lower
and upper kernels using AVX512 ISA.
- These kernels are used for ZTRSM only, zgemm
still uses 12x4 kernel.
- Kernels support row/col/gen storage.
- Kernels support A prefetch, B prefetch,
A_next prefetch, B_next prefetch and c prefetch.
- B prefetch, B_next prefetch and C prefetch
are enabled by default.
- Updated CMakeLists.txt with ZGEMM kernels for
windows build.
AMD-Internal: [CPUPL-3781]
Change-Id: I0fb4b2ec2f4bd66db6499c25f12bcc4bdb09804a
- BLAS compute checks updated to properly check for rs_c and cs_c.
- Updated BLAS compute checks to skip validity check if m==1 or n==1.
For the same reason, added a check just before to validate rs_c and
cs_c are greater than or equal to 1.
- Added tiny size tests to gtestsuite as a sanity check.
- Also updated the Invalid Input Tests to test for the updated checks.
AMD-Internal: [CPUPL-4140]
Change-Id: I984339ec7909778b58409ffcdbeed4ee33f28cfb
- Symbols for gemm_pack_get_size were not being exported properly when
BLIS was built as a shared library.
- Correctly assigned the BLIS_EXPORT_BLAS macro to ?gemm_pack_get_size_
function declaration.
- Added missing gemm_pack and gemm_pack_get_size macros to
bli_macro_defs.h file.
- Removed an unnecessary BLIS_EXPORT_BLAS macro from dgemm_compute
function definition.
- Updated bli_util_api_wrap with no underscore API wrappers for pack and
compute set of BLAS Extension APIs:
1. ?gemm_pack_get_size
2. ?gemm_pack
3. ?gemm_compute
AMD-Internal: [CPUPL-4083]
Change-Id: I78cd7642c2fcbfdf02676e654a377ad2aa5295c1
1. OpenMP based multi-threading parallelism is added for BLAS
extension APIs of Pack and Compute
2. Both pack and compute APIs are parallelized.
3. Multi-threading of pack and compute APIs done with different
number of threads can lead to inconsistent results due to
output difference of the full packed matrix buffer when packed
with different number of threads.
4. In multi-threaded execution, we ensure output of packed buffer
is exactly the same as in single threaded execution.
5. Similarly for compute API, read of packed buffer in multi-
threaded execution is exactly the same as in single-threaded
execution.
6. Routines are added to compute the offsets for thread workload
distribution for MT execution.
1. The offsets are calculated in such a way that it resembles
the reorder buffer traversal in single threaded reordering.
2. The panel boundaries (KCxNC) remain as it is accessed in
single thread, and as a consequence a thread with jc_start
inside the panel cannot consider NC range for reorder.
3. It has to work with NC' < NC, and the offset is calulated
using prev NC panels spanning k dim + cur NC panel spaning
pc loop cur iteration + (NC - NC') spanning current
kc0 (<= KC).
7. Routines to ensure the same are added for MT execution
1. frame/base/bli_pack_compute_utils.c
2. frame/base/bli_pack_compute_utils.h
AMD-Internal: [CPUPL-3560]
Change-Id: I0dad33e0062519de807c32f6071e61fba976d9ac
- Enabled the vectorized AVX-2 code-path for SNRM2_. The
framework queries the architecture ID and calls the
vectorized kernel based on the architecture support.
- In case of not having the architecture support, we use
the default path based on the sumsqv method.
AMD-Internal: [CPUPL-3277]
Change-Id: Ic60c0782dec0b7eb09fac21818eb625e57b1d14f
Details:
- Modified bench to support testing for sizes where matrix
strides are larger than the corresponding dimensions.
- Modified early-return checks in all interface APIs to
check validity of strides in relation to the corresponding
dimension rather than checking if strides are equal to dimensions.
Change-Id: I382529b636a4acc75f6d93d997af22a168a7bfc4
