amd/blis
1
0
Fork 0
mirror of https://github.com/amd/blis.git synced 2026-04-30 12:31:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
537a1f4f85ce1aa008901857cb3182e6b4546d7f
blis/config/template/kernels/3/bli_gemmtrsm_u_opt_mxn.c
Field G. Van Zee 537a1f4f85 Implemented runtime contexts and reorganized code. 
Details:
- Retrofitted a new data structure, known as a context, into virtually
  all internal APIs for computational operations in BLIS. The structure
  is now present within the type-aware APIs, as well as many supporting
  utility functions that require information stored in the context. User-
  level object APIs were unaffected and continue to be "context-free,"
  however, these APIs were duplicated/mirrored so that "context-aware"
  APIs now also exist, differentiated with an "_ex" suffix (for "expert").
  These new context-aware object APIs (along with the lower-level, type-
  aware, BLAS-like APIs) contain the the address of a context as a last
  parameter, after all other operands. Contexts, or specifically, cntx_t
  object pointers, are passed all the way down the function stack into
  the kernels and allow the code at any level to query information about
  the runtime, such as kernel addresses and blocksizes, in a thread-
  friendly manner--that is, one that allows thread-safety, even if the
  original source of the information stored in the context changes at
  run-time; see next bullet for more on this "original source" of info).
  (Special thanks go to Lee Killough for suggesting the use of this kind
  of data structure in discussions that transpired during the early
  planning stages of BLIS, and also for suggesting such a perfectly
  appropriate name.)
- Added a new API, in frame/base/bli_gks.c, to define a "global kernel
  structure" (gks). This data structure and API will allow the caller to
  initialize a context with the kernel addresses, blocksizes, and other
  information associated with the currently active kernel configuration.
  The currently active kernel configuration within the gks cannot be
  changed (for now), and is initialized with the traditional cpp macros
  that define kernel function names, blocksizes, and the like. However,
  in the future, the gks API will be expanded to allow runtime management
  of kernels and runtime parameters. The most obvious application of this
  new infrastructure is the runtime detection of hardware (and the
  implied selection of appropriate kernels). With contexts in place,
  kernels may even be "hot swapped" at runtime within the gks. Once
  execution enters a level-3 _front() function, the memory allocator will
  be reinitialized on-the-fly, if necessary, to accommodate the new
  kernels' blocksizes. If another application thread is executing with
  another (previously loaded) kernel, it will finish in a deterministic
  fashion because its kernel information was loaded into its context
  before computation began, and also because the blocks it checked out
  from the internal memory pools will be unaffected by the newer threads'
  reinitialization of the allocator.
- Reorganized and streamlined the 'ind' directory, which contains much of
  the code enabling use of induced methods for complex domain matrix
  multiplication; deprecated bli_bsv_query.c and bli_ukr_query.c, as
  those APIs' functionality is now mostly subsumed within the global
  kernel structure.
- Updated bli_pool.c to define a new function, bli_pool_reinit_if(),
  that will reinitialize a memory pool if the necessary pool block size
  has increased.
- Updated bli_mem.c to use bli_pool_reinit_if() instead of
  bli_pool_reinit() in the definition of bli_mem_pool_init(), and placed
  usage of contexts where appropriate to communicate cache and register
  blocksizes to bli_mem_compute_pool_block_sizes().
- Simplified control trees now that much of the information resides in
  the context and/or the global kernel structure:
  - Removed blocksize object pointers (blksz_t*) fields from all control
    tree node definitions and replaced them with blocksize id (bszid_t)
    values instead, which may be passed into a context query routine in
    order to extract the corresponding blocksize from the given context.
  - Removed micro-kernel function pointers (func_t*) fields from all
    control tree node definitions. Now, any code that needs these function
    pointers can query them from the local context, as identified by a
    level-3 micro-kernel id (l3ukr_t), level-1f kernel id, (l1fkr_t), or
    level-1v kernel id (l1vkr_t).
  - Removed blksz_t object creation and initialization, as well as kernel
    function object creation and initialization, from all operation-
    specific control tree initialization files (bli_*_cntl.c), since this
    information will now live in the gks and, secondarily, in the context.
- Removed blocksize multiples from blksz_t objects. Now, we track
  blocksize multiples for each blocksize id (bszid_t) in the context
  object.
- Removed the bool_t's that were required when a func_t was initialized.
  These bools are meant to allow one to track the micro-kernel's storage
  preferences (by rows or columns). This preference is now tracked
  separately within the gks and contexts.
- Merged and reorganized many separate-but-related functions into single
  files. This reorganization affects frame/0, 1, 1d, 1m, 1f, 2, 3, and
  util directories, but has the most obvious effect of allowing BLIS
  to compile noticeably faster.
- Reorganized execution paths for level-1v, -1d, -1m, and -2 operations
  in an attempt to reduce overhead for memory-bound operations. This
  includes removal of default use of object-based variants for level-2
  operations. Now, by default, level-2 operations will directly call a
  low-level (non-object based) loop over a level-1v or -1f kernel.
- Converted many common query functions in blk_blksz.c (renamed from
  bli_blocksize.c) and bli_func.c into cpp macros, now defined in their
  respective header files.
- Defined bli_mbool.c API to create and query "multi-bools", or
  heterogeneous bool_t's (one for each floating-point datatype), in the
  same spirit as blksz_t and func_t.
- Introduced two key parameters of the hardware: BLIS_SIMD_NUM_REGISTERS
  and BLIS_SIMD_SIZE. These values are needed in order to compute a third
  new parameter, which may be set indirectly via the aforementioned
  macros or directly: BLIS_STACK_BUF_MAX_SIZE. This value is used to
  statically allocate memory in macro-kernels and the induced methods'
  virtual kernels to be used as temporary space to hold a single
  micro-tile. These values are now output by the testsuite. The default
  value of BLIS_STACK_BUF_MAX_SIZE is computed as
  "2 * BLIS_SIMD_NUM_REGISTERS * BLIS_SIMD_SIZE".
- Cleaned up top-level 'kernels' directory (for example, renaming the
  embarrassingly misleading "avx" and "avx2" directories to "sandybridge"
  and "haswell," respectively, and gave more consistent and meaningful
  names to many kernel files (as well as updating their interfaces to
  conform to the new context-aware kernel APIs).
- Updated the testsuite to query blocksizes from a locally-initialized
  context for test modules that need those values: axpyf, dotxf,
  dotxaxpyf, gemm_ukr, gemmtrsm_ukr, and trsm_ukr.
- Reformatted many function signatures into a standard format that will
  more easily facilitate future API-wide changes.
- Updated many "mxn" level-0 macros (ie: those used to inline double loops
  for level-1m-like operations on small matrices) in frame/include/level0
  to use more obscure local variable names in an effort to avoid variable
  shaddowing. (Thanks to Devin Matthews for pointing these gcc warnings,
  which are only output using -Wshadow.)
- Added a conj argument to setm, so that its interface now mirrors that
  of scalm. The semantic meaning of the conj argument is to optionally
  allow implicit conjugation of the scalar prior to being populated into
  the object.
- Deprecated all type-aware mixed domain and mixed precision APIs. Note
  that this does not preclude supporting mixed types via the object APIs,
  where it produces absolutely zero API code bloat.
2016-04-11 17:21:28 -05:00

323 lines
12 KiB
C
Raw Blame History

/*
BLIS
An object-based framework for developing high-performance BLAS-like
libraries.
Copyright (C) 2014, The University of Texas at Austin
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
- Neither the name of The University of Texas at Austin nor the names
of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "blis.h"
void bli_sgemmtrsm_u_opt_mxn
(
dim_t k,
float* restrict alpha,
float* restrict a10,
float* restrict a11,
float* restrict b01,
float* restrict b11,
float* restrict c11, inc_t rs_c, inc_t cs_c,
auxinfo_t* restrict data,
cntx_t* restrict cntx
)
{
const num_t dt = BLIS_FLOAT;
const inc_t packnr = bli_cntx_get_blksz_max_dt( dt, BLIS_NR, cntx );
const inc_t rs_b = packnr;
const inc_t cs_b = 1;
float* restrict minus_one = bli_sm1;
bli_sgemm_opt_mxn( k,
minus_one,
a12,
b21,
alpha,
b11, rs_b, cs_b,
data );
bli_strsm_u_opt_mxn( a11,
b11,
c11, rs_c, cs_c,
data );
}
void bli_dgemmtrsm_u_opt_mxn
(
dim_t k,
double* restrict alpha,
double* restrict a10,
double* restrict a11,
double* restrict b01,
double* restrict b11,
double* restrict c11, inc_t rs_c, inc_t cs_c,
auxinfo_t* restrict data,
cntx_t* restrict cntx
)
{
/*
Template gemmtrsm_u micro-kernel implementation
This function contains a template implementation for a double-precision
real micro-kernel that fuses a gemm with a trsm_u subproblem.
This micro-kernel performs the following compound operation:
B11 := alpha * B11 - A12 * B21 (gemm)
B11 := inv(A11) * B11 (trsm)
C11 := B11
where A11 is MR x MR and upper triangular, A12 is MR x k, B21 is k x NR,
B11 is MR x NR, and alpha is a scalar. Here, inv() denotes matrix
inverse.
Parameters:
- k: The number of columns of A12 and rows of B21.
- alpha: The address of a scalar to be applied to B11.
- a12: The address of A12, which is the MR x k submatrix of the packed
micro-panel of A that is situated to the right of the MR x MR
triangular submatrix A11. A12 is stored by columns with leading
dimension PACKMR, where typically PACKMR = MR.
- a11: The address of A11, which is the MR x MR upper triangular
submatrix within the packed micro-panel of matrix A that is
situated to the left of A12. A11 is stored by columns with
leading dimension PACKMR, where typically PACKMR = MR. Note
that A11 contains elements in both triangles, though elements
in the unstored triangle are not guaranteed to be zero and
thus should not be referenced.
- b21: The address of B21, which is the k x NR submatrix of the packed
micro-panel of B that is situated above the MR x NR submatrix
B11. B01 is stored by rows with leading dimension PACKNR, where
typically PACKNR = NR.
- b11: The address B11, which is the MR x NR submatrix of the packed
micro-panel of B, situated below B01. B11 is stored by rows
with leading dimension PACKNR, where typically PACKNR = NR.
- c11: The address of C11, which is the MR x NR submatrix of matrix
C, stored according to rs_c and cs_c. C11 is the submatrix
within C that corresponds to the elements which were packed
into B11. Thus, C is the original input matrix B to the overall
trsm operation.
- rs_c: The row stride of C11 (ie: the distance to the next row of C11,
in units of matrix elements).
- cs_c: The column stride of C11 (ie: the distance to the next column of
C11, in units of matrix elements).
- data: The address of an auxinfo_t object that contains auxiliary
information that may be useful when optimizing the gemmtrsm
micro-kernel implementation. (See BLIS KernelsHowTo wiki for
more info.)
- cntx: The address of the runtime context. The context can be queried
for implementation-specific values such as cache and register
blocksizes. However, most micro-kernels intrinsically "know"
these values already, and thus the cntx argument usually can
be safely ignored. (The following template micro-kernel code
does in fact query MR, NR, PACKMR, and PACKNR, as needed, but
only because those values are not hard-coded, as they would be
in a typical optimized micro-kernel implementation.)
Diagram for gemmtrsm_u
The diagram below shows the packed micro-panel operands for trsm_l and
how elements of each would be stored when MR = NR = 4. (The hex digits
indicate the layout and order (but NOT the numeric contents) in memory.
Here, matrix A11 (referenced by a11) is upper triangular. Matrix A11
does contain elements corresponding to the strictly lower triangle,
however, they are not guaranteed to contain zeros and thus these elements
should not be referenced.
a11: a12: NR
________ ___________________ _______
|`. |0 4 8 | b11:|0 1 2 3|
MR | `. |1 5 9 . . . | |4 5 6 7|
| `. |2 6 A | MR |8 9 A B|
|______`.|3_7_B______________| |___.___|
b21:| . |
MR k | . |
| |
| |
NOTE: Storage digits are shown k | |
starting with a12 to avoid | |
obscuring triangular structure | |
of a11. |_______|
Implementation Notes for gemmtrsm
- Register blocksizes. See Implementation Notes for gemm.
- Leading dimensions of a1 and b1: PACKMR and PACKNR. See Implementation
Notes for gemm.
- Edge cases in MR, NR dimensions. See Implementation Notes for gemm.
- Alignment of a1 and b1. The addresses a1 and b1 are aligned according
to PACKMR*sizeof(type) and PACKNR*sizeof(type), respectively.
- Unrolling loops. Most optimized implementations should unroll all
three loops within the trsm subproblem of gemmtrsm. See Implementation
Notes for gemm for remarks on unrolling the gemm subproblem.
- Prefetching next micro-panels of A and B. When invoked from within a
gemmtrsm_l micro-kernel, the addresses accessible via
bli_auxinfo_next_a() and bli_auxinfo_next_b() refer to the next
invocation's a10 and b01, respectively, while in gemmtrsm_u, the
_next_a() and _next_b() macros return the addresses of the next
invocation's a11 and b11 (since those submatrices precede a12 and b21).
(See BLIS KernelsHowTo wiki for more info.)
- Zero alpha. The micro-kernel can safely assume that alpha is non-zero;
"alpha equals zero" handling is performed at a much higher level,
which means that, in such a scenario, the micro-kernel will never get
called.
- Diagonal elements of A11. See Implementation Notes for trsm.
- Zero elements of A11. See Implementation Notes for trsm.
- Output. See Implementation Notes for trsm.
- Optimization. Let's assume that the gemm micro-kernel has already been
optimized. You have two options with regard to optimizing the fused
gemmtrsm micro-kernels:
(1) Optimize only the trsm micro-kernels. This will result in the gemm
and trsm_l micro-kernels being called in sequence. (Likewise for
gemm and trsm_u.)
(2) Fuse the implementation of the gemm micro-kernel with that of the
trsm micro-kernels by inlining both into the gemmtrsm_l and
gemmtrsm_u micro-kernel definitions. This option is more labor-
intensive, but also more likely to yield higher performance because
it avoids redundant memory operations on the packed MR x NR
submatrix B11.
For more info, please refer to the BLIS website and/or contact the
blis-devel mailing list.
*/
const num_t dt = BLIS_DOUBLE;
const inc_t packnr = bli_cntx_get_blksz_max_dt( dt, BLIS_NR, cntx );
const inc_t rs_b = packnr;
const inc_t cs_b = 1;
double* restrict minus_one = bli_dm1;
/* b11 = alpha * b11 - a12 * b21; */
bli_dgemm_opt_mxn( k,
minus_one,
a12,
b21,
alpha,
b11, rs_b, cs_b,
data );
/* b11 = inv(a11) * b11;
c11 = b11; */
bli_dtrsm_u_opt_mxn( a11,
b11,
c11, rs_c, cs_c,
data );
}
void bli_cgemmtrsm_u_opt_mxn
(
dim_t k,
scomplex* restrict alpha,
scomplex* restrict a10,
scomplex* restrict a11,
scomplex* restrict b01,
scomplex* restrict b11,
scomplex* restrict c11, inc_t rs_c, inc_t cs_c,
auxinfo_t* restrict data,
cntx_t* restrict cntx
)
{
const num_t dt = BLIS_SCOMPLEX;
const inc_t packnr = bli_cntx_get_blksz_max_dt( dt, BLIS_NR, cntx );
const inc_t rs_b = packnr;
const inc_t cs_b = 1;
scomplex* restrict minus_one = bli_cm1;
bli_cgemm_opt_mxn( k,
minus_one,
a12,
b21,
alpha,
b11, rs_b, cs_b,
data );
bli_ctrsm_u_opt_mxn( a11,
b11,
c11, rs_c, cs_c,
data );
}
void bli_zgemmtrsm_u_opt_mxn
(
dim_t k,
dcomplex* restrict alpha,
dcomplex* restrict a10,
dcomplex* restrict a11,
dcomplex* restrict b01,
dcomplex* restrict b11,
dcomplex* restrict c11, inc_t rs_c, inc_t cs_c,
auxinfo_t* restrict data,
cntx_t* restrict cntx
)
{
const num_t dt = BLIS_DCOMPLEX;
const inc_t packnr = bli_cntx_get_blksz_max_dt( dt, BLIS_NR, cntx );
const inc_t rs_b = packnr;
const inc_t cs_b = 1;
dcomplex* restrict minus_one = bli_zm1;
bli_zgemm_opt_mxn( k,
minus_one,
a12,
b21,
alpha,
b11, rs_b, cs_b,
data );
bli_ztrsm_u_opt_mxn( a11,
b11,
c11, rs_c, cs_c,
data );
}
Reference in New Issue View Git Blame Copy Permalink