amd/blis
1
0
Fork 0
mirror of https://github.com/amd/blis.git synced 2026-05-24 10:24:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Move edge cases to gemmtrsm ukrs; doc updates.

Browse Source 
Details:
- Moved edge-case handling into the gemmtrsm microkernel. This required
  changing the microkernel API to take m and n dimension parameters as
  well as updating all existing gemmtrsm microkernel function pointer
  types, function signatures, and related definitions to take m and n
  dimensions. Also updated all existing gemmtrsm kernels in the
  'kernels' directory (which for now is limited to haswell and penryn
  kernel sets, plus native and 1m-based reference kernels in
  'ref_kernels') to take m and n dimensions, and implemented edge-case
  handling within those microkernels via a collection of new C
  preprocessor macros defined within bli_edge_case_macro_defs.h. Note
  that the edge-case handling for gemm-like operations had already
  been relocated into the gemm microkernel in 54fa28b.
- Added desriptive comments to GEMM_UKR_SETUP_CT() and related macros in
  bli_edge_case_macro_defs.h to allow for easier reading.
- Updated docs/KernelsHowTo.md to reflect above changes. Also cleaned up
  the bullet under "Implementation Notes for gemm" that covers alignment
  issues. (Thanks to Ivan Korostelev for pointing out the confusing and
  outdated language in issue #591.)
- Other minor tweaks to KernelsHowTo.md.
This commit is contained in:
Field G. Van Zee
2022-02-15 15:01:51 -06:00
parent 2506159346
commit ee9ff988c4
17 changed files with 350 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
docs/KernelsHowTo.md
View File

@@ -113,7 +113,7 @@ Note that all kernels, whether they be reference implementations or based on ful
The first step is to obtain a valid context. Contexts store all of the information
specific to a particular sub-configuration (usually loosely specific to a
microarchitecture or group of closely-related microarchitectuers). If a context is
microarchitecture or group of closely-related microarchitectures). If a context is
not already available in your current scope, a default context for the hardware
for which BLIS was configured (or, in the case of multi-configuration builds, the
hardware on which BLIS is currently running) may be queried via:
@@ -229,7 +229,7 @@ This section seeks to provide developers with a complete reference for each of t
The function prototypes in this section follow the same guidelines as those listed in the [BLIS typed API reference](BLISTypedAPI.md#Notes_for_using_this_reference). Namely:
* Any occurrence of `?` should be replaced with `s`, `d`, `c`, or `z` to form an actual function name.
* Any occurrence of `ctype` should be replaced with the actual C type corresponding to the datatype instance in question.
* Any occurrence of `ctype` should be replaced with the actual C99 language type corresponding to the datatype instance in question.
* Some matrix arguments have associated row and column strides arguments that proceed them, typically listed as `rsX` and `csX` for a given matrix `X`. Row strides are always listed first, and column strides are always listed second. The semantic meaning of a row stride is "the distance, in units of elements, from any given element to the corresponding element (within the same column) of the next row," and the meaning of a column stride is "the distance, in units of elements, from any given element to the corresponding element (within the same row) of the next column." Thus, unit row stride implies column-major storage and unit column stride implies row-major storage.
* All occurrences of `alpha` and `beta` parameters are scalars.
@@ -248,6 +248,8 @@ This section describes in detail the various level-3 microkernels supported by B
```c
void bli_?gemm_<suffix>
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a1,
@@ -264,6 +266,8 @@ where `<suffix>` is implementation-dependent. (Recall that the precise `<suffix>
```c
void bli_?gemm_ukernel
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a1,
@@ -274,6 +278,7 @@ void bli_?gemm_ukernel
cntx_t* restrict cntx
);
```
This function simply queries a microkernel function pointer from the context specified by `cntx`. Note that in the case of either method of calling the microkernel, `cntx` must be a valid pointer. (Passing in `NULL` will *not* result in a default context being used.)
The `gemm` microkernel, sometimes simply referred to as "the BLIS microkernel" or "the microkernel", performs the following operation:
@@ -281,16 +286,20 @@ The `gemm` microkernel, sometimes simply referred to as "the BLIS microkernel" o
C11 := beta * C11 + alpha * A1 * B1
```
where `A1` is an _MR x k_ "micropanel" matrix stored in packed (column-wise) format, `B1` is a _k x NR_ "micropanel" matrix stored in packed (row-wise) format, `C11` is an _MR x NR_ general matrix stored according to its row and column strides `rsc` and `csc`, and `alpha` and beta are scalars.
where `A1` is an _m x k_ "micropanel" matrix stored in packed (column-wise) format, `B1` is a _k x n_ "micropanel" matrix stored in packed (row-wise) format, `C11` is an _m x n_ "microtile" matrix stored according to its row and column strides `rsc` and `csc`, and `alpha` and beta are scalars.
_MR_ and _NR_ are the register blocksizes associated with the microkernel. They are chosen by the developer when the microkernel is written and then encoded into a BLIS configuration, which will reference the microkernel when the BLIS framework is instantiated into a library. For more information on setting register blocksizes and related constants, please see the [BLIS developer configuration guide](ConfigurationHowTo.md).
Here, _m <= MR_ and _n <= NR_, where _MR_ and _NR_ are the register blocksizes associated with the microkernel. They are chosen by the developer when the microkernel is written and then encoded into a BLIS configuration, which will reference the microkernel when the BLIS framework is instantiated into a library. For more information on setting register blocksizes and related constants, please see the [BLIS developer configuration guide](ConfigurationHowTo.md).
**Note:** For many years, BLIS defined its microkernel to operate on microtiles whose dimensions were *exactly* _MR x NR_. However, as of commit 54fa28b, we have augmented the `gemm` microkernel API to pass in _m_ and _n_ dimensions as well as _k_. This change was made as part of our decision to move edge-case handling into the microkernel, whereas previously it was handled outside of the microkernel, within the portable parts of BLIS framework. And while this does mean additional complexity for microkernel authors, adding generic edge-case handling can be done in a relatively painless manner by employing some pre-defined preprocessor macros (which are defined in `bli_edge_case_macro_defs.h`). For examples of how to use these macros, please see the beginning and end of existing microkernel functions residing within the `kernels` directory.
Parameters:
* `m`: The number of rows of `C11` and `A1`.
* `n`: The number of columns of `C11` and `B1`.
* `k`: The number of columns of `A1` and rows of `B1`.
* `alpha`: The address of a scalar to the `A1 * B1` product.
* `a1`: The address of a micropanel of matrix `A` of dimension _MR x k_, stored by columns with leading dimension _PACKMR_, where typically _PACKMR_ = _MR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKMR_.)
* `b1`: The address of a micropanel of matrix `B` of dimension _k x NR_, stored by rows with leading dimension _PACKNR_, where typically _PACKNR_ = _NR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKNR_.)
* `a1`: The address of a micropanel of matrix `A` of dimension _m x k_ (where _m <= MR_), stored by columns with leading dimension _PACKMR_, where typically _PACKMR_ = _MR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKMR_.)
* `b1`: The address of a micropanel of matrix `B` of dimension _k x n_ (where _n <= NR_), stored by rows with leading dimension _PACKNR_, where typically _PACKNR_ = _NR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKNR_.)
* `beta`: The address of a scalar to the input value of matrix `C11`.
* `c11`: The address of a matrix `C11` of dimension _MR x NR_, stored according to `rsc` and `csc`.
* `rsc`: The row stride of matrix `C11` (ie: the distance to the next row, in units of matrix elements).
@@ -321,24 +330,24 @@ The diagram below shows the packed micropanel operands and how elements of each
#### Implementation Notes for gemm
* **Register blocksizes.** The register blocksizes `MR` and `NR`, corresponding to the number of *logical* rows in `a1` and columns in `b1`, respectively, are defined in the context and may be queried via `bli_cntx_get_blksz_def_dt()`. However, you shouldn't need to query these values since the implementation inherently "knows" them already.
* **Leading dimensions of `a1` and `b1`: _PACKMR_ and _PACKNR_.** The packed micropanels `a1` and `b1` are simply stored in column-major and row-major order, respectively. Usually, the width of either micropanel (ie: the number of logical rows of `a1`, or _MR_, and the number of columns of `b1`, or _NR_) is equal to that micropanel's so-called "leading dimension", or number of *physical* rows. Sometimes, it may be beneficial to specify a leading dimension that is larger than the panel width. This may be desirable because it allows each column of `a1` or row of `b1` to maintain a certain alignment in memory that would not otherwise be maintained by _MR_ and/or _NR_. In this case, you should index through `a1` and `b1` using the values _PACKMR_ and _PACKNR_, respectively (which are stored in the context as the blocksize "maximums" associated with the `bszid_t` values `BLIS_MR` and `BLIS_NR`). These values are defined in the context and may be queried via `bli_cntx_get_blksz_max_dt()`. However, you shouldn't need to query these values since the implementation inherently "knows" them already.
* **Register blocksizes.** The register blocksizes `MR` and `NR`, corresponding to the maximum number of *logical* rows in `a1` and columns in `b1`, respectively, are defined in the context and may be queried via `bli_cntx_get_blksz_def_dt()`. However, you shouldn't need to query these values since the implementation inherently "knows" them already.
* **Leading dimensions of `a1` and `b1`: _PACKMR_ and _PACKNR_.** The packed micropanels `a1` and `b1` are simply stored in column-major and row-major order, respectively. Usually, the width of either micropanel (ie: the number of *logical* rows of `a1` and the number of columns of `b1`) is equal to that micropanel's so-called "leading dimension", or number of *physical* rows. Sometimes, it may be beneficial to specify a leading dimension that is larger than the panel width. This may be desirable because it allows each column of `a1` or row of `b1` to maintain a certain alignment in memory that would not otherwise be maintained by _MR_ and/or _NR_, which would othewise serve as the maximum value for each micropanel, respectively. If you want your microkernel to support _MR < PACKMR_ or _NR < PACKNR_, you should index through columns of `a1` and rows of `b1` using the values _PACKMR_ and _PACKNR_, respectively (which are stored in the context as the blocksize "maximums" associated with the `bszid_t` values `BLIS_MR` and `BLIS_NR`). These values are defined in the context and may be queried via `bli_cntx_get_blksz_max_dt()`. However, you shouldn't need to query these values since the microkernel implementation inherently must "know" them already.
* **Storage preference of `c11`.** Usually, an optimized `gemm` microkernel will have a "preferred" storage format for `C11`--typically either contiguous row-storage (i.e. `cs_c` = 1) or contiguous column-storage (i.e. `rs_c` = 1). This preference comes from how the microkernel is most efficiently able to load/store elements of `C11` from/to memory. Most microkernels use vector instructions to access contiguous columns (or column segments) of `C11`. However, the developer may decide that accessing contiguous rows (or row segments) is more desirable. If this is the case, this preference should be indicated via the `bool` argument when registering microkernels via `bli_cntx_set_l3_nat_ukrs()`--`TRUE` indicating a row preference and `FALSE` indicating a column preference. Properly setting this property allows the framework to perform a runtime optimization that will ensure the microkernel preference is honored, if at all possible.
* **Edge cases in _MR_, _NR_ dimensions.** Sometimes the microkernel will be called with micropanels `a1` and `b1` that correspond to edge cases, where only partial results are needed. Zero-padding is handled automatically by the packing function to facilitate reuse of the same microkernel. Similarly, the logic for computing to temporary storage and then saving only the elements that correspond to elements of `C11` that exist (at the edges) is handled automatically within the macrokernel.
* **Alignment of `a1` and `b1`.** By default, the alignment of addresses `a1` and `b1` are aligned only to `sizeof(type)`. If `BLIS_POOL_ADDR_ALIGN_SIZE` is set to some larger multiple of `sizeof(type)`, such as the page size, then the *first* `a1` and `b1` micropanels will be aligned to that value, but subsequent micropanels will only be aligned to `sizeof(type)`, or, if `BLIS_POOL_ADDR_ALIGN_SIZE` is a multiple of `PACKMR` and `PACKNR`, then subsequent micropanels `a1` and `b1` will be aligned to `PACKMR * sizeof(type)` and `PACKNR * sizeof(type)`, respectively.
* **Unrolling loops.** As a general rule of thumb, the loop over _k_ is sometimes moderately unrolled; for example, in our experience, an unrolling factor of _u_ = 4 is fairly common. If unrolling is applied in the _k_ dimension, edge cases must be handled to support values of _k_ that are not multiples of _u_. It is nearly universally true that there should be no loops in the _MR_ or _NR_ directions; in other words, iteration over these dimensions should always be fully unrolled (within the loop over _k_).
* **Edge cases in _MR_, _NR_ dimensions.** Sometimes the microkernel will be called with micropanels `a1` and `b1` that correspond to edge cases, where only partial results are needed. This edge-case handling was once performed by the framework automatically. However, as of commit 54fa28b, edge-case handling is the responsiblity of the microkernel. This means that the kernel author will need to handle all possible values of _m_ and _n_ that are equal to **or** less than _MR_ and _NR_, respectively. Fortunately, this can be implemented outside of the assembly region of the microkernel with preprocessor macros. Please reference the existing microkernels in the `kernels` directory for examples of how this is done. (The macros that are now employed by most of BLIS's microkernels are defined in `bli_edge_case_macro_defs.h`.)
* **Alignment of `a1` and `b1`.** By default, the alignment of addresses `a1` and `b1` are aligned to the page size (4096 bytes). These alignment factors are set by `BLIS_POOL_ADDR_ALIGN_SIZE_A` and `BLIS_POOL_ADDR_ALIGN_SIZE_B`, respectively. Note that these alignment factors control only the alignment of the *first* micropanel within a given packed blockof matrix `A` or packed row-panel of matrix `B`. Subsequent micropanels will only be aligned to `sizeof(type)`, or, if `BLIS_POOL_ADDR_ALIGN_SIZE_A` is a multiple of `PACKMR` and/or `BLIS_POOL_ADDR_ALIGN_SIZE_B` is a multiple of `PACKNR`, then subsequent micropanels `a1` and/or `b1` will be aligned to `PACKMR * sizeof(type)` and/or `PACKNR * sizeof(type)`, respectively.
* **Unrolling loops.** As a general rule of thumb, the loop over _k_ is sometimes moderately unrolled; for example, in our experience, an unrolling factor of _u_ = 4 is fairly common. If unrolling is applied in the _k_ dimension, edge cases must be handled to support values of _k_ that are not multiples of _u_. It is nearly universally true that the microkernel should not contain loops in the _m_ or _n_ directions; in other words, iteration over these dimensions should always be fully unrolled (within the loop over _k_).
* **Zero `beta`.** If `beta` = 0.0 (or 0.0 + 0.0i for complex datatypes), then the microkernel should NOT use it explicitly, as `C11` may contain uninitialized memory (including elements containing `NaN` or `Inf`). This case should be detected and handled separately by overwriting `C11` with the `alpha * A1 * B1` product.
#### Using the auxinfo\_t object
Each microkernel ([gemm](KernelsHowTo.md#gemm-microkernel), [trsm](KernelsHowTo.md#trsm_microkernels), and [gemmtrsm](KernelsHowTo.md#gemmtrsm-microkernels)) takes as its last argument a pointer of type `auxinfo_t`. This BLIS-defined type is defined as a `struct` whose fields contain auxiliary values that may be useful to some microkernel authors, particularly when implementing certain optimization techniques. BLIS provides kernel authors access to the fields of the `auxinfo_t` object via the following function-like preprocessor macros. Each macro takes a single argument, the `auxinfo_t` pointer, and returns one of the values stored within the object.
Each microkernel ([gemm](KernelsHowTo.md#gemm-microkernel), [trsm](KernelsHowTo.md#trsm_microkernels), and [gemmtrsm](KernelsHowTo.md#gemmtrsm-microkernels)) takes as its last argument a pointer of type `auxinfo_t`. This BLIS-defined type is defined as a `struct` whose fields contain auxiliary values that may be useful to some microkernel authors, particularly when implementing certain optimization techniques. BLIS provides kernel authors access to the fields of the `auxinfo_t` object via the following static inline functions. Each function takes a single argument, the `auxinfo_t` pointer, and returns one of the values stored within the object.
* `bli_auxinfo_next_a()`. Returns the address (`void*`) of the micropanel of `A` that will be used the next time the microkernel will be called.
* `bli_auxinfo_next_b()`. Returns the address (`void*`) of the micropanel of `B` that will be used the next time the microkernel will be called.
* `bli_auxinfo_ps_a()`. Returns the panel stride (`inc_t`) of the current micropanel of `A`.
* `bli_auxinfo_ps_b()`. Returns the panel stride (`inc_t`) of the current micropanel of `B`.
The addresses of the next micropanels of `A` and `B` may be used by the microkernel to perform prefetching, if prefetching is supported by the architecture. Similarly, it may be useful to know the precise distance in memory to the next micropanel. (Note that sometimes the next micropanel to be used is **not** the same as the next micropanel in memory.)
The addresses of the next micropanels of `A` and `B` may be used by the microkernel to perform prefetching, if prefetching is supported by the architecture. Similarly, it may be useful to know the precise distance in memory to the next micropanel. (Note that occasionally the next micropanel to be used is **not** the same as the next micropanel in memory.)
Any and all of these values may be safely ignored; they are completely optional. However, BLIS guarantees that all values accessed via the macros listed above will **always** be initialized and meaningful, for every invocation of each microkernel (`gemm`, `trsm`, and `gemmtrsm`).
@@ -348,8 +357,7 @@ Any and all of these values may be safely ignored; they are completely optional.
An example implementation of the `gemm` microkernel may be found in the `template` configuration directory in:
* [config/template/kernels/3/bli\_gemm_opt\_mxn.c](https://github.com/flame/blis/tree/master/config/template/kernels/3/bli_gemm_opt_mxn.c)
Note that this implementation is coded in C99 and lacks several kinds of optimization that are typical of real-world optimized microkernels, such as vector instructions (or intrinsics) and loop unrolling in _MR_ or _NR_. It is meant to serve only as a starting point for a microkernel developer.
Note that this implementation is coded in C99 and lacks several kinds of optimization that are typical of real-world optimized microkernels, such as vector instructions (or intrinsics) and loop unrolling in the _m_ or _n_ dimensions. It is meant to serve only as a starting point for a microkernel developer.
@@ -411,6 +419,8 @@ where `A11` is _MR x MR_ and lower (`trsm_l`) or upper (`trsm_u`) triangular, `B
_MR_ and _NR_ are the register blocksizes associated with the microkernel. They are chosen by the developer when the microkernel is written and then encoded into a BLIS configuration, which will reference the microkernel when the BLIS framework is instantiated into a library. For more information on setting register blocksizes and related constants, please see the [BLIS developer configuration guide](ConfigurationHowTo.md).
**Note:** Although the `gemm` microkernel must handle edge-cases, and therefore must take _m_ and _n_ parameters, the `trsm` microkernels are simpler in that they still assume _m = MR_ and _n = NR_, and therefore do not need these _m_ and _n_ parameters passed in.
Parameters:
* `a11`: The address of `A11`, which is the _MR x MR_ lower (`trsm_l`) or upper (`trsm_u`) triangular submatrix within the packed micropanel of matrix `A`. `A11` is stored by columns with leading dimension _PACKMR_, where typically _PACKMR_ = _MR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKMR_.) Note that `A11` contains elements in both triangles, though elements in the unstored triangle are not guaranteed to be zero and thus should not be referenced.
@@ -454,6 +464,8 @@ Note that these implementations are coded in C99 and lack several kinds of optim
```c
void bli_?gemmtrsm_l_<suffix>
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a10,
@@ -467,6 +479,8 @@ void bli_?gemmtrsm_l_<suffix>
void bli_?gemmtrsm_u_<suffix>
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a12,
@@ -484,6 +498,8 @@ where `<suffix>` is implementation-dependent. (Recall that the precise `<suffix>
```c
void bli_?gemmtrsm_l_ukernel
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a10,
@@ -497,6 +513,8 @@ void bli_?gemmtrsm_l_ukernel
void bli_?gemmtrsm_u_ukernel
(
dim_t m,
dim_t n,
dim_t k,
ctype* restrict alpha,
ctype* restrict a12,
@@ -517,7 +535,7 @@ The `gemmtrsm_l` microkernel performs the following compound operation:
C11 := B11
```
where `A11` is _MR_ x _MR_ and lower triangular, `A10` is _MR_ x _k_, and `B01` is _k_ x _NR_.
where `A11` is _MR x MR_ and lower triangular, `A10` is _MR x k_, and `B01` is _k x NR_.
The `gemmtrsm_u` microkernel performs:
```
@@ -526,20 +544,22 @@ The `gemmtrsm_u` microkernel performs:
C11 := B11
```
where `A11` is _MR_ x _MR_ and upper triangular, `A12` is _MR_ x _k_, and `B21` is _k_ x _NR_.
In both cases, `B11` is _MR_ x _NR_ and `alpha` is a scalar. Here, `inv()` denotes matrix inverse.
where `A11` is _MR x MR_ and upper triangular, `A12` is _MR x k_, and `B21` is _k x NR_.
In both cases, `B11` is _MR x NR_ and `alpha` is a scalar. However, `C11` is _m x n_, and therefore the `C11 := B11` statements amount to a copy of only the top-leftmost _m x n_ elements of `B11`. (Recall that A11 and B11 are packed and therefore guaranteed to reside within fully-sized micropanels, whereas `C11` exists in the caller-provided output matrix and may represent a bottom-right edge case.) Here, `inv()` denotes matrix inverse.
_MR_ and _NR_ are the register blocksizes associated with the microkernel. They are chosen by the developer when the microkernel is written and then encoded into a BLIS configuration, which will reference the microkernel when the BLIS framework is instantiated into a library. For more information on setting register blocksizes and related constants, please see the [BLIS developer configuration guide](ConfigurationHowTo.md).
Parameters:
* `m`: The number of rows of `C11`.
* `n`: The number of columns of `C11`.
* `k`: The number of columns of `A10` and rows of `B01` (`trsm_l`); the number of columns of `A12` and rows of `B21` (`trsm_u`).
* `alpha`: The address of a scalar to be applied to `B11`.
* `a10`, `a12`: The address of `A10` or `A12`, which is the _MR x k_ submatrix of the packed micropanel of `A` that is situated to the left (`trsm_l`) or right (`trsm_u`) of the _MR x MR_ triangular submatrix `A11`. `A10` and `A12` are stored by columns with leading dimension _PACKMR_, where typically _PACKMR_ = _MR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKMR_.)
* `a11`: The address of `A11`, which is the _MR x MR_ lower (`trsm_l`) or upper (`trsm_u`) triangular submatrix within the packed micropanel of matrix `A` that is situated to the right of `A10` (`trsm_l`) or the left of `A12` (`trsm_u`). `A11` is stored by columns with leading dimension _PACKMR_, where typically _PACKMR_ = _MR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKMR_.) Note that `A11` contains elements in both triangles, though elements in the unstored triangle are not guaranteed to be zero and thus should not be referenced.
* `b01`, `b21`: The address of `B01` and `B21`, which is the _k x NR_ submatrix of the packed micropanel of `B` that is situated above (`trsm_l`) or below (`trsm_u`) the _MR x NR_ block `B11`. `B01` and `B21` are stored by rows with leading dimension _PACKNR_, where typically _PACKNR_ = _NR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKNR_.)
* `b11`: The address of `B11`, which is the _MR x NR_ submatrix of the packed micropanel of `B`, situated below `B01` (`trsm_l`) or above `B21` (`trsm_u`). `B11` is stored by rows with leading dimension _PACKNR_, where typically _PACKNR_ = _NR_. (See [Implementation Notes for gemm](KernelsHowTo.md#implementation-notes-for-gemm) for a discussion of _PACKNR_.)
* `c11`: The address of `C11`, which is an _MR x NR_ submatrix of matrix `C`, stored according to `rsc` and `csc`. `C11` is the submatrix within `C` that corresponds to the elements which were packed into `B11`. Thus, `C` is the original input matrix `B` to the overall `trsm` operation.
* `c11`: The address of `C11`, which is an _m x n_ submatrix of matrix `C`, stored according to `rsc` and `csc`, where _m <= MR_ and _n <= NR_. `C11` is the submatrix within `C` that corresponds to the elements which were packed into `B11`. Thus, `C` is the original input matrix `B` to the overall `trsm` operation.
* `rsc`: The row stride of matrix `C11` (ie: the distance to the next row, in units of matrix elements).
* `csc`: The column stride of matrix `C11` (ie: the distance to the next column, in units of matrix elements).
* `data`: The address of an `auxinfo_t` object that contains auxiliary information that may be useful when optimizing the `gemmtrsm` microkernel implementation. (See [Using the auxinfo\_t object](KernelsHowTo.md#Using_the_auxinfo_t_object) for a discussion of the kinds of values available via `auxinfo_t`, and also [Implementation Notes for gemmtrsm](KernelsHowTo.md#implementation-notes-for-gemmtrsm) for caveats.)
@@ -690,7 +710,7 @@ This kernel performs the following operation:
```
y := y + alpha * conja(a) * conjy(x)
```
where `a` is an _m_ x _b_ matrix, `x` is a vector of length _b_, and `y` is a vector of length _m_. Vectors `x` and `y` are stored with strides `incx` and `incy`, respectively. Matrix `a` is stored with row stride `inca` and column stride `lda`, though `inca` is most often (in practice) unit. This kernel is typically implemented as a fused series of _b_ `axpyv` operations updating the same vector `y` (with the elements of `x` serving as the scalars and the columns of `a` serving as the vectors to be scaled).
where `a` is an _m x b_ matrix, `x` is a vector of length _b_, and `y` is a vector of length _m_. Vectors `x` and `y` are stored with strides `incx` and `incy`, respectively. Matrix `a` is stored with row stride `inca` and column stride `lda`, though `inca` is most often (in practice) unit. This kernel is typically implemented as a fused series of _b_ `axpyv` operations updating the same vector `y` (with the elements of `x` serving as the scalars and the columns of `a` serving as the vectors to be scaled).
---
@@ -714,7 +734,7 @@ This kernel performs the following operation:
```
y := beta * y + alpha * conjat(a)^T conjx(x)
```
where `a` is an _m_ x _b_ matrix, where `w` is a vector of length _m_, `y` is a vector of length _b_, and `alpha` is a scalar.
where `a` is an _m x b_ matrix, where `w` is a vector of length _m_, `y` is a vector of length _b_, and `alpha` is a scalar.
Vectors `x` and `y` are stored with strides `incx` and `incy`, respectively. Matrix `a` is stored with row stride `inca` and column stride `lda`, though `inca` is most often (in practice) unit.
This kernel is typically implemented as a series of _b_ `dotxv` operations with the same right-hand operand vector `x` (contracted with the rows of `a^T` and accumulating to the corresponding elements of vector `y`).
@@ -745,7 +765,7 @@ This kernel performs the following operation:
y := beta * y + alpha * conjat(a)^T conjw(w)
z := z + alpha * conja(a) conjx(x)
```
where `a` is an _m_ x _b_ matrix, `w` and `z` are vectors of length _m_, `x` and `y` are vectors of length _b_, and `alpha` and `beta` are scalars.
where `a` is an _m x b_ matrix, `w` and `z` are vectors of length _m_, `x` and `y` are vectors of length _b_, and `alpha` and `beta` are scalars.
Vectors `w`, `z`, `x` and `y` are stored with strides `incw`, `incz`, `incx`, and `incy`, respectively. Matrix `a` is stored with row stride `inca` and column stride `lda`, though `inca` is most often (in practice) unit.
This kernel is typically implemented as a series of _b_ `dotxv` operations with the same right-hand operand vector `w` fused with a series of _b_ `axpyv` operations updating the same vector `z`.

2
frame/3/bli_l3_ft_ukr.h
View File

@@ -69,6 +69,8 @@ INSERT_GENTDEF( gemm )
\
typedef void (*PASTECH3(ch,opname,_ukr,tsuf)) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \

4
frame/3/bli_l3_ind_ukr.h
View File

@@ -43,6 +43,8 @@
\
void PASTEMAC(ch,opname) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a, \
@@ -61,6 +63,8 @@ INSERT_GENTPROT_BASIC0( gemm1m_ukr_name )
\
void PASTEMAC(ch,opname) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \

6
frame/3/bli_l3_ukr_oapi.c
View File

@@ -111,6 +111,8 @@ void PASTEMAC0(opname) \
\
num_t dt = bli_obj_dt( c11 ); \
\
dim_t m = bli_obj_length( c11 ); \
dim_t n = bli_obj_width( c11 ); \
dim_t k = bli_obj_width( a1x ); \
void* buf_a1x = bli_obj_buffer_at_off( a1x ); \
void* buf_a11 = bli_obj_buffer_at_off( a11 ); \
@@ -140,6 +142,8 @@ void PASTEMAC0(opname) \
\
f \
( \
m, \
n, \
k, \
buf_alpha, \
buf_a1x, \
@@ -160,6 +164,8 @@ void PASTEMAC0(opname) \
\
f \
( \
m, \
n, \
k, \
buf_alpha, \
buf_a1x, \

2
frame/3/bli_l3_ukr_prot.h
View File

@@ -59,6 +59,8 @@ void PASTEMAC(ch,opname) \
\
void PASTEMAC(ch,opname) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \

4
frame/3/bli_l3_ukr_tapi.c
View File

@@ -83,6 +83,8 @@ INSERT_GENTFUNC_BASIC2( gemm_ukernel, gemm, BLIS_GEMM_UKR )
\
void PASTEMAC(ch,opname) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \
@@ -105,6 +107,8 @@ void PASTEMAC(ch,opname) \
/* Invoke the typed function for the given datatype. */ \
f \
( \
m, \
n, \
k, \
alpha, \
a1x, \

58
frame/3/trsm/bli_trsm_ll_ker_var2.c
View File

@@ -176,12 +176,14 @@ void PASTEMAC(ch,varname) \
temporary buffer are set so that they match the storage of the
original C matrix. For example, if C is column-stored, ct will be
column-stored as well. */ \
/*
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
*/ \
\
ctype* restrict minus_one = PASTEMAC(ch,m1); \
ctype* restrict a_cast = a; \
@@ -276,10 +278,6 @@ void PASTEMAC(ch,varname) \
know that the underlying buffer was already allocated to have an m
dimension that is a multiple of PACKMR, with the region between the
last row and the next multiple of MR zero-padded accordingly. */ \
\
/* Clear the temporary C buffer in case it has any infs or NaNs. */ \
PASTEMAC(ch,set0s_mxn)( MR, NR, \
ct, rs_ct, cs_ct ); \
\
/* Compute number of primary and leftover components of the m and n
dimensions. */ \
@@ -409,44 +407,20 @@ void PASTEMAC(ch,varname) \
bli_auxinfo_set_next_a( a2, &aux ); \
bli_auxinfo_set_next_b( b2, &aux ); \
\
/* Handle interior and edge cases separately. */ \
if ( m_cur == MR && n_cur == NR ) \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_a10, \
alpha1_cast, \
a10, \
a11, \
b01, \
b11, \
c11, rs_c, cs_c, \
&aux, \
cntx \
); \
} \
else \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_a10, \
alpha1_cast, \
a10, \
a11, \
b01, \
b11, \
ct, rs_ct, cs_ct, \
&aux, \
cntx \
); \
\
/* Copy the result to the bottom edge of C. */ \
PASTEMAC(ch,copys_mxn)( m_cur, n_cur, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c ); \
} \
gemmtrsm_ukr \
( \
m_cur, \
n_cur, \
k_a10, \
alpha1_cast, \
a10, \
a11, \
b01, \
b11, \
c11, rs_c, cs_c, \
&aux, \
cntx \
); \
\
a1 += ps_a_cur; \
} \

58
frame/3/trsm/bli_trsm_lu_ker_var2.c
View File

@@ -176,12 +176,14 @@ void PASTEMAC(ch,varname) \
temporary buffer are set so that they match the storage of the
original C matrix. For example, if C is column-stored, ct will be
column-stored as well. */ \
/*
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
*/ \
\
ctype* restrict minus_one = PASTEMAC(ch,m1); \
ctype* restrict a_cast = a; \
@@ -284,10 +286,6 @@ void PASTEMAC(ch,varname) \
know that the underlying buffer was already allocated to have an m
dimension that is a multiple of PACKMR, with the region between the
last row and the next multiple of MR zero-padded accordingly. */ \
\
/* Clear the temporary C buffer in case it has any infs or NaNs. */ \
PASTEMAC(ch,set0s_mxn)( MR, NR, \
ct, rs_ct, cs_ct ); \
\
/* Compute number of primary and leftover components of the m and n
dimensions. */ \
@@ -419,44 +417,20 @@ void PASTEMAC(ch,varname) \
bli_auxinfo_set_next_a( a2, &aux ); \
bli_auxinfo_set_next_b( b2, &aux ); \
\
/* Handle interior and edge cases separately. */ \
if ( m_cur == MR && n_cur == NR ) \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_a12, \
alpha1_cast, \
a12, \
a11, \
b21, \
b11, \
c11, rs_c, cs_c, \
&aux, \
cntx \
); \
} \
else \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_a12, \
alpha1_cast, \
a12, \
a11, \
b21, \
b11, \
ct, rs_ct, cs_ct, \
&aux, \
cntx \
); \
\
/* Copy the result to the bottom edge of C. */ \
PASTEMAC(ch,copys_mxn)( m_cur, n_cur, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c ); \
} \
gemmtrsm_ukr \
( \
m_cur, \
n_cur, \
k_a12, \
alpha1_cast, \
a12, \
a11, \
b21, \
b11, \
c11, rs_c, cs_c, \
&aux, \
cntx \
); \
\
a1 += ps_a_cur; \
} \

58
frame/3/trsm/bli_trsm_rl_ker_var2.c
View File

@@ -181,12 +181,14 @@ void PASTEMAC(ch,varname) \
temporary buffer are set so that they match the storage of the
original C matrix. For example, if C is column-stored, ct will be
column-stored as well. */ \
/*
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
*/ \
\
ctype* restrict minus_one = PASTEMAC(ch,m1); \
ctype* restrict a_cast = a; \
@@ -302,10 +304,6 @@ void PASTEMAC(ch,varname) \
know that the underlying buffer was already allocated to have an n
dimension that is a multiple of PACKNR, with the region between the
last column and the next multiple of NR zero-padded accordingly. */ \
\
/* Clear the temporary C buffer in case it has any infs or NaNs. */ \
PASTEMAC(ch,set0s_mxn)( MR, NR, \
ct, rs_ct, cs_ct ); \
\
/* Compute number of primary and leftover components of the m and n
dimensions. */ \
@@ -424,44 +422,21 @@ void PASTEMAC(ch,varname) \
bli_auxinfo_set_next_a( b2, &aux ); \
bli_auxinfo_set_next_b( a2, &aux ); \
\
/* Handle interior and edge cases separately. */ \
if ( m_cur == MR && n_cur == NR ) \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_b21, \
alpha1_cast, \
b21, \
b11, \
a12, \
a11, \
c11, cs_c, rs_c, \
&aux, \
cntx \
); \
} \
else \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_b21, \
alpha1_cast, \
b21, \
b11, \
a12, \
a11, \
ct, cs_ct, rs_ct, \
&aux, \
cntx \
); \
gemmtrsm_ukr \
( \
m_cur, \
n_cur, \
k_b21, \
alpha1_cast, \
b21, \
b11, \
a12, \
a11, \
c11, cs_c, rs_c, \
&aux, \
cntx \
); \
\
/* Copy the result to the bottom edge of C. */ \
PASTEMAC(ch,copys_mxn)( m_cur, n_cur, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c ); \
} \
} \
\
a1 += rstep_a; \
@@ -512,6 +487,7 @@ void PASTEMAC(ch,varname) \
&aux, \
cntx \
); \
\
} \
\
a1 += rstep_a; \

58
frame/3/trsm/bli_trsm_ru_ker_var2.c
View File

@@ -181,12 +181,14 @@ void PASTEMAC(ch,varname) \
temporary buffer are set so that they match the storage of the
original C matrix. For example, if C is column-stored, ct will be
column-stored as well. */ \
/*
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
*/ \
\
ctype* restrict minus_one = PASTEMAC(ch,m1); \
ctype* restrict a_cast = a; \
@@ -297,10 +299,6 @@ void PASTEMAC(ch,varname) \
know that the underlying buffer was already allocated to have an n
dimension that is a multiple of PACKNR, with the region between the
last column and the next multiple of NR zero-padded accordingly. */ \
\
/* Clear the temporary C buffer in case it has any infs or NaNs. */ \
PASTEMAC(ch,set0s_mxn)( MR, NR, \
ct, rs_ct, cs_ct ); \
\
/* Compute number of primary and leftover components of the m and n
dimensions. */ \
@@ -417,44 +415,21 @@ void PASTEMAC(ch,varname) \
bli_auxinfo_set_next_a( b2, &aux ); \
bli_auxinfo_set_next_b( a2, &aux ); \
\
/* Handle interior and edge cases separately. */ \
if ( m_cur == MR && n_cur == NR ) \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_b01, \
alpha1_cast, \
b01, \
b11, \
a10, \
a11, \
c11, cs_c, rs_c, \
&aux, \
cntx \
); \
} \
else \
{ \
/* Invoke the fused gemm/trsm micro-kernel. */ \
gemmtrsm_ukr \
( \
k_b01, \
alpha1_cast, \
b01, \
b11, \
a10, \
a11, \
ct, cs_ct, rs_ct, \
&aux, \
cntx \
); \
gemmtrsm_ukr \
( \
m_cur, \
n_cur, \
k_b01, \
alpha1_cast, \
b01, \
b11, \
a10, \
a11, \
c11, cs_c, rs_c, \
&aux, \
cntx \
); \
\
/* Copy the result to the bottom edge of C. */ \
PASTEMAC(ch,copys_mxn)( m_cur, n_cur, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c ); \
} \
} \
\
a1 += rstep_a; \
@@ -505,6 +480,7 @@ void PASTEMAC(ch,varname) \
&aux, \
cntx \
); \
\
} \
\
a1 += rstep_a; \

110
frame/include/bli_edge_case_macro_defs.h
View File

@@ -35,8 +35,11 @@
#ifndef BLIS_EDGE_CASE_MACRO_DEFS_H
#define BLIS_EDGE_CASE_MACRO_DEFS_H
//
// Macros for edge-case handling within gemm microkernels.
//
// Helper macros for edge-case handling within gemm microkernels.
// -- Setup helper macros --
#define GEMM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,alignment) \
\
@@ -62,8 +65,14 @@
beta = &_zero; \
}
// -- Setup macros --
#define GEMM_UKR_SETUP_CT(ch,mr,nr,row_major) \
\
/* Scenario 1: the ukernel contains assembly-level support only for its
IO preference (e.g. only row-oriented or only column-oriented IO).
Use a temporary microtile for the other two cases as well as edge
cases. */ \
GEMM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = ( row_major ? cs_c != 1 : rs_c != 1 ) || \
m != mr || n != nr; \
@@ -71,6 +80,10 @@
#define GEMM_UKR_SETUP_CT_AMBI(ch,mr,nr,row_major) \
\
/* Scenario 2: the ukernel contains assembly-level support for its IO
preference as well as its opposite via in-register transpose
(e.g. both row- and column-oriented IO). Use a temporary microtile
for the general stride case as well as edge cases. */ \
GEMM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = ( cs_c != 1 && rs_c != 1 ) || \
m != mr || n != nr; \
@@ -78,12 +91,16 @@
#define GEMM_UKR_SETUP_CT_ANY(ch,mr,nr,row_major) \
\
/* Scenario 3: Similar to (2) where the assembly region also supports
general stride I0. Use a temporary microtile only for edge cases. */ \
GEMM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = m != mr || n != nr; \
const bool _use_ct = ( m != mr || n != nr ); \
GEMM_UKR_SETUP_CT_POST(ch);
#define GEMM_UKR_SETUP_CT_ALIGNED(ch,mr,nr,row_major,alignment) \
\
/* Scenario 4: Similar to (1), but uses temporary microtile to handle
cases where the pointer to the C microtile is not aligned. */ \
GEMM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,alignment); \
const bool _use_ct = ( row_major ? cs_c != 1 : rs_c != 1 ) || \
m != mr || n != nr || \
@@ -91,8 +108,12 @@
( ( ( row_major ? _rs_c : _cs_c )*sizeof( PASTEMAC(ch,ctype) ) ) % alignment ); \
GEMM_UKR_SETUP_CT_POST(ch);
// -- Flush macros --
#define GEMM_UKR_FLUSH_CT(ch) \
\
/* If we actually used the temporary microtile, accumulate it to the output
microtile. */ \
if ( _use_ct ) \
{ \
PASTEMAC(ch,xpbys_mxn) \
@@ -105,5 +126,90 @@
} \
//
// Macros for edge-case handling within gemmtrsm microkernels.
//
// -- Setup helper macros --
#define GEMMTRSM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,alignment) \
\
PASTEMAC(ch,ctype)* restrict _c = c11; \
const inc_t _rs_c = rs_c; \
const inc_t _cs_c = cs_c; \
PASTEMAC(ch,ctype) _ct[ BLIS_STACK_BUF_MAX_SIZE / sizeof( PASTEMAC(ch,type) ) ] \
__attribute__((aligned(alignment))); \
const inc_t _rs_ct = row_major ? nr : 1; \
const inc_t _cs_ct = row_major ? 1 : mr;
#define GEMMTRSM_UKR_SETUP_CT_POST(ch) \
\
if ( _use_ct ) \
{ \
c11 = _ct; \
rs_c = _rs_ct; \
cs_c = _cs_ct; \
}
// -- Setup macros --
#define GEMMTRSM_UKR_SETUP_CT(ch,mr,nr,row_major) \
\
/* Scenario 1: the ukernel contains assembly-level support only for its
IO preference (e.g. only row-oriented or only column-oriented IO).
Use a temporary microtile for the other two cases as well as edge
cases. */ \
GEMMTRSM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = ( row_major ? cs_c != 1 : rs_c != 1 ) || \
m != mr || n != nr; \
GEMMTRSM_UKR_SETUP_CT_POST(ch);
#define GEMMTRSM_UKR_SETUP_CT_AMBI(ch,mr,nr,row_major) \
\
/* Scenario 2: the ukernel contains assembly-level support for its IO
preference as well as its opposite via in-register transpose
(e.g. both row- and column-oriented IO). Use a temporary microtile
for the general stride case as well as edge cases. */ \
GEMMTRSM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = ( cs_c != 1 && rs_c != 1 ) || \
m != mr || n != nr; \
GEMMTRSM_UKR_SETUP_CT_POST(ch);
#define GEMMTRSM_UKR_SETUP_CT_ANY(ch,mr,nr,row_major) \
\
/* Scenario 3: Similar to (2) where the assembly region also supports
general stride I0. Use a temporary microtile only for edge cases. */ \
GEMMTRSM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,1); \
const bool _use_ct = ( m != mr || n != nr ); \
GEMMTRSM_UKR_SETUP_CT_POST(ch);
#define GEMMTRSM_UKR_SETUP_CT_ALIGNED(ch,mr,nr,row_major,alignment) \
\
/* Scenario 4: Similar to (1), but uses temporary microtile to handle
cases where the pointer to the C microtile is not aligned. */ \
GEMMTRSM_UKR_SETUP_CT_PRE(ch,mr,nr,row_major,alignment); \
const bool _use_ct = ( row_major ? cs_c != 1 : rs_c != 1 ) || \
m != mr || n != nr || \
( (uintptr_t)_c % alignment ) || \
( ( ( row_major ? _rs_c : _cs_c )*sizeof( PASTEMAC(ch,ctype) ) ) % alignment ); \
GEMMTRSM_UKR_SETUP_CT_POST(ch);
// -- Flush macros --
#define GEMMTRSM_UKR_FLUSH_CT(ch) \
\
/* If we actually used the temporary microtile, use it to overwrite the
output microtile. Used by trsm. */ \
if ( _use_ct ) \
{ \
PASTEMAC(ch,copys_mxn) \
( \
m, n, \
_ct, _rs_ct, _cs_ct, \
_c, _rs_c, _cs_c \
); \
} \
#endif

12
kernels/haswell/3/bli_gemmtrsm_l_haswell_asm_d6x8.c
View File

@@ -58,6 +58,8 @@
void bli_sgemmtrsm_l_haswell_asm_6x16
(
dim_t m,
dim_t n,
dim_t k0,
float* restrict alpha,
float* restrict a10,
@@ -81,6 +83,8 @@ void bli_sgemmtrsm_l_haswell_asm_6x16
float* beta = bli_sm1;
GEMMTRSM_UKR_SETUP_CT_ANY( s, 6, 16, true );
begin_asm()
vzeroall() // zero all xmm/ymm registers.
@@ -825,6 +829,8 @@ void bli_sgemmtrsm_l_haswell_asm_6x16
"xmm12", "xmm13", "xmm14", "xmm15",
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( s );
}
@@ -843,6 +849,8 @@ void bli_sgemmtrsm_l_haswell_asm_6x16
void bli_dgemmtrsm_l_haswell_asm_6x8
(
dim_t m,
dim_t n,
dim_t k0,
double* restrict alpha,
double* restrict a10,
@@ -866,6 +874,8 @@ void bli_dgemmtrsm_l_haswell_asm_6x8
double* beta = bli_dm1;
GEMMTRSM_UKR_SETUP_CT_ANY( d, 6, 8, true );
begin_asm()
vzeroall() // zero all xmm/ymm registers.
@@ -1572,6 +1582,8 @@ void bli_dgemmtrsm_l_haswell_asm_6x8
"xmm12", "xmm13", "xmm14", "xmm15",
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( d );
}

12
kernels/haswell/3/bli_gemmtrsm_u_haswell_asm_d6x8.c
View File

@@ -58,6 +58,8 @@
void bli_sgemmtrsm_u_haswell_asm_6x16
(
dim_t m,
dim_t n,
dim_t k0,
float* restrict alpha,
float* restrict a10,
@@ -81,6 +83,8 @@ void bli_sgemmtrsm_u_haswell_asm_6x16
float* beta = bli_sm1;
GEMMTRSM_UKR_SETUP_CT_ANY( s, 6, 16, true );
begin_asm()
vzeroall() // zero all xmm/ymm registers.
@@ -830,6 +834,8 @@ void bli_sgemmtrsm_u_haswell_asm_6x16
"xmm12", "xmm13", "xmm14", "xmm15",
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( s );
}
@@ -848,6 +854,8 @@ void bli_sgemmtrsm_u_haswell_asm_6x16
void bli_dgemmtrsm_u_haswell_asm_6x8
(
dim_t m,
dim_t n,
dim_t k0,
double* restrict alpha,
double* restrict a10,
@@ -871,6 +879,8 @@ void bli_dgemmtrsm_u_haswell_asm_6x8
double* beta = bli_dm1;
GEMMTRSM_UKR_SETUP_CT_ANY( d, 6, 8, true );
begin_asm()
vzeroall() // zero all xmm/ymm registers.
@@ -1583,6 +1593,8 @@ void bli_dgemmtrsm_u_haswell_asm_6x8
"xmm12", "xmm13", "xmm14", "xmm15",
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( d );
}

5
kernels/penryn/3/bli_gemmtrsm_l_penryn_asm_d4x4.c
View File

@@ -56,6 +56,8 @@ void bli_sgemmtrsm_l_penryn_asm_8x4
void bli_dgemmtrsm_l_penryn_asm_4x4
(
dim_t m,
dim_t n,
dim_t k0,
double* restrict alpha,
double* restrict a10,
@@ -76,6 +78,8 @@ void bli_dgemmtrsm_l_penryn_asm_4x4
uint64_t rs_c = rs_c0;
uint64_t cs_c = cs_c0;
GEMMTRSM_UKR_SETUP_CT( d, 4, 4, false );
begin_asm()
mov(var(a10), rax) // load address of a10.
@@ -561,6 +565,7 @@ void bli_dgemmtrsm_l_penryn_asm_4x4
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( d );
}

5
kernels/penryn/3/bli_gemmtrsm_u_penryn_asm_d4x4.c
View File

@@ -56,6 +56,8 @@ void bli_sgemmtrsm_u_penryn_asm_8x4
void bli_dgemmtrsm_u_penryn_asm_4x4
(
dim_t m,
dim_t n,
dim_t k0,
double* restrict alpha,
double* restrict a12,
@@ -76,6 +78,8 @@ void bli_dgemmtrsm_u_penryn_asm_4x4
uint64_t rs_c = rs_c0;
uint64_t cs_c = cs_c0;
GEMMTRSM_UKR_SETUP_CT( d, 4, 4, false );
begin_asm()
mov(var(a12), rax) // load address of a12.
@@ -546,6 +550,7 @@ void bli_dgemmtrsm_u_penryn_asm_4x4
"memory"
)
GEMMTRSM_UKR_FLUSH_CT( d );
}

45
ref_kernels/3/bli_gemmtrsm_ref.c
View File

@@ -39,6 +39,8 @@
\
void PASTEMAC3(ch,opname,arch,suf) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \
@@ -52,8 +54,9 @@ void PASTEMAC3(ch,opname,arch,suf) \
{ \
const num_t dt = PASTEMAC(ch,type); \
\
const inc_t mr = bli_cntx_get_blksz_def_dt( dt, BLIS_MR, cntx ); \
const inc_t nr = bli_cntx_get_blksz_def_dt( dt, BLIS_NR, cntx ); \
const dim_t mr = bli_cntx_get_blksz_def_dt( dt, BLIS_MR, cntx ); \
const dim_t nr = bli_cntx_get_blksz_def_dt( dt, BLIS_NR, cntx ); \
\
const inc_t packnr = bli_cntx_get_blksz_max_dt( dt, BLIS_NR, cntx ); \
\
const inc_t rs_b = packnr; \
@@ -65,13 +68,35 @@ void PASTEMAC3(ch,opname,arch,suf) \
gemm_ukr = bli_cntx_get_l3_nat_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
PASTECH(ch,trsm_ukr_ft) \
trsm_ukr = bli_cntx_get_l3_nat_ukr_dt( dt, trsmkerid, cntx ); \
\
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
/* FGVZ: Should we be querying the preference of BLIS_GEMMTRSM_?_UKR
instead? */ \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : nr ); \
const inc_t cs_ct = ( col_pref ? mr : 1 ); \
\
const bool use_ct = ( m < mr || n < nr ); \
\
ctype* restrict c11_use = c11; \
inc_t rs_c_use = rs_c; \
inc_t cs_c_use = cs_c; \
\
if ( use_ct ) \
{ \
c11_use = ct; \
rs_c_use = rs_ct; \
cs_c_use = cs_ct; \
} \
\
/* lower: b11 = alpha * b11 - a10 * b01; */ \
/* upper: b11 = alpha * b11 - a12 * b21; */ \
gemm_ukr \
( \
mr, \
nr, \
m, \
n, \
k, \
minus_one, \
a1x, \
@@ -88,10 +113,20 @@ void PASTEMAC3(ch,opname,arch,suf) \
( \
a11, \
b11, \
c11, rs_c, cs_c, \
c11_use, rs_c_use, cs_c_use, \
data, \
cntx \
); \
\
if ( use_ct ) \
{ \
PASTEMAC(ch,copys_mxn) \
( \
m, n, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c \
); \
} \
\
/*
PASTEMAC(d,fprintm)( stdout, "gemmtrsm_ukr: b0111p_r after", k+3, 8, \

50
ref_kernels/ind/bli_gemmtrsm1m_ref.c
View File

@@ -39,6 +39,8 @@
\
void PASTEMAC3(ch,opname,arch,suf) \
( \
dim_t m, \
dim_t n, \
dim_t k, \
ctype* restrict alpha, \
ctype* restrict a1x, \
@@ -59,7 +61,7 @@ void PASTEMAC3(ch,opname,arch,suf) \
PASTECH(ch,trsm_ukr_ft) \
ctrsm_vir_ukr = bli_cntx_get_l3_vir_ukr_dt( dt, trsmkerid, cntx ); \
\
const bool col_pref = bli_cntx_l3_nat_ukr_prefers_cols_dt( dt_r, BLIS_GEMM_UKR, cntx ); \
const bool col_pref_r = bli_cntx_l3_nat_ukr_prefers_cols_dt( dt_r, BLIS_GEMM_UKR, cntx ); \
\
const dim_t mr = bli_cntx_get_blksz_def_dt( dt, BLIS_MR, cntx ); \
const dim_t nr = bli_cntx_get_blksz_def_dt( dt, BLIS_NR, cntx ); \
@@ -98,6 +100,28 @@ void PASTEMAC3(ch,opname,arch,suf) \
ctype_r* b_use; \
inc_t rs_b_use; \
inc_t cs_b_use; \
\
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
/* FGVZ: Should we be querying the preference of BLIS_GEMMTRSM_?_UKR
instead? */ \
const bool col_pref = bli_cntx_l3_vir_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : nr ); \
const inc_t cs_ct = ( col_pref ? mr : 1 ); \
\
const bool use_ct = ( m < mr || n < nr ); \
\
ctype* restrict c11_use = c11; \
inc_t rs_c_use = rs_c; \
inc_t cs_c_use = cs_c; \
\
if ( use_ct ) \
{ \
c11_use = ct; \
rs_c_use = rs_ct; \
cs_c_use = cs_ct; \
} \
\
\
/* Handle alphas with non-zero imaginary components. */ \
@@ -113,7 +137,7 @@ void PASTEMAC3(ch,opname,arch,suf) \
{ \
bli_abort(); \
\
/*
/*
ctype_r* restrict one_r = PASTEMAC(chr,1); \
\
const inc_t ld_b = rs_b; \
@@ -125,17 +149,17 @@ void PASTEMAC3(ch,opname,arch,suf) \
b11, rs_b, cs_b, ld_b ); \
\
alpha_r = *one_r; \
*/ \
*/ \
} \
\
\
{ \
/* Set the strides for the temporary bt matrix based on the native
real domain micro-kernel storage preferences. */ \
if ( col_pref ) { rs_bt = 1; cs_bt = mr; \
rs_bt_r = 1; cs_bt_r = mr_r; } \
else { rs_bt = nr; cs_bt = 1; \
rs_bt_r = nr_r; cs_bt_r = 1; } \
if ( col_pref_r ) { rs_bt = 1; cs_bt = mr; \
rs_bt_r = 1; cs_bt_r = mr_r; } \
else { rs_bt = nr; cs_bt = 1; \
rs_bt_r = nr_r; cs_bt_r = 1; } \
\
b_use = ( ctype_r* )bt; \
rs_b_use = rs_bt_r; \
@@ -241,10 +265,20 @@ void PASTEMAC3(ch,opname,arch,suf) \
( \
a11, \
b11, \
c11, rs_c, cs_c, \
c11_use, rs_c_use, cs_c_use, \
data, \
cntx \
); \
\
if ( use_ct ) \
{ \
PASTEMAC(ch,copys_mxn) \
( \
m, n, \
ct, rs_ct, cs_ct, \
c11, rs_c, cs_c \
); \
} \
}
INSERT_GENTFUNCCO_BASIC3( gemmtrsm1m_l, BLIS_CNAME_INFIX, BLIS_REF_SUFFIX, BLIS_TRSM_L_UKR )