Files
blis/frame/include/bli_obj_macro_defs.h
Field G. Van Zee 701b9aa3ff Redesigned control tree infrastructure.
Details:
- Altered control tree node struct definitions so that all nodes have the
  same struct definition, whose primary fields consist of a blocksize id,
  a variant function pointer, a pointer to an optional parameter struct,
  and a pointer to a (single) sub-node. This unified control tree type is
  now named cntl_t.
- Changed the way control tree nodes are connected, and what computation
  they represent, such that, for example, packing operations are now
  associated with nodes that are "inline" in the tree, rather than off-
  shoot braches. The original tree for the classic Goto gemm algorithm was
  expressed (roughly) as:

    blk_var2 -> blk_var3 -> blk_var1 -> ker_var2
                         |           |
                         -> packb    -> packa

  and now, the same tree would look like:

    blk_var2 -> blk_var3 -> packb -> blk_var1 -> packa -> ker_var2

  Specifically, the packb and packa nodes perform their respective packing
  operations and then recurse (without any loop) to a subproblem. This means
  there are now two kinds of level-3 control tree nodes: partitioning and
  non-partitioning. The blocked variants are members of the former, because
  they iteratively partition off submatrices and perform suboperations on
  those partitions, while the packing variants belong to the latter group.
  (This change has the effect of allowing greatly simplified initialization
  of the nodes, which previously involved setting many unused node fields to
  NULL.)
- Changed the way thrinfo_t tree nodes are arranged to mirror the new
  connective structure of control trees. That is, packm nodes are no longer
  off-shoot branches of the main algorithmic nodes, but rather connected
  "inline".
- Simplified control tree creation functions. Partitioning nodes are created
  concisely with just a few fields needing initialization. By contrast, the
  packing nodes require additional parameters, which are stored in a
  packm-specific struct that is tracked via the optional parameters pointer
  within the control tree struct. (This parameter struct must always begin
  with a uint64_t that contains the byte size of the struct. This allows
  us to use a generic function to recursively copy control trees.) gemm,
  herk, and trmm control tree creation continues to be consolidated into
  a single function, with the operation family being used to select
  among the parameter-agnostic macro-kernel wrappers. A single routine,
  bli_cntl_free(), is provided to free control trees recursively, whereby
  the chief thread within a groups release the blocks associated with
  mem_t entries back to the memory broker from which they were acquired.
- Updated internal back-ends, e.g. bli_gemm_int(), to query and call the
  function pointer stored in the current control tree node (rather than
  index into a local function pointer array). Before being invoked, these
  function pointers are first cast to a gemm_voft (for gemm, herk, or trmm
  families) or trsm_voft (for trsm family) type, which is defined in
  frame/3/bli_l3_var_oft.h.
- Retired herk and trmm internal back-ends, since all execution now flows
  through gemm or trsm blocked variants.
- Merged forwards- and backwards-moving variants by querying the direction
  from routines as a function of the variant's matrix operands. gemm and
  herk always move forward, while trmm and trsm move in a direction that
  is dependent on which operand (a or b) is triangular.
- Added functions bli_thread_get_range_mdim(), bli_thread_get_range_ndim(),
  each of which takes additional arguments and hides complexity in managing
  the difference between the way ranges are computed for the four families
  of operations.
- Simplified level-3 blocked variants according to the above changes, so that
  the only steps taken are:
  1. Query partitioning direction (forwards or backwards).
  2. Prune unreferenced regions, if they exist.
  3. Determine the thread partitioning sub-ranges.
  <begin loop>
    4. Determine the partitioning blocksize (passing in the partitioning
       direction)
    5. Acquire the curren iteration's partitions for the matrices affected
       by the current variants's partitioning dimension (m, k, n).
    6. Call the subproblem.
  <end loop>
- Instantiate control trees once per thread, per operation invocation.
  (This is a change from the previous regime in which control trees were
  treated as stateless objects, initialized with the library, and shared
  as read-only objects between threads.) This once-per-thread allocation
  is done primarily to allow threads to use the control tree as as place
  to cache certain data for use in subsequent loop iterations. Presently,
  the only application of this caching is a mem_t entry for the packing
  blocks checked out from the memory broker (allocator). If a non-NULL
  control tree is passed in by the (expert) user, then the tree is copied
  by each thread. This is done in bli_l3_thread_decorator(), in
  bli_thrcomm_*.c.
- Added a new field to the context, and opid_t which tracks the "family"
  of the operation being executed. For example, gemm, hemm, and symm are
  all part of the gemm family, while herk, syrk, her2k, and syr2k are
  all part of the herk family. Knowing the operation's family is necessary
  when conditionally executing the internal (beta) scalar reset on on
  C in blocked variant 3, which is needed for gemm and herk families,
  but must not be performed for the trmm family (because beta has only
  been applied to the current row-panel of C after the first rank-kc
  iteration).
- Reexpressed 3m3 induced method blocked variant in frame/3/gemm/ind
  to comform with the new control tree design, and renamed the macro-
  kernel codes corresponding to 3m2 and 4m1b.
- Renamed bli_mem.c (and its APIs) to bli_memsys.c, and renamed/relocated
  bli_mem_macro_defs.h from frame/include to frame/base/bli_mem.h.
- Renamed/relocated bli_auxinfo_macro_defs.h from frame/include to
  frame/base/bli_auxinfo.h.
- Fixed a minor bug whereby the storage-to-ukr-preference matching
  optimization in the various level-3 front-ends was not being applied
  properly when the context indicated that execution would be via an
  induced method. (Before, we always checked the native micro-kernel
  corresponding to the datatype being executed, whereas now we check
  the native micro-kernel corresponding to the datatype's real projection,
  since that is the micro-kernel that is actually used by induced methods.
- Added an option to the testsuite to skip the testing of native level-3
  complex implementations. Previously, it was always tested, provided that
  the c/z datatypes were enabled. However, some configurations use
  reference micro-kernels for complex datatypes, and testing these
  implementations can slow down the testsuite considerably.
2016-08-26 19:04:45 -05:00

1037 lines
24 KiB
C

/*
BLIS
An object-based framework for developing high-performance BLAS-like
libraries.
Copyright (C) 2014, The University of Texas at Austin
Copyright (C) 2016 Hewlett Packard Enterprise Development LP
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
- Neither the name of The University of Texas at Austin nor the names
of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef BLIS_OBJ_MACRO_DEFS_H
#define BLIS_OBJ_MACRO_DEFS_H
// -- Object query/modification macros --
// Info query
#define bli_obj_is_float( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_FLOAT_TYPE )
#define bli_obj_is_double( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_DOUBLE_TYPE )
#define bli_obj_is_scomplex( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_SCOMPLEX_TYPE )
#define bli_obj_is_dcomplex( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_DCOMPLEX_TYPE )
#define bli_obj_is_int( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_INT_TYPE )
#define bli_obj_is_const( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) == BLIS_BITVAL_CONST_TYPE )
#define bli_obj_domain( obj ) \
\
( (obj).info & BLIS_DOMAIN_BIT )
#define bli_obj_is_real( obj ) \
\
( ( (obj).info & BLIS_DOMAIN_BIT ) == BLIS_BITVAL_REAL )
#define bli_obj_is_complex( obj ) \
\
( ( (obj).info & BLIS_DOMAIN_BIT ) == BLIS_BITVAL_COMPLEX )
#define bli_obj_precision( obj ) \
\
( (obj).info & BLIS_PRECISION_BIT )
#define bli_obj_is_double_precision( obj ) \
\
( ( (obj).info & BLIS_PRECISION_BIT ) == BLIS_BITVAL_DOUBLE_PREC )
#define bli_obj_datatype( obj ) \
\
( (obj).info & BLIS_DATATYPE_BITS )
#define bli_obj_datatype_proj_to_real( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) & ~BLIS_BITVAL_COMPLEX )
#define bli_obj_datatype_proj_to_complex( obj ) \
\
( ( (obj).info & BLIS_DATATYPE_BITS ) & BLIS_BITVAL_COMPLEX )
#define bli_obj_target_datatype( obj ) \
\
( ( (obj).info & BLIS_TARGET_DT_BITS ) >> BLIS_TARGET_DT_SHIFT )
#define bli_obj_execution_datatype( obj ) \
\
( ( (obj).info & BLIS_EXECUTION_DT_BITS ) >> BLIS_EXECUTION_DT_SHIFT )
#define bli_obj_conjtrans_status( obj ) \
\
( (obj).info & BLIS_CONJTRANS_BITS )
#define bli_obj_onlytrans_status( obj ) \
\
( (obj).info & BLIS_TRANS_BIT )
#define bli_obj_has_trans( obj ) \
\
( ( (obj).info & BLIS_TRANS_BIT ) == BLIS_BITVAL_TRANS ) \
#define bli_obj_has_notrans( obj ) \
\
( ( (obj).info & BLIS_TRANS_BIT ) == BLIS_BITVAL_NO_TRANS ) \
#define bli_obj_conj_status( obj ) \
\
( (obj).info & BLIS_CONJ_BIT )
#define bli_obj_has_conj( obj ) \
\
( ( (obj).info & BLIS_CONJ_BIT ) == BLIS_BITVAL_CONJ ) \
#define bli_obj_has_noconj( obj ) \
\
( ( (obj).info & BLIS_CONJ_BIT ) == BLIS_BITVAL_NO_CONJ ) \
#define bli_obj_uplo( obj ) \
\
( (obj).info & BLIS_UPLO_BITS ) \
#define bli_obj_is_upper( obj ) \
\
( ( (obj).info & BLIS_UPLO_BITS ) == BLIS_BITVAL_UPPER )
#define bli_obj_is_lower( obj ) \
\
( ( (obj).info & BLIS_UPLO_BITS ) == BLIS_BITVAL_LOWER )
#define bli_obj_is_upper_after_trans( obj ) \
\
( bli_obj_has_trans( (obj) ) ? bli_obj_is_lower( (obj) ) \
: bli_obj_is_upper( (obj) ) )
#define bli_obj_is_lower_after_trans( obj ) \
\
( bli_obj_has_trans( (obj) ) ? bli_obj_is_upper( (obj) ) \
: bli_obj_is_lower( (obj) ) )
#define bli_obj_is_upper_or_lower( obj ) \
\
( bli_obj_is_upper( obj ) || \
bli_obj_is_lower( obj ) )
#define bli_obj_is_dense( obj ) \
\
( ( (obj).info & BLIS_UPLO_BITS ) == BLIS_BITVAL_DENSE )
#define bli_obj_is_zeros( obj ) \
\
( ( (obj).info & BLIS_UPLO_BITS ) == BLIS_BITVAL_ZEROS )
#define bli_obj_diag( obj ) \
\
( (obj).info & BLIS_UNIT_DIAG_BIT )
#define bli_obj_has_nonunit_diag( obj ) \
\
( ( (obj).info & BLIS_UNIT_DIAG_BIT ) == BLIS_BITVAL_NONUNIT_DIAG )
#define bli_obj_has_unit_diag( obj ) \
\
( ( (obj).info & BLIS_UNIT_DIAG_BIT ) == BLIS_BITVAL_UNIT_DIAG )
#define bli_obj_has_inverted_diag( obj ) \
\
( ( (obj).info & BLIS_INVERT_DIAG_BIT ) == BLIS_BITVAL_INVERT_DIAG )
#define bli_obj_is_pack_rev_if_upper( obj ) \
\
( ( (obj).info & BLIS_PACK_REV_IF_UPPER_BIT ) == BLIS_BITVAL_PACK_REV_IF_UPPER )
#define bli_obj_is_pack_rev_if_lower( obj ) \
\
( ( (obj).info & BLIS_PACK_REV_IF_LOWER_BIT ) == BLIS_BITVAL_PACK_REV_IF_LOWER )
#define bli_obj_pack_schema( obj ) \
\
( (obj).info & BLIS_PACK_SCHEMA_BITS )
#define bli_obj_is_packed( obj ) \
\
( ( (obj).info & BLIS_PACK_BIT ) )
#define bli_obj_is_row_packed( obj ) \
\
( ( (obj).info & BLIS_PACK_RC_BIT ) == ( BLIS_BITVAL_PACKED_UNSPEC ^ \
BLIS_BITVAL_PACKED_ROWS ) )
#define bli_obj_is_col_packed( obj ) \
\
( ( (obj).info & BLIS_PACK_RC_BIT ) == ( BLIS_BITVAL_PACKED_UNSPEC ^ \
BLIS_BITVAL_PACKED_COLUMNS ) )
#define bli_obj_is_panel_packed( obj ) \
\
( ( (obj).info & BLIS_PACK_PANEL_BIT ) )
#define bli_obj_pack_buffer_type( obj ) \
\
( (obj).info & BLIS_PACK_BUFFER_BITS )
#define bli_obj_struc( obj ) \
\
( (obj).info & BLIS_STRUC_BITS )
#define bli_obj_is_general( obj ) \
\
( ( (obj).info & BLIS_STRUC_BITS ) == BLIS_BITVAL_GENERAL )
#define bli_obj_is_hermitian( obj ) \
\
( ( (obj).info & BLIS_STRUC_BITS ) == BLIS_BITVAL_HERMITIAN )
#define bli_obj_is_symmetric( obj ) \
\
( ( (obj).info & BLIS_STRUC_BITS ) == BLIS_BITVAL_SYMMETRIC )
#define bli_obj_is_triangular( obj ) \
\
( ( (obj).info & BLIS_STRUC_BITS ) == BLIS_BITVAL_TRIANGULAR )
#define bli_obj_is_herm_or_symm( obj ) \
\
( bli_obj_is_hermitian( obj ) || \
bli_obj_is_symmetric( obj ) )
// Info modification
#define bli_obj_set_conjtrans( conjtrans, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_CONJTRANS_BITS ) | (conjtrans); \
}
#define bli_obj_set_onlytrans( trans, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_TRANS_BIT ) | (trans); \
}
#define bli_obj_set_conj( conjval, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_CONJ_BIT ) | (conjval); \
}
#define bli_obj_set_uplo( uplo, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_UPLO_BITS ) | (uplo); \
}
#define bli_obj_set_diag( diag, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_UNIT_DIAG_BIT ) | (diag); \
}
#define bli_obj_set_invert_diag( inv_diag, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_INVERT_DIAG_BIT ) | (inv_diag); \
}
#define bli_obj_set_datatype( dt, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_DATATYPE_BITS ) | (dt); \
}
#define bli_obj_set_target_datatype( dt, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_TARGET_DT_BITS ) | ( dt << BLIS_TARGET_DT_SHIFT ); \
}
#define bli_obj_set_execution_datatype( dt, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_EXECUTION_DT_BITS ) | ( dt << BLIS_EXECUTION_DT_SHIFT ); \
}
#define bli_obj_set_pack_schema( pack, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_PACK_SCHEMA_BITS ) | (pack); \
}
#define bli_obj_set_pack_order_if_upper( packordifup, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_PACK_REV_IF_UPPER_BIT ) | (packordifup); \
}
#define bli_obj_set_pack_order_if_lower( packordiflo, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_PACK_REV_IF_LOWER_BIT ) | (packordiflo); \
}
#define bli_obj_set_pack_buffer_type( packbuf, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_PACK_BUFFER_BITS ) | (packbuf); \
}
#define bli_obj_set_struc( struc, obj ) \
{ \
(obj).info = ( (obj).info & ~BLIS_STRUC_BITS ) | (struc); \
}
#define bli_obj_toggle_trans( obj )\
{ \
(obj).info = ( (obj).info ^ BLIS_TRANS_BIT ); \
}
#define bli_obj_toggle_conj( obj )\
{ \
(obj).info = ( (obj).info ^ BLIS_CONJ_BIT ); \
}
#define bli_obj_toggle_uplo( obj ) \
{ \
(obj).info = ( (obj).info ^ BLIS_LOWER_BIT ) ^ BLIS_UPPER_BIT; \
}
#define bli_obj_toggle_region_ref( obj ) \
{ \
if ( bli_obj_is_upper( obj ) ) bli_obj_inc_diag_off( -1, obj ); \
else if ( bli_obj_is_lower( obj ) ) bli_obj_inc_diag_off( 1, obj ); \
\
bli_obj_toggle_uplo( obj ); \
}
#define bli_obj_toggle_uplo_if_trans( trans, obj ) \
{ \
if ( bli_does_trans( trans ) && ( bli_obj_is_upper_or_lower( obj ) ) ) \
{ \
bli_obj_toggle_uplo( obj ); \
bli_obj_negate_diag_offset( obj ); \
} \
}
#define bli_obj_apply_trans( trans, obj ) \
{ \
(obj).info = ( (obj).info ^ (trans) ); \
}
#define bli_obj_apply_conj( conjval, obj ) \
{ \
(obj).info = ( (obj).info ^ (conjval) ); \
}
// Root matrix query
#define bli_obj_root( obj ) \
\
((obj).root)
#define bli_obj_root_uplo( obj ) \
\
bli_obj_uplo( *bli_obj_root( obj ) )
#define bli_obj_root_is_general( obj ) \
\
bli_obj_is_general( *bli_obj_root( obj ) )
#define bli_obj_root_is_hermitian( obj ) \
\
bli_obj_is_hermitian( *bli_obj_root( obj ) )
#define bli_obj_root_is_symmetric( obj ) \
\
bli_obj_is_symmetric( *bli_obj_root( obj ) )
#define bli_obj_root_is_triangular( obj ) \
\
bli_obj_is_triangular( *bli_obj_root( obj ) )
#define bli_obj_root_is_herm_or_symm( obj ) \
\
( bli_obj_is_hermitian( *bli_obj_root( obj ) ) || \
bli_obj_is_symmetric( *bli_obj_root( obj ) ) )
#define bli_obj_root_is_upper( obj ) \
\
bli_obj_is_upper( *bli_obj_root( obj ) )
#define bli_obj_root_is_lower( obj ) \
\
bli_obj_is_lower( *bli_obj_root( obj ) )
// Root matrix modification
#define bli_obj_set_as_root( obj )\
{ \
(obj).root = &(obj); \
}
// Dimension query
#define bli_obj_length( obj ) \
\
( (obj).dim[BLIS_M] )
#define bli_obj_width( obj ) \
\
( (obj).dim[BLIS_N] )
#define bli_obj_dim( mdim, obj ) \
\
( (obj).dim[mdim] )
#define bli_obj_min_dim( obj ) \
\
( bli_min( bli_obj_length( obj ), \
bli_obj_width( obj ) ) )
#define bli_obj_max_dim( obj ) \
\
( bli_max( bli_obj_length( obj ), \
bli_obj_width( obj ) ) )
#define bli_obj_length_after_trans( obj ) \
\
( bli_obj_has_trans( (obj) ) ? bli_obj_width( (obj) ) \
: bli_obj_length( (obj) ) )
#define bli_obj_width_after_trans( obj ) \
\
( bli_obj_has_trans( (obj) ) ? bli_obj_length( (obj) ) \
: bli_obj_width( (obj) ) )
#define bli_obj_get_dims_after_trans( obj, dim_m, dim_n ) \
{ \
if ( bli_obj_has_notrans( trans ) ) \
{ \
dim_m = bli_obj_length( obj ); \
dim_n = bli_obj_width( obj ); \
} \
else \
{ \
dim_m = bli_obj_width( obj ); \
dim_n = bli_obj_length( obj ); \
} \
}
/*
bli_obj_length_stored( obj )
{
if ( lower )
{
if ( diagoff < 0 ) m_stored = m + diagoff;
else m_stored = m
}
else if ( upper )
{
if ( diagoff < 0 ) m_stored = min( m, n - diagoff );
else m_stored = min( m, n - diagoff );
}
}
bli_obj_width_stored( obj )
{
if ( lower )
{
if ( diagoff < 0 ) n_stored = min( n, m + diagoff );
else n_stored = min( n, m + diagoff );
}
else if ( upper )
{
if ( diagoff < 0 ) n_stored = n;
else n_stored = n - diagoff;
}
}
*/
// Note: The purpose of these macros is to obtain the length and width
// of the smallest submatrices of an object that could still encompass
// the stored data above (if obj is upper) or below (if obj is lower)
// the diagonal.
#define bli_obj_length_stored( obj ) \
\
( bli_obj_is_upper( obj ) \
? bli_min( bli_obj_length( obj ), \
bli_obj_width( obj ) - bli_obj_diag_offset( obj ) ) \
: bli_min( bli_obj_length( obj ), \
bli_obj_length( obj ) + bli_obj_diag_offset( obj ) ) \
)
#define bli_obj_width_stored( obj ) \
\
( bli_obj_is_lower( obj ) \
? bli_min( bli_obj_width( obj ), \
bli_obj_length( obj ) + bli_obj_diag_offset( obj ) ) \
: bli_min( bli_obj_width( obj ), \
bli_obj_width( obj ) - bli_obj_diag_offset( obj ) ) \
)
#define bli_obj_length_stored_after_trans( obj ) \
\
( bli_obj_has_trans( obj ) ? bli_obj_width_stored( obj ) \
: bli_obj_length_stored( obj ) )
#define bli_obj_width_stored_after_trans( obj ) \
\
( bli_obj_has_trans( obj ) ? bli_obj_length_stored( obj ) \
: bli_obj_width_stored( obj ) )
#define bli_obj_vector_dim( x ) \
\
( bli_obj_length( x ) == 1 ? bli_obj_width( x ) \
: bli_obj_length( x ) )
#define bli_obj_vector_inc( x ) \
\
( bli_obj_is_1x1( x ) ? 1 : \
( bli_obj_length( x ) == 1 ? bli_obj_col_stride( x ) \
: bli_obj_row_stride( x ) ) \
)
#define bli_obj_is_vector( x ) \
\
( bli_obj_length( x ) == 1 || \
bli_obj_width( x ) == 1 )
#define bli_obj_is_row_vector( x ) \
\
( bli_obj_length( x ) == 1 )
#define bli_obj_is_col_vector( x ) \
\
( bli_obj_width( x ) == 1 )
#define bli_obj_has_zero_dim( obj ) \
\
( bli_obj_length( obj ) == 0 || \
bli_obj_width( obj ) == 0 )
#define bli_obj_is_1x1( x ) \
\
( bli_obj_length( x ) == 1 && \
bli_obj_width( x ) == 1 )
// Dimension modification
#define bli_obj_set_length( dim_m, obj ) \
{ \
(obj).dim[BLIS_M] = dim_m; \
}
#define bli_obj_set_width( dim_n, obj ) \
{ \
(obj).dim[BLIS_N] = dim_n; \
}
#define bli_obj_set_dim( mdim, dim_val, obj ) \
{ \
(obj).dim[mdim] = dim_val; \
}
#define bli_obj_set_dims( dim_m, dim_n, obj ) \
{ \
bli_obj_set_length( dim_m, obj ); \
bli_obj_set_width( dim_n, obj ); \
}
#define bli_obj_set_dims_with_trans( trans, dim_m, dim_n, obj ) \
{ \
if ( bli_does_notrans( trans ) ) \
{ \
bli_obj_set_length( dim_m, obj ); \
bli_obj_set_width( dim_n, obj ); \
} \
else \
{ \
bli_obj_set_length( dim_n, obj ); \
bli_obj_set_width( dim_m, obj ); \
} \
}
// Stride/increment query
#define bli_obj_row_stride( obj ) \
\
( (obj).rs )
#define bli_obj_col_stride( obj ) \
\
( (obj).cs )
#define bli_obj_imag_stride( obj ) \
\
( (obj).is )
#define bli_obj_row_stride_mag( obj ) \
\
( bli_abs( bli_obj_row_stride( obj ) ) )
#define bli_obj_col_stride_mag( obj ) \
\
( bli_abs( bli_obj_col_stride( obj ) ) )
#define bli_obj_imag_stride_mag( obj ) \
\
( bli_abs( bli_obj_imag_stride( obj ) ) )
//
// NOTE: The following two macros differ from their non-obj counterparts
// in that they do not identify m x 1 and 1 x n objects as row-stored and
// column-stored, respectively, which is needed when considering packed
// objects. But this is okay, since none of the invocations of these
// "obj" macros are used on packed matrices.
//
#define bli_obj_is_row_stored( obj ) \
\
( bli_obj_col_stride_mag( obj ) == 1 )
#define bli_obj_is_col_stored( obj ) \
\
( bli_obj_row_stride_mag( obj ) == 1 )
#define bli_obj_is_gen_stored( obj ) \
\
( bli_obj_row_stride_mag( obj ) != 1 && \
bli_obj_col_stride_mag( obj ) != 1 )
#define bli_obj_is_row_tilted( obj ) \
\
( bli_obj_col_stride_mag( obj ) < bli_obj_row_stride_mag( obj ) )
#define bli_obj_is_col_tilted( obj ) \
\
( bli_obj_row_stride_mag( obj ) < bli_obj_col_stride_mag( obj ) )
// Stride/increment modification
#define bli_obj_set_strides( row_stride, col_stride, obj ) \
{ \
(obj).rs = row_stride; \
(obj).cs = col_stride; \
}
#define bli_obj_set_imag_stride( imag_stride, obj ) \
{ \
(obj).is = imag_stride; \
}
// Offset query
#define bli_obj_row_off( obj ) \
\
( (obj).off[BLIS_M] )
#define bli_obj_col_off( obj ) \
\
( (obj).off[BLIS_N] )
#define bli_obj_off( mdim, obj ) \
\
( (obj).off[mdim] )
// Offset modification
#define bli_obj_set_off( mdim, offset, obj ) \
{ \
(obj).off[mdim] = offset; \
}
#define bli_obj_set_offs( offset_m, offset_n, obj ) \
{ \
bli_obj_set_off( BLIS_M, offset_m, obj ); \
bli_obj_set_off( BLIS_N, offset_n, obj ); \
}
#define bli_obj_inc_off( mdim, offset, obj ) \
{ \
(obj).off[mdim] += offset; \
}
#define bli_obj_inc_offm( offset, obj ) \
{ \
bli_obj_inc_off( BLIS_M, offset, obj ); \
}
#define bli_obj_inc_offn( offset, obj ) \
{ \
bli_obj_inc_off( BLIS_N, offset, obj ); \
}
#define bli_obj_inc_offs( offset_m, offset_n, obj ) \
{ \
bli_obj_inc_off( BLIS_M, offset_m, obj ); \
bli_obj_inc_off( BLIS_N, offset_n, obj ); \
}
// Diagonal offset query
#define bli_obj_diag_offset( obj ) \
\
( (obj).diag_off )
#define bli_obj_diag_offset_after_trans( obj ) \
\
( bli_obj_has_trans( obj ) ? -bli_obj_diag_offset( obj ) \
: bli_obj_diag_offset( obj ) )
#define bli_obj_has_main_diag( obj ) \
\
( bli_obj_diag_offset( obj ) == 0 )
#define bli_obj_is_strictly_above_diag( obj ) \
\
( ( doff_t )bli_obj_length( obj ) <= -bli_obj_diag_offset( obj ) )
#define bli_obj_is_strictly_below_diag( obj ) \
\
( ( doff_t )bli_obj_width( obj ) <= bli_obj_diag_offset( obj ) )
#define bli_obj_is_outside_diag( obj ) \
\
( bli_obj_is_strictly_above_diag( obj ) || \
bli_obj_is_strictly_below_diag( obj ) )
#define bli_obj_intersects_diag( obj ) \
\
( !bli_obj_is_strictly_above_diag( obj ) && \
!bli_obj_is_strictly_below_diag( obj ) )
#define bli_obj_is_unstored_subpart( obj ) \
\
( ( bli_obj_root_is_lower( obj ) && bli_obj_is_strictly_above_diag( obj ) ) || \
( bli_obj_root_is_upper( obj ) && bli_obj_is_strictly_below_diag( obj ) ) )
// Diagonal offset modification
#define bli_obj_set_diag_offset( offset, obj ) \
{ \
(obj).diag_off = ( doff_t )(offset); \
}
#define bli_obj_negate_diag_offset( obj ) \
{ \
(obj).diag_off = -(obj).diag_off; \
}
#define bli_obj_inc_diag_off( offset, obj ) \
{ \
(obj).diag_off += ( doff_t )(offset); \
}
// Buffer address query
#define bli_obj_buffer( obj ) \
\
( (obj).buffer )
// Buffer address modification
#define bli_obj_set_buffer( buf, obj ) \
{ \
(obj).buffer = buf; \
}
// Bufferless scalar field query
#define bli_obj_internal_scalar_buffer( obj ) \
\
&( (obj).scalar )
// Bufferless scalar field modification
#define bli_obj_set_internal_scalar( val, obj ) \
{ \
(obj).scalar = val; \
}
#define bli_obj_copy_internal_scalar( a, b ) \
{ \
(b).scalar = (a).scalar; \
}
// Element size query
#define bli_obj_elem_size( obj ) \
\
( (obj).elem_size )
// Element size modification
#define bli_obj_set_elem_size( size, obj ) \
{ \
(obj).elem_size = size; \
}
// Packed matrix info query
#define bli_obj_padded_length( obj ) \
\
( (obj).m_padded )
#define bli_obj_padded_width( obj ) \
\
( (obj).n_padded )
// Packed matrix info modification
#define bli_obj_set_buffer_to_mem( mem_p, obj ) \
{ \
void* buf = bli_mem_buffer( mem_p ); \
bli_obj_set_buffer( buf, obj ); \
} \
#define bli_obj_set_padded_length( m0, obj ) \
{ \
(obj).m_padded = m0; \
}
#define bli_obj_set_padded_width( n0, obj ) \
{ \
(obj).n_padded = n0; \
}
#define bli_obj_set_padded_dims( m0, n0, obj ) \
{ \
bli_obj_set_padded_length( m0, obj ); \
bli_obj_set_padded_width( n0, obj ); \
}
// Packed panel info query
#define bli_obj_panel_length( obj ) \
\
( (obj).m_panel )
#define bli_obj_panel_width( obj ) \
\
( (obj).n_panel )
#define bli_obj_panel_dim( obj ) \
\
( (obj).pd )
#define bli_obj_panel_stride( obj ) \
\
( (obj).ps )
// Packed panel info modification
#define bli_obj_set_panel_length( m0, obj ) \
{ \
(obj).m_panel = m0; \
}
#define bli_obj_set_panel_width( n0, obj ) \
{ \
(obj).n_panel = n0; \
}
#define bli_obj_set_panel_dim( panel_dim, obj ) \
{ \
(obj).pd = panel_dim; \
}
#define bli_obj_set_panel_stride( panel_stride, obj ) \
{ \
(obj).ps = panel_stride; \
}
// -- Miscellaneous object macros --
// Make a full alias (shallow copy)
#define bli_obj_alias_to( a, b ) \
{ \
bli_obj_init_full_shallow_copy_of( a, b ); \
}
// Check if two objects are aliases of one another
#define bli_obj_is_alias_of( a, b ) \
\
( (b).buffer == (a).buffer )
// Create an alias with a trans value applied.
// (Note: trans may include a conj component.)
#define bli_obj_alias_with_trans( trans, a, b ) \
{ \
bli_obj_alias_to( a, b ); \
bli_obj_apply_trans( trans, b ); \
}
// Create an alias with a conj value applied.
#define bli_obj_alias_with_conj( conja, a, b ) \
{ \
bli_obj_alias_to( a, b ); \
bli_obj_apply_conj( conja, b ); \
}
// Initialize object with default properties (info field)
#define bli_obj_set_defaults( obj ) \
{ \
(obj).info = 0x0; \
(obj).info = (obj).info | BLIS_BITVAL_DENSE | BLIS_BITVAL_GENERAL; \
}
// Submatrix/scalar buffer acquisition
#define BLIS_CONSTANT_SLOT_SIZE BLIS_MAX_TYPE_SIZE
#define BLIS_CONSTANT_SIZE ( 5 * BLIS_CONSTANT_SLOT_SIZE )
#define bli_obj_buffer_for_const( dt, obj ) \
\
( void* )( \
( ( char* )( bli_obj_buffer( obj ) ) ) + \
( dim_t )( dt * BLIS_CONSTANT_SLOT_SIZE ) \
)
#define bli_obj_buffer_at_off( obj ) \
\
( void* )( \
( ( char* )( bli_obj_buffer ( obj ) ) + \
( dim_t )( bli_obj_elem_size( obj ) ) * \
( bli_obj_col_off( obj ) * bli_obj_col_stride( obj ) + \
bli_obj_row_off( obj ) * bli_obj_row_stride( obj ) \
) \
) \
)
#define bli_obj_buffer_for_1x1( dt, obj ) \
\
( void* )( bli_obj_is_const( obj ) ? ( bli_obj_buffer_for_const( dt, obj ) ) \
: ( bli_obj_buffer_at_off( obj ) ) \
)
// Swap objects
#define bli_obj_swap( a, b ) \
{ \
obj_t t_; \
t_ = b; b = a; a = t_; \
}
// Swap object pointers
#define bli_obj_swap_pointers( a, b ) \
{ \
obj_t* t_; \
t_ = b; b = a; a = t_; \
}
// If a transposition is needed, induce one: swap dimensions, increments
// and offsets, and then clear the trans bit.
#define bli_obj_induce_trans( obj ) \
{ \
{ \
dim_t m_ = bli_obj_length( obj ); \
dim_t n_ = bli_obj_width( obj ); \
inc_t rs_ = bli_obj_row_stride( obj ); \
inc_t cs_ = bli_obj_col_stride( obj ); \
dim_t offm_ = bli_obj_row_off( obj ); \
dim_t offn_ = bli_obj_col_off( obj ); \
doff_t diag_off_ = bli_obj_diag_offset( obj ); \
\
bli_obj_set_dims( n_, m_, obj ); \
bli_obj_set_strides( cs_, rs_, obj ); \
bli_obj_set_offs( offn_, offm_, obj ); \
bli_obj_set_diag_offset( -diag_off_, obj ); \
\
if ( bli_obj_is_upper_or_lower( obj ) ) \
bli_obj_toggle_uplo( obj ); \
\
/* Note that this macro DOES NOT touch the transposition bit! If
the calling code is using this macro to handle an object whose
transposition bit is set prior to computation, that code needs
to manually clear or toggle the bit, via
bli_obj_set_onlytrans() or bli_obj_toggle_trans(),
respectively. */ \
} \
}
// Sometimes we need to "reflect" a partition because the data we want is
// actually stored on the other side of the diagonal. The nuts and bolts of
// this macro look a lot like an induced transposition, except that the row
// and column strides are left unchanged (which, of course, drastically
// changes the effect of the macro).
#define bli_obj_reflect_about_diag( obj ) \
{ \
{ \
dim_t m_ = bli_obj_length( obj ); \
dim_t n_ = bli_obj_width( obj ); \
dim_t offm_ = bli_obj_row_off( obj ); \
dim_t offn_ = bli_obj_col_off( obj ); \
doff_t diag_off_ = bli_obj_diag_offset( obj ); \
\
bli_obj_set_dims( n_, m_, obj ); \
bli_obj_set_offs( offn_, offm_, obj ); \
bli_obj_set_diag_offset( -diag_off_, obj ); \
\
bli_obj_toggle_trans( obj ); \
} \
}
#endif