mirror of
https://github.com/amd/blis.git
synced 2026-05-11 09:39:59 +00:00
ecbebe7c2e
Details: - Defined a new struct datatype, rntm_t (runtime), to house the thrloop field of the cntx_t (context). The thrloop array holds the number of ways of parallelism (thread "splits") to extract per level-3 algorithmic loop until those values can be used to create a corresponding node in the thread control tree (thrinfo_t structure), which (for any given level-3 invocation) usually happens by the time the macrokernel is called for the first time. - Relocating the thrloop from the cntx_t remedies a thread-safety issue when invoking level-3 operations from two or more application threads. The race condition existed because the cntx_t, a pointer to which is usually queried from the global kernel structure (gks), is supposed to be a read-only. However, the previous code would write to the cntx_t's thrloop field *after* it had been queried, thus violating its read-only status. In practice, this would not cause a problem when a sequential application made a multithreaded call to BLIS, nor when two or more application threads used the same parallelization scheme when calling BLIS, because in either case all application theads would be using the same ways of parallelism for each loop. The true effects of the race condition were limited to situations where two or more application theads used *different* parallelization schemes for any given level-3 call. - In remedying the above race condition, the application or calling library can now specify the parallelization scheme on a per-call basis. All that is required is that the thread encode its request for parallelism into the rntm_t struct prior to passing the address of the rntm_t to one of the expert interfaces of either the typed or object APIs. This allows, for example, one application thread to extract 4-way parallelism from a call to gemm while another application thread requests 2-way parallelism. Or, two threads could each request 4-way parallelism, but from different loops. - A rntm_t* parameter has been added to the function signatures of most of the level-3 implementation stack (with the most notable exception being packm) as well as all level-1v, -1d, -1f, -1m, and -2 expert APIs. (A few internal functions gained the rntm_t* parameter even though they currently have no use for it, such as bli_l3_packm().) This required some internal calls to some of those functions to be updated since BLIS was already using those operations internally via the expert interfaces. For situations where a rntm_t object is not available, such as within packm/unpackm implementations, NULL is passed in to the relevant expert interfaces. This is acceptable for now since parallelism is not obtained for non-level-3 operations. - Revamped how global parallelism is encoded. First, the conventional environment variables such as BLIS_NUM_THREADS and BLIS_*_NT are only read once, at library initialization. (Thanks to Nathaniel Smith for suggesting this to avoid repeated calls getenv(), which can be slow.) Those values are recorded to a global rntm_t object. Public APIs, in bli_thread.c, are still available to get/set these values from the global rntm_t, though now the "set" functions have additional logic to ensure that the values are set in a synchronous manner via a mutex. If/when NULL is passed into an expert API (meaning the user opted to not provide a custom rntm_t), the values from the global rntm_t are copied to a local rntm_t, which is then passed down the function stack. Calling a basic API is equivalent to calling the expert APIs with NULL for the cntx and rntm parameters, which means the semantic behavior of these basic APIs (vis-a-vis multithreading) is unchanged from before. - Renamed bli_cntx_set_thrloop_from_env() to bli_rntm_set_ways_for_op() and reimplemented, with the function now being able to treat the incoming rntm_t in a manner agnostic to its origin--whether it came from the application or is an internal copy of the global rntm_t. - Removed various global runtime APIs for setting the number of ways of parallelism for individual loops (e.g. bli_thread_set_*_nt()) as well as the corresponding "get" functions. The new model simplifies these interfaces so that one must either set the total number of threads, OR set all of the ways of parallelism for each loop simultaneously (in a single function call). - Updated sandbox/ref99 according to above changes. - Rewrote/augmented docs/Multithreading.md to document the three methods (and two specific ways within each method) of requesting parallelism in BLIS. - Removed old, disabled code from bli_l3_thrinfo.c. - Whitespace changes to code (e.g. bli_obj.c) and docs/BuildSystem.md.
507 lines
14 KiB
C
507 lines
14 KiB
C
/*
|
|
|
|
BLIS
|
|
An object-based framework for developing high-performance BLAS-like
|
|
libraries.
|
|
|
|
Copyright (C) 2014, The University of Texas at Austin
|
|
|
|
Redistribution and use in source and binary forms, with or without
|
|
modification, are permitted provided that the following conditions are
|
|
met:
|
|
- Redistributions of source code must retain the above copyright
|
|
notice, this list of conditions and the following disclaimer.
|
|
- Redistributions in binary form must reproduce the above copyright
|
|
notice, this list of conditions and the following disclaimer in the
|
|
documentation and/or other materials provided with the distribution.
|
|
- Neither the name of The University of Texas at Austin nor the names
|
|
of its contributors may be used to endorse or promote products
|
|
derived from this software without specific prior written permission.
|
|
|
|
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*/
|
|
|
|
// Guard the function definitions so that they are only compiled when
|
|
// #included from files that define the typed API macros.
|
|
#ifdef BLIS_ENABLE_TAPI
|
|
|
|
//
|
|
// Define BLAS-like interfaces with typed operands.
|
|
//
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
trans_t transa, \
|
|
conj_t conjx, \
|
|
dim_t m, \
|
|
dim_t n, \
|
|
ctype* alpha, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* beta, \
|
|
ctype* y, inc_t incy \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
dim_t m_y, n_x; \
|
|
\
|
|
/* Determine the dimensions of y and x. */ \
|
|
bli_set_dims_with_trans( transa, m, n, &m_y, &n_x ); \
|
|
\
|
|
/* If y has zero elements, return early. */ \
|
|
if ( bli_zero_dim1( m_y ) ) return; \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* If x has zero elements, or if alpha is zero, scale y by beta and
|
|
return early. */ \
|
|
if ( bli_zero_dim1( n_x ) || PASTEMAC(ch,eq0)( *alpha ) ) \
|
|
{ \
|
|
PASTEMAC2(ch,scalv,BLIS_TAPI_EX_SUF) \
|
|
( \
|
|
BLIS_NO_CONJUGATE, \
|
|
m_y, \
|
|
beta, \
|
|
y, incy, \
|
|
cntx, \
|
|
NULL \
|
|
); \
|
|
return; \
|
|
} \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_does_notrans( transa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_does_trans( transa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
transa, \
|
|
conjx, \
|
|
m, \
|
|
n, \
|
|
alpha, \
|
|
a, rs_a, cs_a, \
|
|
x, incx, \
|
|
beta, \
|
|
y, incy, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC3( gemv, gemv, gemv_unf_var1, gemv_unf_var2 )
|
|
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
conj_t conjx, \
|
|
conj_t conjy, \
|
|
dim_t m, \
|
|
dim_t n, \
|
|
ctype* alpha, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* y, inc_t incy, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
/* If x or y has zero elements, or if alpha is zero, return early. */ \
|
|
if ( bli_zero_dim2( m, n ) || PASTEMAC(ch,eq0)( *alpha ) ) return; \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
conjx, \
|
|
conjy, \
|
|
m, \
|
|
n, \
|
|
alpha, \
|
|
x, incx, \
|
|
y, incy, \
|
|
a, rs_a, cs_a, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC3( ger, ger, ger_unb_var1, ger_unb_var2 )
|
|
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, conjh, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
uplo_t uploa, \
|
|
conj_t conja, \
|
|
conj_t conjx, \
|
|
dim_t m, \
|
|
ctype* alpha, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* beta, \
|
|
ctype* y, inc_t incy \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* If x has zero elements, or if alpha is zero, scale y by beta and
|
|
return early. */ \
|
|
if ( bli_zero_dim1( m ) || PASTEMAC(ch,eq0)( *alpha ) ) \
|
|
{ \
|
|
PASTEMAC2(ch,scalv,BLIS_TAPI_EX_SUF) \
|
|
( \
|
|
BLIS_NO_CONJUGATE, \
|
|
m, \
|
|
beta, \
|
|
y, incy, \
|
|
cntx, \
|
|
NULL \
|
|
); \
|
|
return; \
|
|
} \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_is_lower( uploa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_is_upper( uploa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
uploa, \
|
|
conja, \
|
|
conjx, \
|
|
conjh, /* used by variants to distinguish hemv from symv */ \
|
|
m, \
|
|
alpha, \
|
|
a, rs_a, cs_a, \
|
|
x, incx, \
|
|
beta, \
|
|
y, incy, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC4( hemv, hemv, BLIS_CONJUGATE, hemv_unf_var1, hemv_unf_var3 )
|
|
INSERT_GENTFUNC_BASIC4( symv, hemv, BLIS_NO_CONJUGATE, hemv_unf_var1, hemv_unf_var3 )
|
|
|
|
|
|
#undef GENTFUNCR
|
|
#define GENTFUNCR( ctype, ctype_r, ch, chr, opname, ftname, conjh, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
uplo_t uploa, \
|
|
conj_t conjx, \
|
|
dim_t m, \
|
|
ctype_r* alpha, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
ctype alpha_local; \
|
|
\
|
|
/* If x has zero elements, or if alpha is zero, return early. */ \
|
|
if ( bli_zero_dim1( m ) || PASTEMAC(chr,eq0)( *alpha ) ) return; \
|
|
\
|
|
/* Make a local copy of alpha, cast into the complex domain. This
|
|
allows us to use the same underlying her variants to implement
|
|
both her and syr operations. */ \
|
|
PASTEMAC2(chr,ch,copys)( *alpha, alpha_local ); \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_is_lower( uploa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_is_upper( uploa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
uploa, \
|
|
conjx, \
|
|
conjh, /* used by variants to distinguish her from syr */ \
|
|
m, \
|
|
&alpha_local, \
|
|
x, incx, \
|
|
a, rs_a, cs_a, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNCR_BASIC4( her, her, BLIS_CONJUGATE, her_unb_var1, her_unb_var2 )
|
|
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, conjh, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
uplo_t uploa, \
|
|
conj_t conjx, \
|
|
dim_t m, \
|
|
ctype* alpha, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
/* If x has zero elements, or if alpha is zero, return early. */ \
|
|
if ( bli_zero_dim1( m ) || PASTEMAC(ch,eq0)( *alpha ) ) return; \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_is_lower( uploa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_is_upper( uploa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
uploa, \
|
|
conjx, \
|
|
conjh, /* used by variants to distinguish her2 from syr2 */ \
|
|
m, \
|
|
alpha, \
|
|
x, incx, \
|
|
a, rs_a, cs_a, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC4( syr, her, BLIS_NO_CONJUGATE, her_unb_var1, her_unb_var2 )
|
|
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, conjh, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
uplo_t uploa, \
|
|
conj_t conjx, \
|
|
conj_t conjy, \
|
|
dim_t m, \
|
|
ctype* alpha, \
|
|
ctype* x, inc_t incx, \
|
|
ctype* y, inc_t incy, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
/* If x has zero elements, or if alpha is zero, return early. */ \
|
|
if ( bli_zero_dim1( m ) || PASTEMAC(ch,eq0)( *alpha ) ) return; \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_is_lower( uploa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_is_upper( uploa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
uploa, \
|
|
conjx, \
|
|
conjy, \
|
|
conjh, \
|
|
m, \
|
|
alpha, \
|
|
x, incx, \
|
|
y, incy, \
|
|
a, rs_a, cs_a, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC4( her2, her2, BLIS_CONJUGATE, her2_unf_var1, her2_unf_var4 )
|
|
INSERT_GENTFUNC_BASIC4( syr2, her2, BLIS_NO_CONJUGATE, her2_unf_var1, her2_unf_var4 )
|
|
|
|
|
|
#undef GENTFUNC
|
|
#define GENTFUNC( ctype, ch, opname, ftname, rvarname, cvarname ) \
|
|
\
|
|
void PASTEMAC2(ch,opname,EX_SUF) \
|
|
( \
|
|
uplo_t uploa, \
|
|
trans_t transa, \
|
|
diag_t diaga, \
|
|
dim_t m, \
|
|
ctype* alpha, \
|
|
ctype* a, inc_t rs_a, inc_t cs_a, \
|
|
ctype* x, inc_t incx \
|
|
BLIS_TAPI_EX_PARAMS \
|
|
) \
|
|
{ \
|
|
bli_init_once(); \
|
|
\
|
|
BLIS_TAPI_EX_DECLS \
|
|
\
|
|
/* If x has zero elements, return early. */ \
|
|
if ( bli_zero_dim1( m ) ) return; \
|
|
\
|
|
/* Obtain a valid context from the gks if necessary. */ \
|
|
if ( cntx == NULL ) cntx = bli_gks_query_cntx(); \
|
|
\
|
|
/* If alpha is zero, set x to zero and return early. */ \
|
|
if ( PASTEMAC(ch,eq0)( *alpha ) ) \
|
|
{ \
|
|
PASTEMAC2(ch,setv,BLIS_TAPI_EX_SUF) \
|
|
( \
|
|
BLIS_NO_CONJUGATE, \
|
|
m, \
|
|
alpha, \
|
|
x, incx, \
|
|
cntx, \
|
|
NULL \
|
|
); \
|
|
return; \
|
|
} \
|
|
\
|
|
/* Declare a void function pointer for the current operation. */ \
|
|
PASTECH2(ch,ftname,_ft) f; \
|
|
\
|
|
/* Choose the underlying implementation. */ \
|
|
if ( bli_does_notrans( transa ) ) \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,rvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,cvarname); \
|
|
} \
|
|
else /* if ( bli_does_trans( transa ) ) */ \
|
|
{ \
|
|
if ( bli_is_row_stored( rs_a, cs_a ) ) f = PASTEMAC(ch,cvarname); \
|
|
else /* column or general stored */ f = PASTEMAC(ch,rvarname); \
|
|
} \
|
|
\
|
|
/* Invoke the variant chosen above, which loops over a level-1v or
|
|
level-1f kernel to implement the current operation. */ \
|
|
f( \
|
|
uploa, \
|
|
transa, \
|
|
diaga, \
|
|
m, \
|
|
alpha, \
|
|
a, rs_a, cs_a, \
|
|
x, incx, \
|
|
cntx \
|
|
); \
|
|
}
|
|
|
|
INSERT_GENTFUNC_BASIC3( trmv, trmv, trmv_unf_var1, trmv_unf_var2 )
|
|
INSERT_GENTFUNC_BASIC3( trsv, trmv, trsv_unf_var1, trsv_unf_var2 )
|
|
|
|
|
|
#endif
|
|
|