Class Matmul

Namespace

NVMathNet.LinAlg

Assembly

NVMath.Net.dll

Stateful cuBLASLt matrix multiplication wrapper. Mirrors the design of nvmath.linalg.Matmul:

1. Construct with dimension info and options.
2. Call NVMathNet.LinAlg.Matmul.PlanAsync(NVMathNet.LinAlg.MatmulPlanPreferences,System.Threading.CancellationToken) to select an algorithm.
3. Call NVMathNet.LinAlg.Matmul.ExecuteAsync(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double,System.Threading.CancellationToken) (possibly multiple times).
4. Dispose to free CUDA resources.

Computes: D = alpha * op(A) * op(B) + beta * C where op() is optional transpose.

public sealed class Matmul : IAsyncDisposable, IDisposable

Inheritance

object

Matmul

Implements

IAsyncDisposable

IDisposable

Inherited Members

object.Equals(object?)

object.Equals(object?, object?)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object?, object?)

object.ToString()

Constructors

Matmul(long, long, long, long, MatmulOptions?, CudaStream?)

Creates a Matmul descriptor.

public Matmul(long m, long n, long k, long batchCount = 1, MatmulOptions? options = null, CudaStream? stream = null)

Parameters

m long

Rows of A and C/D.

n long

Columns of B and C/D.

k long

Inner dimension (cols of A / rows of B).

batchCount long

Number of independent GEMMs in a batched operation. Default: 1.

options MatmulOptions?

Operation options; null uses defaults.

stream CudaStream?

Optional external stream.

Properties

AlgorithmCount

Number of candidate algorithms found by NVMathNet.LinAlg.Matmul.Plan(NVMathNet.LinAlg.MatmulPlanPreferences) (best-first).

public int AlgorithmCount { get; }

Property Value

int

SelectedAlgorithm

Index of the algorithm that NVMathNet.LinAlg.Matmul.Execute(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double) uses (0 = heuristic's top choice).

public int SelectedAlgorithm { get; }

Property Value

int

Methods

Autotune(nint, nint, nint, nint, double, double, int)

Times every candidate algorithm on the given operands and selects the fastest. Only the candidates found while planning are timed: plan with NVMathNet.LinAlg.MatmulPlanPreferences.RequestedAlgoCount greater than 1, otherwise there is a single candidate and nothing to choose between. D is overwritten by the trial runs (and so is C when it aliases D), so run this on scratch data or with beta = 0 before the real computation.

public float[] Autotune(nint pA, nint pB, nint pC, nint pD, double alpha = 1, double beta = 0, int iterations = 3)

Parameters

pA nint

Device pointer to A.

pB nint

Device pointer to B.

pC nint

Device pointer to C.

pD nint

Device pointer to D.

alpha double

Scalar for the product.

beta double

Scalar for C.

iterations int

Timed runs per candidate (after one warm-up run).

Returns

float[]

Average milliseconds per candidate; System.Single.PositiveInfinity for candidates that failed.

AutotuneAsync(nint, nint, nint, nint, double, double, int, CancellationToken)

Asynchronous version of NVMathNet.LinAlg.Matmul.Autotune(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double,System.Int32).

public Task<float[]> AutotuneAsync(nint pA, nint pB, nint pC, nint pD, double alpha = 1, double beta = 0, int iterations = 3, CancellationToken ct = default)

Parameters

pA nint

pB nint

pC nint

pD nint

alpha double

beta double

iterations int

ct CancellationToken

Returns

Task<float[]>

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()

DisposeAsync()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources asynchronously.

public ValueTask DisposeAsync()

Returns

ValueTask

A task that represents the asynchronous dispose operation.

Execute(nint, nint, nint, nint, double, double)

Executes D = alpha*op(A)op(B) + betaC on device buffers. Prefer NVMathNet.LinAlg.Matmul.ExecuteAsync(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double,System.Threading.CancellationToken) in async contexts.

public void Execute(nint pA, nint pB, nint pC, nint pD, double alpha = 1, double beta = 0)

Parameters

pA nint

pB nint

pC nint

pD nint

alpha double

beta double

Execute(nint, nint, nint, nint, Complex, Complex)

Executes D = alpha*op(A)op(B) + betaC with complex scalars (for complex data types and NVMathNet.LinAlg.MatrixDataType.Complex64 / NVMathNet.LinAlg.MatrixDataType.Complex128 scale types). For real scale types only the real part is used.

public void Execute(nint pA, nint pB, nint pC, nint pD, Complex alpha, Complex beta)

Parameters

pA nint

pB nint

pC nint

pD nint

alpha Complex

beta Complex

ExecuteAsync(nint, nint, nint, nint, double, double, CancellationToken)

Asynchronously executes the matrix multiplication and returns a System.Threading.Tasks.Task that completes once the work has been submitted, or once it has finished when NVMathNet.LinAlg.MatmulOptions.Blocking is set.

public Task ExecuteAsync(nint pA, nint pB, nint pC, nint pD, double alpha = 1, double beta = 0, CancellationToken ct = default)

Parameters

pA nint

pB nint

pC nint

pD nint

alpha double

beta double

ct CancellationToken

Returns

Task

ExecuteAsync(nint, nint, nint, nint, Complex, Complex, CancellationToken)

Executes D = alpha*op(A)op(B) + betaC with complex scalars (for complex data types and NVMathNet.LinAlg.MatrixDataType.Complex64 / NVMathNet.LinAlg.MatrixDataType.Complex128 scale types). For real scale types only the real part is used.

public Task ExecuteAsync(nint pA, nint pB, nint pC, nint pD, Complex alpha, Complex beta, CancellationToken ct = default)

Parameters

pA nint

pB nint

pC nint

pD nint

alpha Complex

beta Complex

ct CancellationToken

Returns

Task

ExecuteAsync<T>(DeviceBuffer<T>, DeviceBuffer<T>, DeviceBuffer<T>, DeviceBuffer<T>, Complex, Complex, CancellationToken)

Executes D = alpha*op(A)op(B) + betaC with complex scalars (for complex data types and NVMathNet.LinAlg.MatrixDataType.Complex64 / NVMathNet.LinAlg.MatrixDataType.Complex128 scale types). For real scale types only the real part is used.

public Task ExecuteAsync<T>(DeviceBuffer<T> a, DeviceBuffer<T> b, DeviceBuffer<T> c, DeviceBuffer<T> d, Complex alpha, Complex beta, CancellationToken ct = default) where T : unmanaged, INumberBase<T>

Parameters

a DeviceBuffer<T>

b DeviceBuffer<T>

c DeviceBuffer<T>

d DeviceBuffer<T>

alpha Complex

beta Complex

ct CancellationToken

Returns

Task

Type Parameters

T

ExecuteAsync<T>(DeviceBuffer<T>, DeviceBuffer<T>, DeviceBuffer<T>, DeviceBuffer<T>, double, double, CancellationToken)

Asynchronously executes the matrix multiplication and returns a System.Threading.Tasks.Task that completes once the work has been submitted, or once it has finished when NVMathNet.LinAlg.MatmulOptions.Blocking is set.

public Task ExecuteAsync<T>(DeviceBuffer<T> a, DeviceBuffer<T> b, DeviceBuffer<T> c, DeviceBuffer<T> d, double alpha = 1, double beta = 0, CancellationToken ct = default) where T : unmanaged, INumberBase<T>

Parameters

a DeviceBuffer<T>

b DeviceBuffer<T>

c DeviceBuffer<T>

d DeviceBuffer<T>

alpha double

beta double

ct CancellationToken

Returns

Task

Type Parameters

T

Plan(MatmulPlanPreferences?)

Synchronously selects the best algorithm.

public void Plan(MatmulPlanPreferences? preferences = null)

Parameters

preferences MatmulPlanPreferences?

Optional fine-grained algorithm preferences. null uses the defaults from the NVMathNet.LinAlg.MatmulOptions supplied at construction.

PlanAsync(MatmulPlanPreferences?, CancellationToken)

Selects the best algorithm on a background thread.

public Task PlanAsync(MatmulPlanPreferences? preferences = null, CancellationToken ct = default)

Parameters

preferences MatmulPlanPreferences?

Optional fine-grained algorithm preferences. null uses the defaults from the NVMathNet.LinAlg.MatmulOptions supplied at construction.

ct CancellationToken

Cancellation token.

Returns

Task

SelectAlgorithm(int)

Selects which planned candidate algorithm NVMathNet.LinAlg.Matmul.Execute(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double) uses. Request several candidates through NVMathNet.LinAlg.MatmulPlanPreferences.RequestedAlgoCount.

public void SelectAlgorithm(int index)

Parameters

index int

Candidate index in [0, AlgorithmCount).

SetBias(nint, MatrixDataType?, long)

Sets the bias vector used by the Bias family of epilogs.

public void SetBias(nint bias, MatrixDataType? biasType = null, long batchStride = 0)

Parameters

bias nint

Device pointer to the bias vector (length M).

biasType MatrixDataType?

Element type of the bias; null keeps the cuBLASLt default.

batchStride long

Stride in elements between batches; 0 shares one bias.

SetEpilogAux(nint, long, MatrixDataType?, long)

Sets the auxiliary output written by the *Aux epilogs (NVMathNet.LinAlg.MatmulEpilog.GeluAux, NVMathNet.LinAlg.MatmulEpilog.GeluAuxBias, NVMathNet.LinAlg.MatmulEpilog.ReluAux, NVMathNet.LinAlg.MatmulEpilog.ReluAuxBias), used by the backward pass.

public void SetEpilogAux(nint aux, long ld, MatrixDataType? auxType = null, long batchStride = 0)

Parameters

aux nint

Device pointer to the aux buffer (16-byte aligned).

ld long

Leading dimension in elements; for GELU a multiple of 8 and at least M. For ReLU the aux buffer is a bit mask and ld counts bits (a multiple of 128).

auxType MatrixDataType?

Element type of the GELU aux output; null keeps the cuBLASLt default.

batchStride long

Stride between batches in elements (bits for ReLU); 0 for non-batched.

SetScalePointers(nint, nint, nint, nint, nint)

Sets the device pointers to the scalar scale factors used by FP8 matmul and the optional amax(D) output. Pass 0 to leave a pointer unchanged. Call before NVMathNet.LinAlg.Matmul.Execute(System.IntPtr,System.IntPtr,System.IntPtr,System.IntPtr,System.Double,System.Double).

public void SetScalePointers(nint aScale = 0, nint bScale = 0, nint cScale = 0, nint dScale = 0, nint amaxD = 0)

Parameters

aScale nint

Device pointer to the float scale of A.

bScale nint

Device pointer to the float scale of B.

cScale nint

Device pointer to the float scale of C.

dScale nint

Device pointer to the float scale of D.

amaxD nint

Device pointer receiving the absolute maximum of D.

SynchronizeAsync(CancellationToken)

Waits for all work on the internal stream to complete.

public Task SynchronizeAsync(CancellationToken ct = default)

Parameters

ct CancellationToken

Returns

Task