Class CachingMemoryManager

Namespace

NVMathNet.Memory

Assembly

NVMath.Net.dll

A memory manager that keeps freed blocks and hands them out again, avoiding repeated cudaMalloc/cudaFree calls (which are slow and synchronise the device). Requests up to 1 GiB are rounded up to a power of two (minimum 512 bytes) so blocks are interchangeable; larger requests are cached by exact size. Blocks are cached per CUDA device: a block is only handed out again to a caller whose current device is the one it was allocated on.

public sealed class CachingMemoryManager : IDeviceMemoryManager, IDisposable

Inheritance

object

CachingMemoryManager

Implements

IDeviceMemoryManager

IDisposable

Inherited Members

object.Equals(object?)

object.Equals(object?, object?)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object?, object?)

object.ToString()

Remarks

A freed block can be handed out again immediately, so work that still uses it must have finished first. Synchronise the stream before disposing a NVMathNet.DeviceBuffer`1 (or an object that owns one) whenever kernels that touch it may still be queued.

Constructors

CachingMemoryManager(IDeviceMemoryManager?)

Creates a caching manager on top of inner.

public CachingMemoryManager(IDeviceMemoryManager? inner = null)

Parameters

inner IDeviceMemoryManager?

The underlying allocator; null uses NVMathNet.Memory.CudaMallocMemoryManager.Instance.

Properties

CachedBytes

Bytes currently held in the cache (allocated from the inner manager but not in use).

public long CachedBytes { get; }

Property Value

long

InUseBytes

Bytes currently handed out to callers (including rounding).

public long InUseBytes { get; }

Property Value

long

Methods

Allocate(nuint)

Allocates sizeInBytes bytes of device memory.

public nint Allocate(nuint sizeInBytes)

Parameters

sizeInBytes nuint

Requested size; always greater than zero.

Returns

nint

A device pointer that stays valid until passed to NVMathNet.Memory.IDeviceMemoryManager.Free(System.IntPtr).

Dispose()

Trims the cache and stops caching. Blocks still in use are released to the inner manager when they are freed.

public void Dispose()

Free(nint)

Releases memory previously returned by NVMathNet.Memory.IDeviceMemoryManager.Allocate(System.UIntPtr) on this manager. A pooling manager may hand the block out again immediately, so callers must make sure no queued GPU work still uses it (synchronise the stream first).

public void Free(nint pointer)

Parameters

pointer nint

The device pointer to release.

Trim()

Returns all cached (unused) blocks to the inner manager.

public void Trim()