Retries and timeouts

What is retried

Unary reads (Get*, List*, and Health) that fail with Unavailable are retried, up to four attempts in all, with exponential backoff and jitter. Writes are never retried by default, because a lost response could repeat a side effect. Retries stop at the call's deadline and when the caller cancels. Streaming calls are not retried: Watch.WatchAsync reconnects on its own, and a broken exec or forward stream surfaces its error.

RetryPolicy controls this through OpenShellClientOptions.Retry:

var client = new OpenShellClient(new OpenShellClientOptions
{
    Address = new Uri("https://gateway.example.com"),
    Retry = RetryPolicy.Default with
    {
        MaxAttempts = 6,                          // the first attempt plus five retries
        MaxBackoff = TimeSpan.FromSeconds(10),    // a gateway RetryInfo delay above this ends the retries
    },
    DefaultTimeout = TimeSpan.FromSeconds(30),    // deadline for each unary call, retries included
});

Setting	Default
MaxAttempts	4 (the first attempt included; 1 disables retries)
InitialBackoff	100 ms
MaxBackoff	5 s
BackoffMultiplier	2
Jitter	0.2 (the fraction of each delay that is randomized)
RetryableStatusCodes	Unavailable
IsIdempotent	reads: Get*, List*, Health

OpenShellClientOptions.DefaultTimeout sets a deadline for unary calls that do not set their own; null (the default) means none.

Server-suggested delays and MUTATION_LOCK_TIMEOUT

When the gateway attaches a retry delay (google.rpc.RetryInfo), the next attempt waits at least that long. A delay longer than MaxBackoff ends the retries and surfaces the error, with the delay in OpenShellException.RetryDelay.

The gateway does this, for example, when a mutation waited 10 seconds for the lock of a sandbox or workspace that another mutation held: the call fails with Unavailable, an ErrorInfo whose Reason is MUTATION_LOCK_TIMEOUT, and a retry delay. Because the failed call is a write, the SDK does not repeat it; decide in your code whether repeating it is safe:

try
{
    await client.Sandboxes.StopAsync("agent-1");
}
catch (OpenShellException e) when (e.ErrorInfo?.Reason == "MUTATION_LOCK_TIMEOUT")
{
    // Another mutation held the sandbox lock for longer than the gateway waits (10 s). The gateway
    // suggests when to try again; whether repeating this write is safe is your decision.
    await Task.Delay(e.RetryDelay ?? TimeSpan.FromSeconds(1));
    await client.Sandboxes.StopAsync("agent-1");
}

RetryDelay says when to try again, not whether repeating a mutation is safe.

Retrying writes

To retry a write you have made safe to repeat, for example one that always carries a request id, widen IsIdempotent. It receives the gRPC method name, such as GetSandbox or CreateSandbox. Use RetryPolicy.None to turn retries off.

// Writes are not retried by default. Widen IsIdempotent for methods you make safe to repeat, for
// example by always sending a request id. It receives the gRPC method name.
var client = new OpenShellClient(new OpenShellClientOptions
{
    Address = new Uri("https://gateway.example.com"),
    Retry = RetryPolicy.Default with
    {
        IsIdempotent = method => method.StartsWith("Get", StringComparison.Ordinal)
            || method.StartsWith("List", StringComparison.Ordinal)
            || method is "Health" or "CreateSandbox",
    },
});

// Or turn retries off.
var noRetries = new OpenShellClient(new OpenShellClientOptions
{
    Address = new Uri("https://gateway.example.com"),
    Retry = RetryPolicy.None,
});

Most mutations accept a request id (RequestId on the options records, or a requestId parameter): the gateway answers a repeated request with the original result, which makes a retry safe.

Cancellation

Every method takes a CancellationToken. Cancelling it throws OperationCanceledException (not OpenShellException), also in the middle of retries and waits.

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
try
{
    await client.Exec.RunAsync("agent-1", ["sleep", "60"], cancellationToken: cts.Token);
}
catch (OperationCanceledException)
{
    // Your own cancellation surfaces as OperationCanceledException, not OpenShellException.
}