Table of Contents

A fleet health check

Check a few hundred servers in a minute: connect to many at once, run one compound command on each, and report every host, whether it answered, failed or timed out. One slow or unreachable host must not hold up the report.

The approach

  • Bounded parallelism. Parallel.ForEachAsync with MaxDegreeOfParallelism keeps a fixed number of connections open at a time, so the checking machine, the network and any bastion are not flooded.
  • A time limit per host, linked to the overall token, so a host that accepts TCP and then stalls costs at most that limit.
  • One connection per host, one command per connection. Several checks are chained in one command line, so each host costs one round of connect, authenticate, run.
  • Every failure is a result. Exceptions become report lines with the SshFailureReason, which already says whether the host was unreachable, refused the login, or presented an unknown host key.

The code

public sealed record HostReport(string Host, bool Ok, string Detail, TimeSpan Elapsed);

/// <summary>Checks many hosts at once, each with its own connection and time limit, and reports every host, reachable or not.</summary>
public static async Task<IReadOnlyList<HostReport>> CheckAsync(IEnumerable<string> hosts, CancellationToken cancellationToken)
{
    var reports = new ConcurrentBag<HostReport>();
    var parallel = new ParallelOptions { MaxDegreeOfParallelism = 16, CancellationToken = cancellationToken };

    await Parallel.ForEachAsync(hosts, parallel, async (host, ct) =>
    {
        var started = TimeProvider.System.GetTimestamp();
        using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
        timeout.CancelAfter(TimeSpan.FromSeconds(20));   // one slow host must not hold up the report

        try
        {
            // Each host by its ssh_config alias (or user@host). No prompts are possible here, so an unknown host key fails.
            await using var client = await SshClient.ConnectAsync(host, cancellationToken: timeout.Token);
            var result = await client.RunCommandAsync(
                "cat /proc/loadavg; df -P / | tail -1; systemctl is-system-running", timeout.Token);
            var lines = result.StandardOutputText.Split('\n', StringSplitOptions.RemoveEmptyEntries);
            var diskUsed = lines.Length > 1 ? lines[1].Split(' ', StringSplitOptions.RemoveEmptyEntries)[4] : "?";
            var state = lines.Length > 2 ? lines[2] : "?";
            var ok = state is "running" && diskUsed.TrimEnd('%') is var d && int.TryParse(d, out var used) && used < 90;
            reports.Add(new(host, ok, $"load {lines.FirstOrDefault()?.Split(' ')[0]}, disk {diskUsed}, {state}", Elapsed(started)));
        }
        catch (OperationCanceledException) when (!ct.IsCancellationRequested)
        {
            reports.Add(new(host, false, "timed out", Elapsed(started)));
        }
        catch (SshException ex)
        {
            reports.Add(new(host, false, $"{ex.Reason}: {ex.Message}", Elapsed(started)));
        }
    });

    return [.. reports.OrderBy(r => r.Ok).ThenBy(r => r.Host, StringComparer.Ordinal)];
}

private static TimeSpan Elapsed(long started) => TimeProvider.System.GetElapsedTime(started);

// fleet-check hosts.txt: one alias or user@host per line.
var hosts = (await File.ReadAllLinesAsync(args[0])).Where(l => l.Length > 0 && !l.StartsWith('#'));
var reports = await CheckAsync(hosts, CancellationToken.None);
foreach (var r in reports)
{
    Console.WriteLine($"{(r.Ok ? "ok  " : "FAIL")} {r.Host,-30} {r.Elapsed.TotalSeconds,5:F1}s  {r.Detail}");
}

return reports.All(r => r.Ok) ? 0 : 1;

Sample output:

FAIL app-17                           20.0s  timed out
FAIL db-3                              0.4s  HostKeyRejected: The host key SHA256:... is not in known_hosts, and StrictHostKeyChecking is yes.
ok   app-1                             0.9s  load 0.12, disk 41%, running
ok   app-2                             1.1s  load 0.30, disk 57%, running

For production

  • Unattended runs. Without AskHostKey, an unknown host fails instead of prompting; keep StrictHostKeyChecking yes and manage known_hosts (or @cert-authority lines) with your provisioning.
  • Bastions. With ProxyJump, every host makes its own connection to the bastion. With many hosts, raise the bastion's MaxStartups and MaxSessions as needed, or connect to the bastion once and open each target over OpenTcpTunnelAsync (see Jump hosts: by hand).
  • Output parsing. The parsing here is deliberately simple. For anything richer, have the server print JSON (jq, or a small script you deploy) and deserialize it.
  • Metrics. Instead of printing, record a gauge per host, or let SshSharp's own metrics (sshsharp.connect.duration and sshsharp.commands, see Logging and diagnostics) feed your dashboards.

See also

Edit this page