A deployment script

Deploy a release to a set of servers, one at a time: upload the archive, unpack it next to the previous releases, switch a current symlink atomically, restart the service, check its health, and roll back if the check fails. Everything goes over one SSH connection per server: the upload (SFTP), the commands, and even the HTTP health check.

The approach

Connect by ssh_config alias, so bastions, the deploy key and known_hosts are configuration, not code.
Upload with Resume, so a retry after a dropped connection sends only the rest.
Releases side by side (/srv/myapp/releases/<version>), with current a symbolic link. Switching is creating a new link and renaming it over the old one: rename(2) is atomic, so current always points at a complete release.
Health check through the connection: CreateHttpConnectCallback lets HttpClient reach http://localhost:8080 on the server, without opening the service's port to the network.
Stop at the first failure, leaving the rest of the fleet on the old version.

The code

/// <summary>
/// Uploads a release, unpacks it next to the previous ones, switches the "current" link atomically, restarts the service and
/// checks its health; on a failed health check, switches back.
/// </summary>
public static async Task<bool> DeployAsync(string host, string archive, string version, CancellationToken cancellationToken)
{
    const string root = "/srv/myapp";
    var release = $"{root}/releases/{version}";
    var upload = $"{root}/incoming/{version}.tar.gz";

    // By ssh_config alias: the bastion (ProxyJump), the deploy key and known_hosts all come from the configuration.
    await using var client = await SshClient.ConnectAsync(host, cancellationToken: cancellationToken);
    await using var sftp = await client.OpenSftpAsync(cancellationToken);

    // 1. Upload. Resume makes a retry after a dropped connection send only the rest.
    await sftp.UploadFileAsync(archive, upload, new SftpTransferOptions
    {
        Resume = true,
        Progress = new Progress<SftpTransferProgress>(p => Console.Write($"\rupload {p.BytesTransferred * 100 / Math.Max(1, p.TotalBytes ?? 1)}%")),
    }, cancellationToken);
    Console.WriteLine();

    // 2. Unpack beside the other releases.
    await RunAsync(client, $"mkdir -p {Quote(release)} && tar -xzf {Quote(upload)} -C {Quote(release)} && rm -f {Quote(upload)}", cancellationToken);

    // 3. Switch: a new link, renamed over the old one, so "current" always points at a complete release.
    var previous = await sftp.ReadSymbolicLinkAsync($"{root}/current", cancellationToken);
    await SwitchAsync(sftp, root, release, cancellationToken);

    // 4. Restart and check, through the same connection: "localhost" is the server's own loopback.
    await RunAsync(client, "sudo systemctl restart myapp", cancellationToken);
    if (await IsHealthyAsync(client, cancellationToken))
    {
        Console.WriteLine($"{host}: {version} is live");
        return true;
    }

    // 5. Roll back.
    Console.Error.WriteLine($"{host}: {version} failed its health check; rolling back to {previous}");
    await SwitchAsync(sftp, root, previous, cancellationToken);
    await RunAsync(client, "sudo systemctl restart myapp", cancellationToken);
    return false;
}

private static async Task SwitchAsync(SftpClient sftp, string root, string target, CancellationToken cancellationToken)
{
    var next = $"{root}/current.next";
    if (await sftp.ExistsAsync(next, cancellationToken))
    {
        await sftp.DeleteFileAsync(next, cancellationToken);
    }

    await sftp.CreateSymbolicLinkAsync(next, target, cancellationToken);
    await sftp.RenameAsync(next, $"{root}/current", overwrite: true, cancellationToken);   // rename(2): atomic
}

private static async Task<bool> IsHealthyAsync(SshClient client, CancellationToken cancellationToken)
{
    using var http = new HttpClient(new SocketsHttpHandler { ConnectCallback = client.CreateHttpConnectCallback(), UseProxy = false })
    {
        Timeout = TimeSpan.FromSeconds(5),
    };

    for (var attempt = 0; attempt < 10; attempt++)
    {
        try
        {
            using var response = await http.GetAsync(new Uri("http://localhost:8080/health"), cancellationToken);
            if (response.IsSuccessStatusCode)
            {
                return true;
            }
        }
        catch (HttpRequestException)
        {
            // Not listening yet.
        }

        await Task.Delay(TimeSpan.FromSeconds(3), cancellationToken);
    }

    return false;
}

/// <summary>Runs a command and throws, with its stderr, when it fails.</summary>
private static async Task RunAsync(SshClient client, string command, CancellationToken cancellationToken)
{
    var result = await client.RunCommandAsync(command, cancellationToken);
    if (result.ExitStatus != 0)
    {
        throw new InvalidOperationException($"'{command}' failed with exit status {result.ExitStatus}: {result.StandardErrorText.Trim()}");
    }
}

private static string Quote(string value) => "'" + value.Replace("'", "'\\''", StringComparison.Ordinal) + "'";

// deploy <version> <archive> <host>...: one host after another, stopping at the first failure.
using var cancel = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cancel.Cancel(); };

var (version, archive, hosts) = (args[0], args[1], args[2..]);
foreach (var host in hosts)
{
    if (!await DeployAsync(host, archive, version, cancel.Token))
    {
        return 1;   // the rest of the fleet keeps the old version
    }
}

return 0;

A matching ~/.ssh/config:

Host bastion
  HostName bastion.example.com
  User jump

Host web-*
  HostName %h.internal
  User deploy
  ProxyJump bastion
  IdentityFile ~/.ssh/deploy_ed25519
  StrictHostKeyChecking yes

Run it as deploy 2.4.1 myapp-2.4.1.tar.gz web-1 web-2 web-3.

For production

First deployment. The script reads the previous current link to roll back to; on a fresh server, create it first (or treat a missing link as "nothing to roll back to").
sudo. sudo systemctl restart must not prompt: give the deploy user a NOPASSWD rule for exactly that command. If it must prompt, see Interactive shells and SshExpect.
Old releases. Remove releases beyond the last few (ls -1t releases | tail -n +6 | xargs rm -rf, quoted for your paths).
Parallel rollout. To deploy to several servers at once, run DeployAsync for a batch with Task.WhenAll; see the fleet health check for bounded parallelism.
Logging. Pass a logger through SshConfigConnectOptions.Logger to record which algorithms and host keys each server used.

Table of Contents

A deployment script

The approach

The code

For production

See also