Table of Contents

A background worker

A partner drops files into an outbox directory on their SFTP server. A worker service fetches new files every five minutes, saves them locally, and moves each one to done on the server once it has it. It runs for months: connections drop, the partner's server restarts, and the worker carries on.

The approach

  • A hosted service (BackgroundService) with a PeriodicTimer, on the generic host.
  • The shared connection from dependency injection (ISshConnectionProvider): opened on first use and again after a drop, with a growing delay after failures, so a server that is down is not hammered.
  • An SFTP session per run, closed at the end of the run; the SSH connection stays open between runs.
  • At-least-once processing. A file is moved to done only after it is saved, so a failure in between means it is fetched again at the next run, never lost. Make the downstream processing idempotent.

The code

// Program.cs
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddSshClient("partner")
    .BindConfiguration("Ssh:Partner")
    .ValidateOnStart();
builder.Services.AddHostedService<IngestWorker>();
await builder.Build().RunAsync();

/// <summary>Every few minutes, downloads new files from a partner's outbox and moves them to "done" on the server.</summary>
internal sealed class IngestWorker(ISshConnectionProvider connections, IConfiguration configuration, ILogger<IngestWorker> logger)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var inbox = configuration["Ingest:LocalDirectory"] ?? "inbox";
        Directory.CreateDirectory(inbox);
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(5));
        do
        {
            try
            {
                var count = await RunOnceAsync(inbox, stoppingToken);
                logger.LogInformation("Fetched {Count} files", count);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                return;
            }
            catch (Exception ex) when (ex is SshException or IOException or TimeoutException)
            {
                // The connection dropped or could not be opened; the provider opens a new one on the next run, backing off
                // after failures. Files not yet moved to "done" are fetched again then.
                logger.LogWarning(ex, "Fetching failed; trying again at the next run");
            }
        }
        while (await timer.WaitForNextTickAsync(stoppingToken));
    }

    private async Task<int> RunOnceAsync(string inbox, CancellationToken cancellationToken)
    {
        var client = await connections.GetClientAsync("partner", cancellationToken);   // shared, reconnecting: do not dispose
        await using var sftp = await client.OpenSftpAsync(cancellationToken);          // a session per run

        var count = 0;
        await foreach (var entry in sftp.ListDirectoryAsync("outbox", cancellationToken))
        {
            if (!entry.Attributes.IsRegularFile || entry.Name.EndsWith(".partial", StringComparison.Ordinal))
            {
                continue;   // still being written by the partner
            }

            var local = Path.Combine(inbox, Path.GetFileName(entry.Name));
            await sftp.DownloadFileAsync(entry.FullPath, local, new SftpTransferOptions { Preserve = true }, cancellationToken);
            await sftp.RenameAsync(entry.FullPath, $"done/{entry.Name}", overwrite: true, cancellationToken);
            count++;
        }

        return count;
    }
}

appsettings.json:

{
  "Ssh": {
    "Partner": {
      "Host": "sftp.example.com",
      "UserName": "user",
      "PrivateKeyFiles": [ "/run/secrets/partner_ed25519" ],
      "KnownHostsFiles": [ "/etc/ingest/known_hosts" ],
      "KeepAliveInterval": "00:00:30"
    }
  },
  "Ingest": { "LocalDirectory": "/var/lib/ingest/inbox" }
}

For production

  • Secrets. Mount the key (and any passphrase) from your platform's secret store, as above; do not bake it into the image.
  • Host key. Ship the partner's host key in a known_hosts file. AcceptNewHostKeys would trust whatever answers first.
  • Health. Add builder.Services.AddHealthChecks().AddSsh("partner") and expose it to your orchestrator.
  • Partial files. The example skips names ending in .partial; agree on a convention with the partner (upload under a temporary name, then rename), or wait for a file's size to stop changing.
  • Large files. Add Resume = true to continue interrupted downloads, or Segments for faster downloads over long distances.

See also

Edit this page