Is there a way to throttle the number of threads or connections on an S3 backup at any given time? We’re currently trying to backup to an internal Mino cluster, but give the number of shards for any given database (400+), and each being very sizable, it looks like the MultiPart upload functionality of the S3 library is making north of 2000 connections to the S3 endpoint at any given time. This is resulting in some intermittent functionality with the backups, with the process occasionally overwhelming a node in the S3 cluster and erroring out with subprocess timeouts. Ideally, I’d like to be able to throttle the connections so that we can control it a little better (as well as figure out where the tipping point is). Does that sort of lever exist for the backup to s3 process?
Hi Steve!
Are you running the managed or self-hosted service and what version number?
Self-hosted 7.0.13. Trying to get the backups right so we can safely upgrade.