Backing up a large Postgres DB to S3
I recently backed up a large PostgreSQL database, spanning a few dozen terabytes, from AWS RDS to an AWS S3 bucket in a resource-constrained environment. The process wasn’t as straightforward as I imagined it would be, so I decided to share everything I learned. Let’s say we have limited storage on the machine where we want to back up the database – meaning the size of the backup exceeds the total storage capacity of the machine. Additionally, our priority is the speed at which this backup is taken rather than the cost of storing it in S3. Let’s explore how we can perform a backup under these constraints. If you don’t have the same constraints, you may want to tweak a few parameters which we’ll discuss after the tl;dr section. ...