Backup S3 buckets daily to EC2 instance via Cron

Simple shell script to cleverly sync an S3 bucket to local folder on an EC2 instance.


~/bin/s3backup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash

BUCKET=test-bucket
LOG_DIR=/var/log/backup/
BACKUP_DIR=/data/backup/

today=`date '+%Y-%m-%d'`
yesterday=`date -d "yesterday 12:00" '+%Y-%m-%d'`

cd $BACKUP_DIR;

if [ -d $today ]; then
echo "Already backed up: $today"
return 1;
fi;

if [ -d $yesterday ]; then
echo "Previously backed up: $yesterday. Copying to sync on top of it..."

# -p to preserve attribute data so that there is no diff with s3 due to modified-time
cp -pr $yesterday $today >> $LOG_DIR/$today.log 2>&1
fi;

aws s3 sync s3://$BUCKET/ $today >> $LOG_DIR/$today.log 2>&1

cron

For running this in cron, you need to add a couple more lines to the script

1
2
PATH=/usr/local/bin:$PATH
HOME=/home/ubuntu/

crontab -e

Add a job to run at 2 AM everyday.

1
2
# crontab -l
0 2 * * * /home/ubuntu/bin/s3backup

This script needs awscli python package to be installed and configured.