Simple shell script to cleverly sync an S3 bucket to local folder on an EC2 instance.
#!/bin/bash BUCKET=test-bucket LOG_DIR=/var/log/backup/ BACKUP_DIR=/data/backup/ today=`date '+%Y-%m-%d'` yesterday=`date -d "yesterday 12:00" '+%Y-%m-%d'` cd $BACKUP_DIR; if [ -d $today ]; then echo "Already backed up: $today" return 1; fi; if [ -d $yesterday ]; then echo "Previously backed up: $yesterday. Copying to sync on top of it..." # -p to preserve attribute data so that there is no diff with s3 due to modified-time cp -pr $yesterday $today >> $LOG_DIR/$today.log 2>&1 fi; aws s3 sync s3://$BUCKET/ $today >> $LOG_DIR/$today.log 2>&1
For running this in cron, you need to add a couple more lines to the script
Add a job to run at 2 AM everyday.
# crontab -l 0 2 * * * /home/ubuntu/bin/s3backup
This script needs awscli python package to be installed and configured.