Simple shell script to cleverly sync an S3 bucket to local folder on an EC2 instance.


~/bin/s3backup

#!/bin/bash

BUCKET=test-bucket
LOG_DIR=/var/log/backup/
BACKUP_DIR=/data/backup/

today=`date '+%Y-%m-%d'`
yesterday=`date -d "yesterday 12:00" '+%Y-%m-%d'`

cd $BACKUP_DIR;

if [ -d $today ]; then
    echo "Already backed up: $today"
    return 1;
fi;

if [ -d $yesterday ]; then
    echo "Previously backed up: $yesterday. Copying to sync on top of it..."

    # -p to preserve attribute data so that there is no diff with s3 due to modified-time
    cp -pr $yesterday $today >> $LOG_DIR/$today.log 2>&1
fi;

aws s3 sync s3://$BUCKET/ $today >> $LOG_DIR/$today.log 2>&1

cron

For running this in cron, you need to add a couple more lines to the script

PATH=/usr/local/bin:$PATH
HOME=/home/ubuntu/

crontab -e

Add a job to run at 2 AM everyday.

# crontab -l
0 2 * * * /home/ubuntu/bin/s3backup

This script needs awscli python package to be installed and configured.