Skip to main content

Back up Amazon S3

Mirror the contents of an S3 bucket (or prefix) to a local directory so Lighthouse can keep an independent, versioned copy outside AWS — useful for off-cloud redundancy and protection against accidental bucket deletion.

Prerequisites​

  • The AWS CLI v2.
  • AWS credentials with s3:ListBucket and s3:GetObject on the source bucket.

1. Sign in & create an API key​

You need a Lighthouse account, a workspace, and an API key before you can upload a backup.

  1. Sign in to the portal and claim your free 5 GB workspace — see Web Portal & Free Workspace.

  2. Create an API key scoped backup:write, backup:read, snapshots:read — see API Keys.

  3. Export the credentials so the upload step can read them:

    export LH_API_KEY="lh_xxxxxxxxxxxxxxxxxxxxxxxx"
    export LH_WORKSPACE_ID="your-workspace-uuid"

Already have a key? Continue to the sync step below.

2. Sync the bucket locally​

Create the local dump directory once:

mkdir -p ./db-dumps

aws s3 sync downloads only objects that are new or changed since the last run — a natural fit for the "overwrite in place, upload incrementally" pattern.

aws s3 sync s3://your-bucket ./db-dumps/your-bucket

Common refinements:

# Only a prefix/folder:
aws s3 sync s3://your-bucket/path/ ./db-dumps/your-bucket

# Mirror deletions too (remove local files deleted in S3):
aws s3 sync s3://your-bucket ./db-dumps/your-bucket --delete

# Limit by pattern:
aws s3 sync s3://your-bucket ./db-dumps/your-bucket \
  --exclude "*" --include "*.json"
Server-side vs. off-cloud copy

If you only need redundancy within AWS, S3 cross-region replication may be simpler. Use this guide when you specifically want a copy outside AWS (on Lighthouse).

3. Verify the sync​

du -sh ./db-dumps/your-bucket
find ./db-dumps/your-bucket -type f | wc -l   # local object count

# Compare against the bucket's object count:
aws s3 ls s3://your-bucket --recursive --summarize | tail -n 2

4. Restore (recovery test)​

Push the local copy back into a different bucket/prefix to confirm it round-trips, without overwriting the source:

aws s3 sync ./db-dumps/your-bucket s3://your-restore-bucket

5. Upload to Lighthouse​

Your mirror is now in ./db-dumps. After creating an authenticated SDK client with the API key from step 1, upload the directory as a snapshot:

snapshot, err := client.Backup([]string{"./db-dumps"}, &sdktypes.BackupOptions{})
if err != nil {
	log.Fatal(err)
}
log.Printf("snapshotId=%s", snapshot.SnapshotID)

To run this on a schedule, use Automated backup with scheduling. Because both aws s3 sync and the SDK upload are incremental, repeated runs only transfer what changed.

The sync command for this bucket, ready to drop into the scheduled job's make_dump():

aws s3 sync s3://your-bucket ./db-dumps/your-bucket --delete