Skip to main content

Back up Amazon DynamoDB

DynamoDB is a managed NoSQL service, so a "dump" means exporting table items to a local file you can upload. This guide covers the two common approaches: a quick scan for small/medium tables, and the AWS-native point-in-time export to S3 for large tables.

Prerequisites​

  • The AWS CLI v2:
    • macOS: brew install awscli
    • Linux: AWS-provided installer
  • AWS credentials configured (aws configure) with at least dynamodb:Scan (and dynamodb:DescribeTable) on the table.

1. Sign in & create an API key​

You need a Lighthouse account, a workspace, and an API key before you can upload a backup.

  1. Sign in to the portal and claim your free 5 GB workspace — see Web Portal & Free Workspace.

  2. Create an API key scoped backup:write, backup:read, snapshots:read — see API Keys.

  3. Export the credentials so the upload step can read them:

    export LH_API_KEY="lh_xxxxxxxxxxxxxxxxxxxxxxxx"
    export LH_WORKSPACE_ID="your-workspace-uuid"

Already have a key? Continue to the dump step below.

2. Create the dump​

Create the local dump directory once:

mkdir -p ./db-dumps

Option A — scan to JSON (simple, small/medium tables)​

aws dynamodb scan \
  --table-name app_table \
  --output json \
  > ./db-dumps/app_table.json

This writes every item (in DynamoDB JSON format) to one file. The AWS CLI paginates automatically — DynamoDB returns at most 1 MB per underlying request, but the CLI follows LastEvaluatedKey for you and emits a single merged Items array, so the command above is complete as written.

For a large table, write newline-delimited JSON instead — it streams, diffs better between runs, and avoids holding one huge array in memory:

aws dynamodb scan --table-name app_table --output json \
  | jq -c '.Items[]' > ./db-dumps/app_table.ndjson
Don't hand-roll pagination

--starting-token takes the CLI's own opaque pagination token, not the LastEvaluatedKey attribute map from the response — feeding one into the other fails. Let the CLI paginate, or use --no-paginate with --starting-token only if you are deliberately checkpointing across runs.

Cost & throughput

A full scan reads the entire table and consumes read capacity. For large or production tables, prefer Option B (managed export), which does not consume table capacity.

Option B — Point-in-time export to S3 (large tables, no capacity cost)​

Requires point-in-time recovery (PITR) enabled on the table.

aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/app_table \
  --s3-bucket your-export-bucket \
  --export-format DYNAMODB_JSON

Then pull the export down so the BaaS job can upload it (see Amazon S3):

aws s3 sync s3://your-export-bucket/AWSDynamoDB ./db-dumps/dynamodb-export

3. Verify the dump​

ls -lh ./db-dumps/app_table.json
jq '.Items | length' ./db-dumps/app_table.json   # item count; also proves it parses

If you used the newline-delimited form:

jq -s 'length' ./db-dumps/app_table.ndjson   # item count

4. Restore (recovery test)​

For the scan output, re-import items with batch-write-item. The API accepts a maximum of 25 items per call, so the items must be chunked — _nwise does that:

jq -c '[.Items[] | { PutRequest: { Item: . } }] | _nwise(25)
       | { "app_table_restore": . }' ./db-dumps/app_table.json \
  | while read -r batch; do
      aws dynamodb batch-write-item --request-items "$batch"
    done

For the newline-delimited form, slurp it first:

jq -s -c '[.[] | { PutRequest: { Item: . } }] | _nwise(25)
          | { "app_table_restore": . }' ./db-dumps/app_table.ndjson \
  | while read -r batch; do
      aws dynamodb batch-write-item --request-items "$batch"
    done
Check for unprocessed items

batch-write-item can partially succeed under throttling and return the remainder in UnprocessedItems. For anything beyond a small recovery test, capture the response and retry those items with backoff.

For Option B exports, use AWS's import-table from S3.

5. Upload to Lighthouse​

Your export is now in ./db-dumps. Construct the SDK client with the API key from step 1 and upload the directory as a snapshot. This reads LH_API_KEY and LH_WORKSPACE_ID from the environment (exported in step 1):

package main

import (
	"log"
	"os"

	sdkclient "github.com/lighthouse-web3/baas-go-sdk/client"
	sdktypes "github.com/lighthouse-web3/baas-go-sdk/types"
)

func main() {
	client, err := sdkclient.NewBackupClient(sdkclient.BackupClientOptions{
		APIURL:      "https://baas-api.lighthouse.storage", // API host, not the portal
		APIKey:      os.Getenv("LH_API_KEY"),
		WorkspaceID: os.Getenv("LH_WORKSPACE_ID"),
	})
	if err != nil {
		log.Fatalf("client init: %v", err)
	}

	snapshot, err := client.Backup([]string{"./db-dumps"}, &sdktypes.BackupOptions{
		Description: "dynamodb export",
	})
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("snapshotId=%s", snapshot.SnapshotID)
}

Run it from the directory containing db-dumps:

go run .

To run this on a schedule, use Automated backup with scheduling.

The dump command for this table, ready to drop into the scheduled job's make_dump():

aws dynamodb scan --table-name app_table --output json > ./db-dumps/app_table.json