Skip to main content

🪣 Memory on S3 / R2 / MinIO

Copy s3.mjs below into your own project and run it with npx tsx s3.mjs.

s3.mjs - the full file​

// s3.mjs — durable: batched + s3 + keyword/local. Needs S3_BUCKET (+ creds/endpoint).
import 'dotenv/config'
import '@lighthouse-ai/store-s3'
import '@lighthouse-ai/embed-keyword'
import '@lighthouse-ai/embed-local'
import { join } from 'node:path'
import { createEmbedder, createStorage } from '@lighthouse-ai/core'
import { BatchedEngine } from '@lighthouse-ai/engine-batched'

if (!process.env.S3_BUCKET) {
  throw new Error('S3_BUCKET is not set. Add it to project-root .env:\n  S3_BUCKET=my-bucket\n  # MinIO also needs S3_ENDPOINT=http://localhost:9000 + S3_FORCE_PATH_STYLE=1')
}

const namespace = process.env.MEMORY_NAMESPACE ?? 'demo-s3'
const dataDir = process.env.MEMORY_DIR ?? join(process.cwd(), '.memory-sdk')

const embedderName = process.env.MEMORY_EMBEDDER ?? 'local'
const embedder = embedderName === 'keyword' ? null : await createEmbedder(embedderName)
console.log('embedder:', embedder?.id ?? 'keyword:off')

// No opts — the adapter reads S3_* from env.
const storage = await createStorage('s3', {})
console.log('storage: s3', {
  bucket: process.env.S3_BUCKET,
  endpoint: process.env.S3_ENDPOINT ?? '(aws)',
  prefix: process.env.S3_PREFIX ?? 'memory-sdk',
})
const memory = new BatchedEngine(storage, { namespace, embedder, dataDir })

// 1. STORE + 2. RETRIEVE
const stored = await memory.remember('User prefers dark mode.', { tags: ['preference'] })
console.log('remembered:', stored)
console.log('recall:', await memory.recall('what are user preferences?'))
console.log('get:', await memory.get(stored.id))
console.log('list:', await memory.list({ limit: 5 }))
console.log('status:', await memory.status())

// 3. DURABILITY — flush pending, then prove rebuild works from S3 blobs.
console.log('flush:', await memory.flush())
console.log('rebuild:', await memory.rebuild())
npx tsx s3.mjs

Why S3?​

S3 (or R2 / MinIO through the same adapter) is the choice when you already have a bucket, need hard-delete semantics, or want CDN-backed reads. Blobs are content-addressed objects under your prefix, forget() is a real delete, and there is no Walrus quota math. This tutorial covers bucket wiring, key layout, and the rebuild proof.

Prerequisites​

  • Offline tutorial completed.
  • The S3 adapter installed: npm install @lighthouse-ai/core @lighthouse-ai/engine-batched @lighthouse-ai/store-s3 @lighthouse-ai/embed-keyword dotenv (swap @lighthouse-ai/embed-keyword for @lighthouse-ai/embed-local for semantic recall).
  • A bucket plus credentials in project-root .env:
S3_BUCKET=your-bucket-name
S3_REGION=us-east-1
# MinIO (docker):
# S3_ENDPOINT=http://localhost:9000
# S3_FORCE_PATH_STYLE=1
# AWS_ACCESS_KEY_ID=minioadmin
# AWS_SECRET_ACCESS_KEY=change-this-password
# R2:
# S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com
# S3_FORCE_PATH_STYLE=1
# Optional:
# S3_PREFIX=memory-sdk
# S3_PUBLIC_BASE_URL=https://cdn.example.com/myprefix
# MEMORY_NAMESPACE=demo-s3
# MEMORY_EMBEDDER=keyword   # word-match only; default is "local" (semantic)

The adapter reads everything from env - createStorage('s3', {}) takes no per-call opts. The demo throws S3_BUCKET is not set… early rather than failing mid-upload.

Step 1 - Understand the key layout​

S3 objects are content-addressed, not sequential (this is what s3.mjs relies on for rebuild):

  • <prefix>/blobs/<cid>.json - the batch blob (same content = same CID, deduplicated naturally).
  • <prefix>/names/<fileName> - per-namespace alias (mem-batch.<namespace>.<batchId>.json) used for listing and rebuild.

Default prefix is memory-sdk (overridable with S3_PREFIX). gatewayUrl() returns S3_PUBLIC_BASE_URL + key when set, else an s3://bucket/prefix/… URI. Set S3_PUBLIC_BASE_URL to your CDN when you want browser-fetchable links.

Step 2 - Store and retrieve (same engine, new backend)​

s3.mjs wires createStorage('s3', {}) (no opts — the adapter reads S3_* from env), then runs the same remember / recall / get / list / status flow. The local index still lives in .memory-sdk/ — S3 holds the blobs, the JSON holds the searchable index. HOW search works (embedder) and WHERE bytes live (storage) stay independent: swap MEMORY_EMBEDDER=keyword for offline word-match without touching the bucket config.

Step 3 - Flush, then prove rebuild works from S3​

s3.mjs flushes the pending batch as one object, then rebuilds from S3 blobs. flush() uploads; rebuild() lists <prefix>/names/mem-batch.<namespace>.* and re-reads them. Because S3 listing is cheap and authoritative, rebuild is typically faster here than on Walrus/Filecoin — but snapshots are still the fast path at scale.

Step 4 - Know the S3 differences​

  • forget() = hard delete (alias always removed; blob removed when unreferenced). No renewal period, no lapse - gone means gone.
  • storage.getBlobIds() returns [] - there are no Walrus-native ids here.
  • MinIO developers: S3_ENDPOINT=http://localhost:9000 + S3_FORCE_PATH_STYLE=1 + the default minioadmin creds give you a local S3 with zero cloud cost. R2 users: same two settings with your account endpoint.

Troubleshooting​

SymptomFix
S3_BUCKET is not setAdd it to project-root .env - your script should read it via dotenv.
AccessDenied / NoSuchBucketRegion/endpoint mismatch (R2 needs path-style + account endpoint; MinIO needs http://localhost:9000 + path-style).
gatewayUrl is s3://…Set S3_PUBLIC_BASE_URL to your CDN so links are HTTPS-fetchable.
Recall misses after embedder switchVectors carry embeddingModel; switching models falls back to keyword until backfilled - expected, not a bug.

Next steps​