Skip to main content

🪣 Memory on S3 / R2 / MinIO

Copy s3.mjs below into your own project and run it with npx tsx s3.mjs.

tip

Using the hosted dashboard or MCP server rather than running the SDK yourself? You can connect a bucket there without any code - see Connect Your Own Bucket (Dashboard).

s3.mjs - the full file​

// s3.mjs — durable: batched + s3 + keyword/local. Needs S3_BUCKET (+ creds/endpoint).
import 'dotenv/config'
import '@lighthouse-ai/store-s3'
import '@lighthouse-ai/embed-keyword'
import '@lighthouse-ai/embed-local'
import { join } from 'node:path'
import { createEmbedder, createStorage } from '@lighthouse-ai/core'
import { BatchedEngine } from '@lighthouse-ai/engine-batched'

if (!process.env.S3_BUCKET) {
  throw new Error('S3_BUCKET is not set. Add it to project-root .env:\n  S3_BUCKET=my-bucket\n  # MinIO also needs S3_ENDPOINT=http://localhost:9000 + S3_FORCE_PATH_STYLE=1')
}

const namespace = process.env.MEMORY_NAMESPACE ?? 'demo-s3'
const dataDir = process.env.MEMORY_DIR ?? join(process.cwd(), '.memory-sdk')

const embedderName = process.env.MEMORY_EMBEDDER ?? 'local'
const embedder = embedderName === 'keyword' ? null : await createEmbedder(embedderName)
console.log('embedder:', embedder?.id ?? 'keyword:off')

// No opts — the adapter reads S3_* from env.
const storage = await createStorage('s3', {})
console.log('storage: s3', {
  bucket: process.env.S3_BUCKET,
  endpoint: process.env.S3_ENDPOINT ?? '(aws)',
  prefix: process.env.S3_PREFIX ?? 'memory-sdk',
})
const memory = new BatchedEngine(storage, { namespace, embedder, dataDir })

// 1. STORE + 2. RETRIEVE
const stored = await memory.remember('User prefers dark mode.', { tags: ['preference'] })
console.log('remembered:', stored)
console.log('recall:', await memory.recall('what are user preferences?'))
console.log('get:', await memory.get(stored.id))
console.log('list:', await memory.list({ limit: 5 }))
console.log('status:', await memory.status())

// 3. DURABILITY — flush pending, then prove rebuild works from S3 blobs.
console.log('flush:', await memory.flush())
console.log('rebuild:', await memory.rebuild())
npx tsx s3.mjs

Why S3?​

S3 (or R2 / MinIO through the same adapter) is the choice when you already have a bucket, need hard-delete semantics, or want CDN-backed reads. Blobs are content-addressed objects under your prefix, forget() is a real delete, and there is no Walrus quota math. This tutorial covers bucket wiring, key layout, and the rebuild proof.

Prerequisites​

  • Offline tutorial completed.
  • The S3 adapter installed: npm install @lighthouse-ai/core @lighthouse-ai/engine-batched @lighthouse-ai/store-s3 @lighthouse-ai/embed-keyword @lighthouse-ai/embed-local dotenv (embed-local powers the default semantic recall).
  • A bucket plus credentials in project-root .env:
S3_BUCKET=your-bucket-name
S3_REGION=us-east-1
# MinIO (docker):
# S3_ENDPOINT=http://localhost:9000
# S3_FORCE_PATH_STYLE=1
# AWS_ACCESS_KEY_ID=minioadmin
# AWS_SECRET_ACCESS_KEY=change-this-password
# R2:
# S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com
# S3_FORCE_PATH_STYLE=1
# Optional:
# S3_PREFIX=memory-sdk
# S3_PUBLIC_BASE_URL=https://cdn.example.com/myprefix
# MEMORY_NAMESPACE=demo-s3
# MEMORY_EMBEDDER=keyword   # word-match only; default is "local" (semantic)

The adapter reads everything from env - createStorage('s3', {}) takes no per-call opts. The demo throws S3_BUCKET is not set… early rather than failing mid-upload.

Step 1 - Understand the key layout​

S3 objects are content-addressed, not sequential (this is what s3.mjs relies on for rebuild):

  • <prefix>/blobs/<cid>.json - the batch blob (same content = same CID, deduplicated naturally).
  • <prefix>/names/<fileName> - per-namespace alias (mem-batch.<namespace>.<batchId>.json) used for listing and rebuild.

Default prefix is memory-sdk (overridable with S3_PREFIX). gatewayUrl() returns S3_PUBLIC_BASE_URL + key when set, else an s3://bucket/prefix/… URI. Set S3_PUBLIC_BASE_URL to your CDN when you want browser-fetchable links.

Step 2 - Store and retrieve (same engine, new backend)​

s3.mjs wires createStorage('s3', {}) (no opts — the adapter reads S3_* from env), then runs the same remember / recall / get / list / status flow. The local index still lives in .memory-sdk/ — S3 holds the blobs, the JSON holds the searchable index. HOW search works (embedder) and WHERE bytes live (storage) stay independent: swap MEMORY_EMBEDDER=keyword for offline word-match without touching the bucket config.

Step 3 - Flush, then prove rebuild works from S3​

s3.mjs flushes the pending batch as one object, then rebuilds from S3 blobs. flush() uploads; rebuild() lists <prefix>/names/mem-batch.<namespace>.* and re-reads them. Because S3 listing is cheap and authoritative, rebuild is typically faster here than on Walrus/Filecoin — but snapshots are still the fast path at scale.

Step 4 - Know the S3 differences​

  • forget() = hard delete (alias always removed; blob removed when unreferenced). No renewal period, no lapse - gone means gone.
  • storage.getBlobIds() returns [] - there are no Walrus-native ids here.
  • MinIO developers: S3_ENDPOINT=http://localhost:9000 + S3_FORCE_PATH_STYLE=1 + the default minioadmin creds give you a local S3 with zero cloud cost. R2 users: same two settings with your account endpoint.

Troubleshooting​

SymptomFix
S3_BUCKET is not setAdd it to project-root .env - your script should read it via dotenv.
AccessDenied / NoSuchBucketRegion/endpoint mismatch (R2 needs path-style + account endpoint; MinIO needs http://localhost:9000 + path-style).
gatewayUrl is s3://…Set S3_PUBLIC_BASE_URL to your CDN so links are HTTPS-fetchable.
Recall misses after embedder switchVectors carry embeddingModel; switching models falls back to keyword until backfilled - expected, not a bug.

Next steps​