Skip to main content

🐋 Batched Memory on Walrus

Copy walrus.mjs below into your own project and run it with npx tsx walrus.mjs.

walrus.mjs - the full file​

// walrus.mjs — durable: batched + lh-ipfs-walrus + local. Needs a Sui-wallet LIGHTHOUSE_API_KEY.
import 'dotenv/config'
import '@lighthouse-ai/store-lighthouse'
import '@lighthouse-ai/embed-keyword'
import '@lighthouse-ai/embed-local'
import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createEmbedder, createStorage } from '@lighthouse-ai/core'
import { BatchedEngine } from '@lighthouse-ai/engine-batched'

const apiKey = process.env.LIGHTHOUSE_API_KEY
if (!apiKey) throw new Error('LIGHTHOUSE_API_KEY is not set. Add LIGHTHOUSE_API_KEY=lh_... to project-root .env (Sui-wallet key for Walrus).')

const namespace = process.env.MEMORY_NAMESPACE ?? 'demo-walrus'
const dataDir = process.env.MEMORY_DIR ?? join(process.cwd(), '.memory-sdk')

const embedderName = process.env.MEMORY_EMBEDDER ?? 'local'
const embedder = embedderName === 'keyword' ? null : await createEmbedder(embedderName)
console.log('embedder:', embedder?.id ?? 'keyword:off')

const storage = await createStorage('lh-ipfs-walrus', { apiKey }) // 'walrus' alias also works
const memory = new BatchedEngine(storage, { namespace, embedder, dataDir })

// 1. STORE
const stored = await memory.remember('User prefers dark mode.', { tags: ['preference'] })
console.log('remembered:', stored)

// 2. FLUSH — required! Without this it stays pending and never hits Walrus.
// Every blob costs ~63 MB of quota after erasure-coding, so batching matters.
const flushed = await memory.flush()
console.log('flushed:', flushed)
console.log('gateway:', flushed.gatewayUrl)

// Walrus-only: native blob IDs for Sui ecosystem tooling.
if (flushed.cid) console.log('blobIds:', await storage.getBlobIds(flushed.cid))

// 3. RETRIEVE — semantic query has zero word overlap on purpose.
console.log('recall:', await memory.recall('what theme does the user like?'))
console.log('get:', await memory.get(stored.id))
console.log('list:', await memory.list())
console.log('status:', await memory.status())

// 4. FRESH MACHINE — any empty dir works, proves the Walrus copy is enough.
const freshDir = mkdtempSync(join(tmpdir(), 'cmem-walrus-fresh-'))
const fresh = new BatchedEngine(await createStorage('lh-ipfs-walrus', { apiKey }), {
  namespace, dataDir: freshDir, embedder,
})
console.log('rebuild on fresh dir:', await fresh.rebuild())
console.log('recall on fresh dir:', await fresh.recall('what theme does the user like?'))
npx tsx walrus.mjs
# First run with local downloads the MiniLM model once (~25 MB), then works offline.

Why Walrus?​

Walrus (via Lighthouse on Sui) gives you Walrus-native blob IDs for on-chain references and Sui ecosystem tooling. The trade-off is quota: every blob is erasure-coded and counts ~63 MB against your account regardless of real size. This tutorial teaches quota-aware batching - the one habit that makes Walrus affordable - plus blob-ID retrieval and gateway reads.

It mirrors the Filecoin demo deliberately: only the network differs, so compare the two outputs side by side.

Prerequisites​

  • Filecoin tutorial completed (batching, flush, rebuild are assumed).
  • The durable packages installed: npm install @lighthouse-ai/core @lighthouse-ai/engine-batched @lighthouse-ai/store-lighthouse @lighthouse-ai/embed-local dotenv.
  • A Sui-wallet Lighthouse API key (Walrus uploads require it - a plain EVM key fails here). Project-root .env:
LIGHTHOUSE_API_KEY=lh_...
# optional:
MEMORY_NAMESPACE=demo-walrus
MEMORY_EMBEDDER=local   # semantic (default); use "keyword" for word-match only

Step 1 - Point the same engine at Walrus​

walrus.mjs fixes storage to lh-ipfs-walrus — nothing else changes versus Filecoin. That interchangeability is the point of the storage abstraction.

The local index still lives in .memory-sdk/ (overridable with MEMORY_DIR); Walrus holds the blobs, the JSON holds the searchable index (text + tags + vectors).

Step 2 - Store, then flush on purpose​

walrus.mjs stores one memory and flushes manually to make the durability moment explicit (unlike the Filecoin file, which stores five and leans on auto-flush). A single-memory blob still costs the full ~63 MB of quota — which is exactly why production code sets a high flushEvery and lets memories accumulate before uploading.

Rule of thumb: flushEvery: 1 on Walrus is maximum durability at maximum cost; the default 10 (or higher) packs ten memories into one 63 MB charge.

Step 3 - Read Walrus-native blob IDs​

walrus.mjs calls storage.getBlobIds(flushed.cid) when a CID exists. Only lh-ipfs-walrus returns ids here (filecoin / s3 / local-fs return []). These ids are what Sui contracts and Walrus tooling reference — the CID is the IPFS view, the blob IDs are the Walrus view of the same bytes.

Step 4 - Retrieve semantically, then from scratch​

walrus.mjs retrieves (recall with zero word overlap, get, list, status), then proves durability in a real temp dir (mkdtempSync) rather than a repo subfolder — to emphasize that any empty dir works. First run with local downloads the MiniLM model once (~25 MB), then works offline. Every flushed blob is also curlable: curl https://gateway-walrus.lighthouse.storage/ipfs/<CID>.

Troubleshooting​

SymptomFix
Upload rejected with plain keyWalrus needs a Sui-wallet Lighthouse key - sign in with Sui in the Files App and mint a fresh key.
Quota evaporatesYou are flushing single-memory blobs. Raise MEMORY_FLUSH_EVERY and flush manually at session end instead.
getBlobIds() returns []You are on filecoin/s3/local-fs, or you passed the batch id instead of the CID.
rebuild() is slowExpected - it fetches every batch blob. Use snapshotIndex() → rebuildLocal(cid) for the fast path (see Cross-Device tutorial).

Next steps​