♻️ Rebuild & Recovery

The local search index is a cache - the source of truth is on the network. There are three recovery paths: re-read everything, restore one snapshot CID, or track that CID in the pointer service.

Path 1 - Rebuild from storage (no pointer needed)

Batched re-reads every batch blob for the namespace; memwal asks the relayer to repair its vector index:

// batched: re-read every mem-batch.<namespace>.* blob
const result = await memory.rebuild()
// { restored: 42, source: 'blobs' }

// memwal: rebuild missing relayer vector-index entries from on-chain data
await memory.restore({ maxBlobs: 200 })

This needs only your storage credentials (LIGHTHOUSE_API_KEY / S3 creds / MEMWAL_*) - but it lists and fetches every blob, so it gets slower as history grows.

Path 2 - Snapshot + restore (fast path, one CID)

Both engines can pin the entire local index as one snapshot blob and merge it back on another machine:

// either engine
const snap = await memory.snapshotIndex()
// batched: { cid, entries, gatewayUrl }
// memwal:  { cid, pinned, entries, gatewayUrl }

// fresh machine, same namespace (+ same memwal network for memwal)
await memory.rebuildLocal(snap.cid)
// { added, total }

Snapshots merge - entries this machine already has are skipped - and are namespace-checked (a snapshot for alice won't load into bob).

warning

Snapshots are plaintext (like batch blobs and pinned memwal mirrors) - the service and gateways only ever see the CID, never anything else, but anyone with the CID can read it.

Path 3 - Pointer service (cross-device)

memory-backend maps each user + namespace × network to the latest index-snapshot CID, so a fresh device can rebuild without copying CIDs. The zero-dep client is @lighthouse-ai/cloud-sync (npm install @lighthouse-ai/cloud-sync).

Sign in once before backup:

CM_API_URL=https://memory-api.lighthouse.storage
npx -y @lighthouse-ai/cloud-sync login   # `memory-login` — browser dashboard handoff, saves JWT to ~/.memory-sdk-cloud/token (0600)

Auth stays lazy - everything before backup needs no token. Precedence per call: explicit token → saved token file (~/.memory-sdk-cloud/token, override with tokenFile) → error (Missing session JWT — run the cloud-sync login once.).

Raw API (both engines speak the same pair - snapshotIndex() / rebuildLocal() - so this is identical for batched and memwal):

// backup (after remember/flush or cron)
const snap = await memory.snapshotIndex() // { cid, entries }
await fetch(`${API}/v1/pointers`, {
  method: 'PUT',
  headers: { 'content-type': 'application/json', authorization: `Bearer ${token}` },
  body: JSON.stringify({ snapshotCid: snap.cid, engine: 'batched', namespace: 'default', network: 'mainnet' }),
})

// restore (fresh device, no local index)
const { pointer } = await fetch(
  `${API}/v1/pointers/current?namespace=default&network=mainnet`,
  { headers: { authorization: `Bearer ${token}` } }
).then((r) => r.json())
await memory.rebuildLocal(pointer.snapshotCid)

Method	Path	Auth	Body / Query
GET	/v1/auth/me	yes	- → { user: { id: "clerk:<sub>", email? } }
PUT	/v1/pointers	yes	{ snapshotCid, engine?, namespace?, network?, memories? } (upserts latest)
GET	/v1/pointers?namespace=&network=	yes	list (filtered)
GET	/v1/pointers/current?namespace=default&network=mainnet	yes	single pointer for rebuild

MVP stores the latest CID per namespace × network (PUT overwrites). Pointer key is <namespace>#<network> (e.g. default#mainnet). Auth is Authorization: Bearer <Session JWT>; defaults are engine: 'memwal' / namespace: 'default' / network: 'mainnet', and memories is an optional int.

Durability model

Failure	Recovery
Session ends	Batched: flushed memories are on storage; pending are in the local index - nothing lost. Memwal: everything is already on the relayer
Local machine lost before flush (batched)	Pending (unflushed) memories are gone - flush before ending important sessions, or set MEMORY_FLUSH_EVERY=1
Local index deleted	rebuild() (batched, from blobs) / restore() (memwal, relayer) - or instant rebuildLocal(snapshotCid)
Snapshots unreadable	Batched falls back to rebuild() from blobs
New machine	Storage creds + rebuild() - or storage creds + pointer-service token + restoreIndex() (no CID to copy)

Forgetting memories

await memory.forget(id)
// batched: { removed, blobDeleted?, note }
// memwal:  { removed, blobDeleted, note }

• Batched, pending - simply dropped locally, never uploaded.
• Batched, flushed - removed from the index. Because batched memories share one blob, the backing file is deleted only when no other memory references it (blobDeleted: true). On S3/local-fs that's a hard delete; on Lighthouse deletion stops renewal - content stays readable until the period expires, then is reclaimed.
• Memwal - local copy removed and IPFS mirror unpinned (blobDeleted). The relayer has no delete API: the SEAL-encrypted blob stays on Walrus until its storage period lapses and may still surface in recall as indexed: false. Pinned mirrors are plaintext; memwal blobs are encrypted.