♻️ Rebuild & Recovery

The local search index is a cache - the source of truth is on the network. There are three recovery paths: re-read everything, restore one snapshot CID, or track that CID in the pointer service.

Path 1 - Rebuild from storage (no pointer needed)

Batched re-reads every batch blob for the namespace; memwal asks the relayer to repair its vector index:

// batched: re-read every mem-batch.<namespace>.* blob
const result = await memory.rebuild()
// { restored: 42, source: 'blobs' }

// memwal: rebuild missing relayer vector-index entries from on-chain data
await memory.restore({ maxBlobs: 200 })

This needs only your storage credentials (LIGHTHOUSE_API_KEY / S3 creds / MEMWAL_*) - but it lists and fetches every blob, so it gets slower as history grows.

warning

rebuild() discards pending memories. It replaces the local index with only what is in the batch blobs and sets pending: [] - anything you remembered since the last flush is lost. Call flush() first if the index still has pending memories (status().pendingMemories > 0).

Path 2 - Snapshot + restore (fast path, one CID)

Both engines can pin the entire local index as one snapshot blob and merge it back on another machine:

// either engine
const snap = await memory.snapshotIndex()
// batched: { cid, entries, gatewayUrl }
// memwal:  { cid, pinned, entries, gatewayUrl }

// fresh machine, same namespace (+ same memwal network for memwal)
await memory.rebuildLocal(snap.cid)
// { added, total }

Snapshots merge - entries this machine already has are skipped - and are namespace-checked (a snapshot for alice won't load into bob).

warning

Snapshots are plaintext (like batch blobs and pinned memwal mirrors) - the service and gateways only ever see the CID, never anything else, but anyone with the CID can read it.

Path 3 - Pointer service (cross-device)

memory-backend maps each user + namespace × network to the latest index-snapshot CID, so a fresh device can rebuild without copying CIDs. The zero-dep client is @lighthouse-ai/cloud-sync (npm install @lighthouse-ai/cloud-sync).

Sign in once before backup:

export CM_API_URL=https://memory-api.lighthouse.storage
export CM_DASHBOARD_URL=https://memory.lighthouse.storage
npx -y @lighthouse-ai/cloud-sync login   # `memory-login` — browser dashboard handoff, saves JWT to ~/.memory-sdk-cloud/token (0600)

Auth stays lazy - everything before backup needs no token. Precedence per call: explicit token → saved token file (~/.memory-sdk-cloud/token, override with tokenFile) → error (Missing Clerk session JWT — run `npx -y @lighthouse-ai/cloud-sync login` once, or pass `token` explicitly.).

Raw API (both engines speak the same pair - snapshotIndex() / rebuildLocal() - so this is identical for batched and memwal):

// backup (after remember/flush or cron)
const snap = await memory.snapshotIndex() // { cid, entries }
await fetch(`${API}/v1/pointers`, {
  method: 'PUT',
  headers: { 'content-type': 'application/json', authorization: `Bearer ${token}` },
  body: JSON.stringify({ snapshotCid: snap.cid, engine: 'batched', namespace: 'default', network: 'mainnet' }),
})

// restore (fresh device, no local index)
const { pointer } = await fetch(
  `${API}/v1/pointers/current?namespace=default&network=mainnet`,
  { headers: { authorization: `Bearer ${token}` } }
).then((r) => r.json())
await memory.rebuildLocal(pointer.snapshotCid)

Method	Path	Auth	Body / Query
GET	/v1/auth/me	yes	- → { user: { id: "clerk:<sub>", email? } }
PUT	/v1/pointers	yes	{ snapshotCid, engine?, namespace?, network?, memories? } (upserts latest)
GET	/v1/pointers?namespace=&network=	yes	list (filtered)
GET	/v1/pointers/current?namespace=default&network=mainnet	yes	single pointer for rebuild

MVP stores the latest CID per namespace × network (PUT overwrites). Pointer key is <namespace>#<network> (e.g. default#mainnet). Auth is Authorization: Bearer <Session JWT>; defaults are engine: 'memwal' / namespace: 'default' / network: 'mainnet', and memories is an optional int.

Durability model

Failure	Recovery
Session ends	Batched: flushed memories are on storage; pending are in the local index - nothing lost. Memwal: everything is already on the relayer
Local machine lost before flush (batched)	Pending (unflushed) memories are gone - flush before ending important sessions, or set MEMORY_FLUSH_EVERY=1
Local index deleted	rebuild() (batched, from blobs) / restore() (memwal, relayer) - or instant rebuildLocal(snapshotCid)
Snapshots unreadable	Batched falls back to rebuild() from blobs
New machine	Storage creds + rebuild() - or storage creds + pointer-service token + restoreIndex() (no CID to copy)

Forgetting memories

await memory.forget(id)
// batched: { removed, blobDeleted?, note }
// memwal:  { removed, blobDeleted, note }

• Batched, pending - simply dropped locally, never uploaded.
• Batched, flushed - removed from the index. Because batched memories share one blob, the backing file is deleted only when no other memory references it (blobDeleted: true). On S3/local-fs that's a hard delete; on Lighthouse deletion stops renewal - content stays readable until the period expires, then is reclaimed.
• Memwal - local copy removed and IPFS mirror unpinned (blobDeleted). The relayer has no delete API: the SEAL-encrypted blob stays on Walrus until its storage period lapses and may still surface in recall as indexed: false. Pinned mirrors are plaintext; memwal blobs are encrypted.

warning

Forgetting a flushed batched memory does not erase it. forget() only edits the local index - it records no tombstone. Until its whole batch blob is deleted:

• The record stays in the blob. If other memories share the batch, the blob is kept and the record's content stays readable to anyone with the CID (batch blobs are plaintext).
• It comes back on rebuild(). Rebuild re-reads every batch blob, so a forgotten record in a kept blob is restored to the index.
• It stays in earlier snapshots. Any snapshotIndex() CID taken before the forget still contains it, and rebuildLocal() / restoreIndex() from that snapshot restores it. Take a fresh snapshot (and back it up) after forgetting.

Even when the blob is deleted, Lighthouse storage only stops renewal, and copies already fetched through public IPFS gateways may persist. Don't store anything you may need to truly erase - see the encryption warning in the Introduction.