Skip to main content

🗄️ Durable Memory on Filecoin

Copy filecoin.mjs below into your own project and run it with npx tsx filecoin.mjs. It mirrors the runnable demo flow step by step.

filecoin.mjs - the full file​

// filecoin.mjs — durable: batched + lh-ipfs-filecoin + local. Needs LIGHTHOUSE_API_KEY.
import 'dotenv/config'
import '@lighthouse-ai/store-lighthouse'
import '@lighthouse-ai/embed-keyword'
import '@lighthouse-ai/embed-local'
import { mkdirSync, rmSync } from 'node:fs'
import { join } from 'node:path'
import { createEmbedder, createStorage } from '@lighthouse-ai/core'
import { BatchedEngine } from '@lighthouse-ai/engine-batched'
import { backupIndex, restoreIndex } from '@lighthouse-ai/cloud-sync'

const apiKey = process.env.LIGHTHOUSE_API_KEY
if (!apiKey) throw new Error('LIGHTHOUSE_API_KEY is not set. Add LIGHTHOUSE_API_KEY=lh_... to project-root .env.')

const namespace = process.env.MEMORY_NAMESPACE ?? 'demo-filecoin'
const dataDir = process.env.MEMORY_DIR ?? join(process.cwd(), '.memory-sdk')

const embedderName = process.env.MEMORY_EMBEDDER ?? 'local'
const embedder = embedderName === 'keyword' ? null : await createEmbedder(embedderName)
console.log('embedder:', embedder?.id ?? 'keyword:off')

const storage = await createStorage('lh-ipfs-filecoin', { apiKey })
const flushEvery = Number(process.env.MEMORY_FLUSH_EVERY) || 5
const memory = new BatchedEngine(storage, { namespace, embedder, dataDir, flushEvery })

// 1. REMEMBER — first four stay pending, 5th trips flushEvery and uploads one blob
const memoriesToStore = [
  { content: 'User prefers dark mode.', tags: ['preference'] },
  { content: 'User works in IST timezone, starts day around 9am.', tags: ['preference'] },
  { content: 'Project deadline is end of September, Filecoin integration first.', tags: ['project'] },
  { content: 'User likes concise code reviews with examples.', tags: ['preference'] },
  { content: 'Staging API runs at https://staging.lighthouse.storage', tags: ['infra'] },
]
const storedIds = []
for (const [i, m] of memoriesToStore.entries()) {
  const res = await memory.remember(m.content, { tags: m.tags })
  console.log(`remembered ${i + 1}/${memoriesToStore.length}:`, res)
  storedIds.push(res.id)
}

// 2. FLUSH — manual only if auto-flush did not fire
const statusBeforeFlush = await memory.status()
if (statusBeforeFlush.pendingMemories > 0) {
  console.log('flushed (manual):', await memory.flush())
} else {
  console.log('auto-flushed on 5th remember — no manual flush needed.')
}

// 3. RETRIEVE
console.log('recall:', await memory.recall('what theme does the user like?'))
console.log('get:', await memory.get(storedIds[0]))
console.log('list:', await memory.list())
console.log('status:', await memory.status())

// 4. FRESH MACHINE — same namespace, empty dir, rebuild from blobs (no pointer needed)
const freshDir = join(dataDir, 'fresh-filecoin')
rmSync(freshDir, { recursive: true, force: true })
mkdirSync(freshDir, { recursive: true })
const fresh = new BatchedEngine(await createStorage('lh-ipfs-filecoin', { apiKey }), {
  namespace, dataDir: freshDir, embedder,
})
console.log('rebuild on fresh dir:', await fresh.rebuild())
console.log('recall on fresh dir:', await fresh.recall('what theme does the user like?'))

// 5-6. CLOUD BACKUP (optional — needs CM_API_URL + `memory-login` once)
if (!process.env.CM_API_URL) {
  console.log('SKIP cloud backup: CM_API_URL is not set.')
} else {
  const syncOpts = { apiUrl: process.env.CM_API_URL, namespace, network: 'ipfs-filecoin', engine: 'batched' }
  try {
    console.log('cloud backup:', (await backupIndex(memory, syncOpts)).pointer)
    const viaCloudDir = join(dataDir, 'fresh-filecoin-cloud')
    rmSync(viaCloudDir, { recursive: true, force: true })
    mkdirSync(viaCloudDir, { recursive: true })
    const viaCloud = new BatchedEngine(await createStorage('lh-ipfs-filecoin', { apiKey }), {
      namespace, dataDir: viaCloudDir, embedder,
    })
    console.log('restore via pointer:', (await restoreIndex(viaCloud, syncOpts)).result)
  } catch (err) {
    if (err instanceof Error && err.message.startsWith('Missing Clerk session JWT')) {
      console.log('SKIP cloud backup: not logged in. Run cloud-sync login once, then re-run.')
    } else throw err
  }
}
npx tsx filecoin.mjs

Why Filecoin?​

Filecoin (via Lighthouse IPFS) is the default durable storage: quota tracks real bytes, any Lighthouse API key works, and every batch blob is fetchable from gateway.lighthouse.storage. This tutorial is the full batched lifecycle - buffer, flush, retrieve, rebuild on a fresh machine, and optional cloud backup - so treat it as the reference for all durable storages.

Prerequisites​

  • Offline tutorial completed (you know remember / recall / status).
  • The durable packages installed: npm install @lighthouse-ai/core @lighthouse-ai/engine-batched @lighthouse-ai/store-lighthouse @lighthouse-ai/embed-local dotenv (plus @lighthouse-ai/cloud-sync for steps 5-6).
  • A Lighthouse API key from the Files App (API Keys section). Put it in a project-root .env (git-ignored):
LIGHTHOUSE_API_KEY=lh_...
  • Optional for steps 5–6: CM_API_URL (pointer service) + npx -y @lighthouse-ai/cloud-sync login once to save the Clerk JWT (see Cross-Device Backup).

Step 1 - Configure the engine for durability​

filecoin.mjs sets namespace = MEMORY_NAMESPACE ?? 'demo-filecoin', dataDir = MEMORY_DIR ?? <project>/.memory-sdk, and embedder = MEMORY_EMBEDDER ?? 'local' (keyword maps to null). Storage is fixed to lh-ipfs-filecoin with flushEvery: 5 (overridable via MEMORY_FLUSH_EVERY).

flushEvery: 5 (overridable via MEMORY_FLUSH_EVERY) means the 5th remember auto-flushes. On Filecoin a lower value is affordable (real-bytes quota); on Walrus you would keep it high (63 MB per blob regardless of size). This contrast is the point of the next tutorial.

Step 2 - Buffer five memories, watch auto-flush​

The demo stores five tagged memories (preferences, project deadline, infra URL). Each remember returns { id, flushed, pendingCount, cid?, gatewayUrl? } - the first four are pending, the fifth trips flushEvery and uploads one batch blob.

Why five heterogeneous memories? It gives recall something to discriminate: a theme query should rank the dark-mode memory first, not the deadline or infra URL. Tags (preference, project, infra) let you pre-filter before ranking.

Log line to watch: remembered 5/5: { id, flushed: true, … cid: 'baf…' } - that CID is the batch blob holding all five.

Step 3 - Flush explicitly and read the status​

filecoin.mjs checks status().pendingMemories and flushes only when needed:

status() tells you whether anything is still pending. flush() uploads leftovers as one blob and returns { flushedMemories, cid, gatewayUrl }. The demo handles both cases: if auto-flush already fired, it reports auto-flushed on 5th remember; otherwise it flushes manually.

Habit to adopt: flush before ending important sessions, or set MEMORY_FLUSH_EVERY=1 for maximum durability. Unflushed memories die with the local index.

Step 4 - Retrieve three ways​

filecoin.mjs then retrieves three ways — semantic recall() (deliberately zero word overlap with User prefers dark mode.), exact get(storedIds[0]), newest-first list(), plus status() for counts + lastSnapshotCid:

The recall query deliberately shares no keywords with User prefers dark mode. - with the local embedder it still ranks first via cosine similarity. Switch to MEMORY_EMBEDDER=keyword and re-run to feel the difference: semantic recall degrades to word-match.

gatewayUrl on any result is directly curlable: curl https://gateway.lighthouse.storage/ipfs/<CID>.

Step 5 - Prove durability on a fresh machine​

filecoin.mjs wipes <dataDir>/fresh-filecoin, constructs a new BatchedEngine with the same namespace, and calls rebuild() ({ restored: 5, source: 'blobs' }), then recalls on the fresh engine:

rebuild() lists every mem-batch.<namespace>.* blob and re-reads them - no pointer needed, only the API key. This is the slow path (every blob is fetched), which is why snapshots exist.

Step 6 - Cloud backup via pointer (optional)​

With CM_API_URL set and a saved JWT, filecoin.mjs backs up and restores through the pointer service (backupIndex snapshots → one CID and upserts namespace × network → CID; restoreIndex fetches that CID and merges it):

A third engine on yet another fresh dir (fresh-filecoin-cloud) proves the pointer alone is enough — no CID copied by hand. Skipped with SKIP cloud backup: CM_API_URL is not set or the Missing Clerk session JWT path when prerequisites are missing — the durable steps above still pass.

Troubleshooting​

SymptomFix
LIGHTHOUSE_API_KEY is not setAdd it to project-root .env, not the shell profile, so your scripts share it.
flushedMemories: 0 surprisesAuto-flush already fired - check status().pendingMemories before calling flush().
rebuild() returns 0Wrong namespace or wrong network (ipfs-filecoin vs lh-ipfs-walrus never mix).
Cloud 404 on restoreNo backup yet for that namespace × network - run backup first.

Next steps​