📦 Batched Engine

BatchedEngine (@lighthouse-ai/engine-batched) is the DIY engine: you embed locally, memories buffer as pending, and flush as one batch blob per flushEvery memories on any storage.

import '@lighthouse-ai/store-lighthouse'
import '@lighthouse-ai/embed-local'
import { createStorage, createEmbedder } from '@lighthouse-ai/core'
import { BatchedEngine } from '@lighthouse-ai/engine-batched'

const storage = await createStorage('lh-ipfs-filecoin', { apiKey: process.env.LIGHTHOUSE_API_KEY })
const embedder = await createEmbedder('local') // or null for keyword-only
const memory = new BatchedEngine(storage, { namespace: 'demo', agent: 'support-bot', flushEvery: 10, embedder })

Constructor - new BatchedEngine(storage, opts?)

Option	Default	Purpose
namespace	MEMORY_NAMESPACE ?? 'default'	Isolates memories per agent/project; snapshots are namespace-checked
agent	MEMORY_AGENT ?? 'agent'	Recorded on every memory
dataDir	MEMORY_DIR ?? ./.memory-sdk	Index + local-fs blob location
flushEvery	MEMORY_FLUSH_EVERY ?? 10 (min 1)	Batch size before auto-flush
embedder	null (keyword)	Embedder instance or null for word-match only
indexStore	FileJsonIndexStore	Custom index backend (e.g. Dynamo)

flushEvery: 1 uploads on every write (maximum durability). Higher values pack more memories per blob (maximum quota efficiency - critical on Walrus where every blob costs ~63 MB of quota after erasure coding).

remember(content, opts?)

const res = await memory.remember('Customer ACME is on enterprise.', {
  tags: ['customer'],
  metadata: { source: 'support-bot' },
})
// pending: { id: '80cc2055-…', flushed: false, pendingCount: 1 }
// tripped flushEvery: { id, flushed: true, pendingCount: 0, cid: 'baf…', gatewayUrl: 'https://…' }

• Trims content; throws on empty strings.
• Embeds locally if an embedder is set (graceful keyword fallback if the model cannot load - status() reports which mode is active).
• Pending memories are searchable immediately via recall() / list() but live only on the local machine until flushed.
• Returns RememberResult: { id, flushed?, pendingCount?, cid?, gatewayUrl? }.

flush()

const out = await memory.flush()
// { flushedMemories: 3, cid: 'baf…', gatewayUrl: 'https://gateway…/ipfs/baf…' }
// { flushedMemories: 0 } when nothing was pending

Uploads all pending memories as one mem-batch.<namespace>.<batchId>.json blob, assigns the CID to every new index entry, clears the pending queue, and records lastSnapshotCid. Call it manually before ending important sessions, or rely on auto-flush.

recall(query, opts?)

const matches = await memory.recall('how do I escalate for ACME?', {
  tags: ['customer'], // only memories carrying at least one of these tags
  limit: 5,           // default 5
})
// [{ id, content, score, semanticScore, keywordScore, tags, cid, gatewayUrl, ... }]

Search runs fully locally over flushed and pending memories - no network round trips.

Scoring with local embedder (default):

score = 0.7 × cosine(query, memory) + 0.3 × min(1, keywordScore)
keywordScore = hits / √words + 0.5 per tag appearing in the query

• Vectors carry their model id (embeddingModel); recall only compares same-model vectors, so switching models safely falls back to keyword until backfilled.
• Pass embedder: null (or MEMORY_EMBEDDER=keyword) for word-match only - zero deps, fully offline.
• Ties break by recency. Empty queries return everything up to limit.

list(opts?)

await memory.list({ limit: 20 }) // newest first, flushed + pending, score: 0

Returns RecallMatch[] with score: 0, keywordScore: 0 - use it for browsing, not ranking. No tag filter here (filter client-side, or use recall() with tags).

get(idOrCid)

await memory.get('80cc2055-…') // by memory id
await memory.get('baf…')         // by batch CID (fetches the blob, finds the record)

Checks pending first, then the index, then fetches the batch blob from storage. Throws if the id is not found in the referenced batch.

forget(idOrCid)

await memory.forget('80cc2055-…')
// pending: { removed: true, blobDeleted: false, note: 'Removed from pending queue…' }
// flushed: { removed: true, blobDeleted: true|false, note: '…' }

• Pending - dropped locally, never uploaded.
• Flushed - removed from the index. The backing blob is deleted only when no other memory references it (blobDeleted: true). On S3/local-fs that is a hard delete; on Lighthouse deletion stops renewal - content stays readable until the period expires.

status()

await memory.status()
// {
//   engine: 'batched', storage: 'lh-ipfs-filecoin', namespace: 'demo', agent: 'agent',
//   memories: 42, pendingMemories: 3, flushEvery: 10,
//   embeddings: 'local:Xenova/all-MiniLM-L6-v2' | 'keyword:off',
//   indexPath: '.memory-sdk/demo.index.json', lastSnapshotCid: 'baf…'
// }

rebuild()

await memory.rebuild() // { restored: 42, source: 'blobs' }

Re-reads every mem-batch.<namespace>.* blob from storage (sorted by createdAt), deduplicates by id, and replaces the local index. This is the slow path - no pointer needed, only storage credentials - but it lists and fetches every blob, so it slows as history grows. Snapshots are the fast path.

snapshotIndex() / rebuildLocal(snapshotCid)

const snap = await memory.snapshotIndex()
// { cid: 'baf…', entries: 42, gatewayUrl: 'https://…' }

await freshMemory.rebuildLocal(snap.cid) // { added: 42, total: 42 }

Pins the entire local index (entries + pending) as one mem-index.<namespace>.json snapshot blob and merges it back elsewhere. Merges skip entries this machine already has; snapshots are namespace-checked (alice won't load into bob).

warning

Snapshots and batch blobs are plaintext - anyone with the CID can read them. Do not store secrets.

Batch blob shape

{
  "v": 1,
  "kind": "batch",
  "namespace": "default",
  "batchId": "80cc2055-…",
  "createdAt": "2026-07-22T14:59:03.427Z",
  "records": [
    {
      "v": 1, "id": "…", "namespace": "default", "agent": "support-bot",
      "content": "Customer ACME is on enterprise.",
      "tags": ["customer"], "metadata": {},
      "createdAt": "…", "embedding": [0.0123, -0.0456],
      "embeddingModel": "local:Xenova/all-MiniLM-L6-v2",
      "origin": "batched"
    }
  ]
}

Blobs are content-addressed (same content = same CID). CIDs are CIDv1 raw + sha2-256 and must stay stable.