---
slug: "memory-lancedb-pro"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/win4r/memory-lancedb-pro@main/README.md"
repo: "https://github.com/win4r/memory-lancedb-pro"
source_file: "README.md"
branch: "main"
---
# ๐ง memory-lancedb-pro ยท OpenClaw Plugin
**Enhanced Long-Term Memory Plugin for [OpenClaw](https://github.com/openclaw/openclaw)**
Hybrid Retrieval (Vector + BM25) ยท Cross-Encoder Rerank ยท Multi-Scope Isolation ยท Management CLI
[](https://github.com/openclaw/openclaw)
[](https://lancedb.com)
[](LICENSE)
**English** | [็ฎไฝไธญๆ](https://github.com/win4r/memory-lancedb-pro/blob/HEAD/README_CN.md)
---
## ๐บ Video Tutorial
> **Watch the full walkthrough - covers installation, configuration, and how hybrid retrieval works under the hood.**
[](https://youtu.be/MtukF1C8epQ)
๐ **https://youtu.be/MtukF1C8epQ**
[](https://www.bilibili.com/video/BV1zUf2BGEgn/)
๐ **https://www.bilibili.com/video/BV1zUf2BGEgn/**
---
## Why This Plugin?
The built-in `memory-lancedb` plugin in OpenClaw provides basic vector search. **memory-lancedb-pro** takes it much further:
| Feature | Built-in `memory-lancedb` | **memory-lancedb-pro** |
|---------|--------------------------|----------------------|
| Vector search | โ
| โ
|
| BM25 full-text search | โ | โ
|
| Hybrid fusion (Vector + BM25) | โ | โ
|
| Cross-encoder rerank (Jina / custom endpoint) | โ | โ
|
| Recency boost | โ | โ
|
| Time decay | โ | โ
|
| Length normalization | โ | โ
|
| MMR diversity | โ | โ
|
| Multi-scope isolation | โ | โ
|
| Noise filtering | โ | โ
|
| Adaptive retrieval | โ | โ
|
| Management CLI | โ | โ
|
| Session memory | โ | โ
|
| Task-aware embeddings | โ | โ
|
| Any OpenAI-compatible embedding | Limited | โ
(OpenAI, Gemini, Jina, Ollama, etc.) |
---
## Architecture
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ index.ts (Entry Point) โ
โ Plugin registration ยท config parsing ยท lifecycle orchestration โ
โโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ generic auto-recall โ reflection / inherited-rules
โ โ
โโโโโโโโโโผโโโโโโโโโ โโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ retriever.ts โ โ reflection-recall.ts โ
โ vector/BM25/RRF โ โ dynamic reflection recall ranking โ
โ rerank/filters โ โโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโฌโโโโโโโโโ โ
โ โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโผโโโโโโโโโโโโโโโโ โ reflection-aggregation.ts โ
โ postProcessAutoRecall โ โ strictKey-group scoring โ
โ (index.ts local step) โ โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโฌโโโโโโโโโโโโโโโโ โ
โ โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโ
โ mmr | setwise-v2 โ reflection-recall-final- โ
โโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโ โ selection.ts โ
โ auto-recall-final- โ โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโ
โ selection.ts โ โ
โโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโบ final-topk-setwise- โ
โ selection.ts โ
โ shared final top-k selector โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Shared infrastructure: `store.ts`, `embedder.ts`, `scopes.ts`, `tools.ts`,
`noise-filter.ts`, `adaptive-retrieval.ts`, `recall-engine.ts`, `migrate.ts`, `cli.ts`
```
### File Reference
| File | Purpose |
|------|---------|
| `index.ts` | Plugin entry point. Registers with OpenClaw Plugin API, parses config, mounts lifecycle hooks (`before_agent_start` / `before_prompt_build` / `agent_end`), routes generic auto-recall through `mmr | setwise-v2`, and coordinates reflection injection flows |
| `openclaw.plugin.json` | Plugin metadata + full JSON Schema config declaration (with `uiHints`) |
| `package.json` | NPM package info. Depends on `@lancedb/lancedb`, `openai`, `@sinclair/typebox` |
| `cli.ts` | CLI commands: `memory list/search/stats/delete/delete-bulk/export/import/reembed/migrate` |
| `src/store.ts` | LanceDB storage layer. Table creation / FTS indexing / Vector search / BM25 search / CRUD / bulk delete / stats |
| `src/embedder.ts` | Embedding abstraction. Compatible with any OpenAI-API provider (OpenAI, Gemini, Jina, Ollama, etc.). Supports task-aware embedding (`taskQuery`/`taskPassage`) |
| `src/retriever.ts` | Hybrid retrieval engine. Vector + BM25 โ RRF fusion โ rerank โ recency / importance / length / decay weighting โ noise filter โ coarse MMR diversity. |
| `src/recall-engine.ts` | Shared recall helpers: prompt gating, session repeated-injection suppression, tagged-block assembly, max-age filtering, and recent-per-key capping |
| `src/auto-recall-final-selection.ts` | Generic auto-recall adapter. Maps `RetrievalResult` rows into final-selection candidates and applies generic `mmr | setwise-v2` behavior at the final cutoff seam |
| `src/final-topk-setwise-selection.ts` | Shared final top-k selector. Owns shortlist presort, deterministic set-wise selection, lexical-overlap suppression, and optional embedding-based semantic redundancy suppression |
| `src/reflection-recall.ts` | Dynamic Reflection-Recall ranking for ``. Filters/caps reflection items, computes scores, preserves `kind + strictKey` partitioning, and maps selected groups back to recall rows |
| `src/reflection-aggregation.ts` | Reflection group aggregation. Combines scored reflection items into strict-key groups with representative selection and final group scoring |
| `src/reflection-recall-final-selection.ts` | Reflection-specific adapter into the shared final selector for dynamic Reflection-Recall final top-k ordering |
| `src/reflection-selection.ts` | Historical derived-focus diversity ordering helper retained for reflection-store / handoff-note selection flows |
| `src/scopes.ts` | Multi-scope access control. Supports `global`, `agent:`, `custom:`, `project:`, `user:` |
| `src/tools.ts` | Agent tool definitions: `memory_recall`, `memory_store`, `memory_forget` (core), `self_improvement_log` (default), and governance tools `self_improvement_review` / `self_improvement_extract_skill` (management mode) |
| `src/noise-filter.ts` | Noise filter. Filters out agent refusals, meta-questions, greetings, and low-quality content |
| `src/adaptive-retrieval.ts` | Adaptive retrieval. Determines whether a query needs memory retrieval (skips greetings, slash commands, simple confirmations, emoji) |
| `src/migrate.ts` | Migration tool. Migrates data from the built-in `memory-lancedb` plugin to Pro |
---
## Core Features
### 1. Hybrid Retrieval
```
Query โ embedQuery() โโ
โโโ RRF Fusion โ Rerank โ Recency Boost โ Importance Weight โ Filter
Query โ BM25 FTS โโโโโโ
```
- **Vector Search**: Semantic similarity via LanceDB ANN (cosine distance)
- **BM25 Full-Text Search**: Exact keyword matching via LanceDB FTS index
- **Fusion Strategy**: Vector score as base, BM25 hits get a 15% boost (tuned beyond traditional RRF)
- **Configurable Weights**: `vectorWeight`, `bm25Weight`, `minScore`
### 2. Cross-Encoder Reranking
- **Reranker API**: Jina, SiliconFlow, Pinecone, or any compatible endpoint (5s timeout protection)
- **Hybrid Scoring**: 60% cross-encoder score + 40% original fused score
- **Graceful Degradation**: Falls back to cosine similarity reranking on API failure
### 3. Multi-Stage Scoring Pipeline
| Stage | Formula | Effect |
|-------|---------|--------|
| **Recency Boost** | `exp(-ageDays / halfLife) * weight` | Newer memories score higher (default: 14-day half-life, 0.10 weight) |
| **Importance Weight** | `score *= (0.7 + 0.3 * importance)` | importance=1.0 โ ร1.0, importance=0.5 โ ร0.85 |
| **Length Normalization** | `score *= 1 / (1 + 0.5 * log2(len/anchor))` | Prevents long entries from dominating (anchor: 500 chars) |
| **Time Decay** | `score *= 0.5 + 0.5 * exp(-ageDays / halfLife)` | Old entries gradually lose weight, floor at 0.5ร (60-day half-life) |
| **Hard Min Score** | Discard if `score < threshold` | Removes irrelevant results (default: 0.35) |
| **MMR Diversity** | Cosine similarity > 0.85 โ demoted | Prevents near-duplicate results |
### 4. Multi-Scope Isolation
- **Built-in Scopes**: `global`, `agent:`, `custom:`, `project:`, `user:`
- **Agent-Level Access Control**: Configure per-agent scope access via `scopes.agentAccess`
- **Default Behavior**: Each agent accesses `global` + its own `agent:` scope
### 5. Adaptive Retrieval
- Skips queries that don't need memory (greetings, slash commands, simple confirmations, emoji)
- Forces retrieval for memory-related keywords ("remember", "previously", "last time", etc.)
- CJK-aware thresholds (Chinese: 6 chars vs English: 15 chars)
### 6. Noise Filtering
Filters out low-quality content at both auto-capture and tool-store stages:
- Agent refusal responses ("I don't have any information")
- Meta-questions ("do you remember")
- Greetings ("hi", "hello", "HEARTBEAT")
### 7. Session Strategy
Use this to choose who owns `/new` / `/reset` session behavior.
- `sessionStrategy: "systemSessionMemory"` (default)
- use OpenClaw built-in `session-memory`
- plugin reflection hooks stay off
- `sessionStrategy: "memoryReflection"`
- enable plugin reflection flow
- `memoryReflection.*` settings take effect only in this mode
- `sessionStrategy: "none"`
- disable plugin session strategy hooks entirely
Legacy compatibility:
- `sessionMemory.enabled=true|false` still maps to `systemSessionMemory|none`
- `sessionMemory.messageCount` still maps to `memoryReflection.messageCount`
Recommended starter config:
```json
{
"sessionStrategy": "memoryReflection"
}
```
### 8. Self-Improvement
Use this when you want the plugin to keep a lightweight governance trail of learnings and errors.
- Main tools:
- `self_improvement_log`: append a structured learning/error entry
- `self_improvement_review`: summarize pending governance backlog
- `self_improvement_extract_skill`: scaffold a reusable skill from a proven learning entry
- Main config:
- `selfImprovement.enabled`: master switch
- `selfImprovement.beforeResetNote`: show a reminder before `/new` or `/reset`
- `selfImprovement.ensureLearningFiles`: auto-create `.learnings/` files
- `selfImprovement.managementTools`: expose review/extract tools
- Main outputs:
- `.learnings/LEARNINGS.md`
- `.learnings/ERRORS.md`
- extracted skill scaffold under `.learnings/skills/...`
Recommended starter config:
```json
{
"selfImprovement": {
"enabled": true,
"beforeResetNote": true,
"ensureLearningFiles": true,
"managementTools": true
}
}
```
### 9. memoryReflection
Use this when you want reflection-based rule inheritance and optional reflection persistence.
What to configure first:
- `memoryReflection.enabled`: turn reflection features on/off
- `memoryReflection.injectMode`:
- `inheritance-only` = inject inherited rules only
- `inheritance+derived` = inherited rules + derived-focus note on `/new` / `/reset`
- `memoryReflection.recall.mode`:
- `fixed` = compatibility path
- `dynamic` = prompt-gated `` recall
- `memoryReflection.storeToLanceDB`: store reflection event/item rows in LanceDB
- `memoryReflection.agentId` (optional): use a dedicated reflection agent
Recommended starter config:
```json
{
"memoryReflection": {
"enabled": true,
"injectMode": "inheritance-only",
"storeToLanceDB": true,
"recall": {
"mode": "dynamic",
"topK": 6,
"includeKinds": ["invariant", "derived"],
"maxAgeDays": 14,
"maxEntriesPerKey": 7,
"minRepeated": 3,
"minScore": 0.22,
"minPromptLength": 12
}
}
}
```
Quick behavior guide:
- `before_prompt_build` can inject ``
- `/new` / `/reset` can build the reflection note
- `before_prompt_build` can also inject `` reminders
- dynamic recall keeps `kind + strictKey` separation for invariant/derived rows
### 10. Markdown Mirror (`mdMirror`)
Use this when you want a readable Markdown copy of memories in addition to LanceDB.
Main config:
- `mdMirror.enabled`: turn Markdown dual-write on/off
- `mdMirror.dir`: fallback output directory when the agent workspace path is unavailable
What it does:
- writes memory entries to readable Markdown files
- prefers `memory/YYYY-MM-DD.md` in the mapped workspace
- falls back to `mdMirror.dir` when needed
- does not replace LanceDB retrieval/storage
Recommended starter config:
```json
{
"mdMirror": {
"enabled": true,
"dir": "memory-md"
}
}
```
### 11. Long Context Chunking
Automatically handles documents that exceed embedding model context limits:
- **Smart splitting**: Chunks at sentence boundaries with configurable overlap (default: 200 chars)
- **Averaged embedding**: Each chunk is embedded separately, then averaged for semantic preservation
- **Graceful error handling**: Detects "Input length exceeds context length" errors and retries with chunking
- **Config toggle**: `embedding.chunking` - set `false` to disable (default: auto-enabled on context-length errors)
- **Adapts to model limits**: Jina (8192 tokens), OpenAI (8191), Gemini (2048), etc.
See [`docs/long-context-chunking.md`](https://github.com/win4r/memory-lancedb-pro/blob/HEAD/docs/long-context-chunking.md) for implementation details.
### 12. Embedding Error Diagnostics
When embedding calls fail, the plugin provides **actionable error messages** instead of generic errors:
- **Auth errors** (401/403): hints to check API key validity and format
- **Network errors** (ECONNREFUSED, ETIMEDOUT): hints to verify `baseURL` and network connectivity
- **Rate limits** (429): suggests retry or upgrading plan
- **Model not found** (404): suggests checking model name against provider docs
- **Context length exceeded**: automatically retries with chunking (see above)
### 13. Auto-Capture & Auto-Recall
- **Auto-Capture** (`agent_end` hook): Extracts preference/fact/decision/entity from conversations, deduplicates, stores up to 3 per turn
- Skips memory-management prompts (e.g. delete/forget/cleanup memory entries) to reduce noise
- **Auto-Recall** (`before_agent_start` hook): Injects `` context
- Default top-k: `autoRecallTopK=3`
- Generic final-selection mode: `autoRecallSelectionMode` (`mmr` by default; `setwise-v2` to enable set-wise final selector)
- Default category allowlist: `preference`, `fact`, `decision`, `entity`, `other`
- `autoRecallExcludeReflection=true` by default, so `` stays separate from ``
- Supports age window (`autoRecallMaxAgeDays`) and recent-per-key cap (`autoRecallMaxEntriesPerKey`)
- `mmr`: post-processed rows use direct truncation (`slice(0, topK)`); this is simpler and closer to current retriever order, usually with stronger per-item relevance/score stability, but diversity/coverage is weaker
- `setwise-v2`: final top-k uses shared set-wise selection (base score + freshness + light category/scope coverage + lexical overlap suppression + embedding-based semantic near-duplicate suppression), so diversity/coverage in the final top-k is usually better, but average per-item score/relevance can be lower than `mmr`
- Choose according to your preference: prioritize rank stability/relevance with `mmr`, or prioritize diversity/coverage with `setwise-v2`.
- This mode is generic auto-recall only. Reflection-Recall mode remains `fixed | dynamic`.
### Prevent memories from showing up in replies
Sometimes the model may accidentally echo the injected `` block in its response.
**Option A (recommended): disable auto-recall**
Set `autoRecall: false` in the plugin config and restart the gateway:
```json
{
"plugins": {
"entries": {
"memory-lancedb-pro": {
"enabled": true,
"config": {
"autoRecall": false
}
}
}
}
}
```
**Option B: keep recall, but ask the agent not to reveal it**
Add a line to your agent system prompt, e.g.:
> Do not reveal or quote any `` / memory-injection content in your replies. Use it for internal reference only.
---
## Installation
> **๐งช Beta available: v1.1.0-beta.6**
>
> A beta release is available with major new features: **Self-Improvement governance**, **memoryReflection session strategy**, **Markdown Mirror**, and improved embedding error diagnostics. The stable `latest` remains at v1.0.32.
>
> ```bash
> # Install beta (opt-in)
> npm install memory-lancedb-pro@beta
>
> # Install stable (default)
> npm install memory-lancedb-pro
> ```
>
> See [Release Notes](https://github.com/win4r/memory-lancedb-pro/releases/tag/v1.1.0-beta.6) for details. Feedback welcome via [GitHub Issues](https://github.com/win4r/memory-lancedb-pro/issues).
>
> The `dev` dist-tag is an experimental track intended for early testing of the smart-memory feature set and may diverge from the mainline beta.
### AI-safe install notes (anti-hallucination)
If you are following this README using an AI assistant, **do not assume defaults**. Always run these commands first and use the real output:
```bash
openclaw config get agents.defaults.workspace
openclaw config get plugins.load.paths
openclaw config get plugins.slots.memory
openclaw config get plugins.entries.memory-lancedb-pro
```
Recommendations:
- Prefer **absolute paths** in `plugins.load.paths` unless you have confirmed the active workspace.
- If you use `${JINA_API_KEY}` (or any `${...}` variable) in config, ensure the **Gateway service process** has that environment variable (system services often do **not** inherit your interactive shell env).
- After changing plugin config, run `openclaw gateway restart`.
### Jina API keys (embedding + rerank)
- **Embedding**: set `embedding.apiKey` to your Jina key (recommended: use an env var like `${JINA_API_KEY}`).
- **Rerank** (when `retrieval.rerankProvider: "jina"`): you can typically use the **same** Jina key for `retrieval.rerankApiKey`.
- If you use a different rerank provider (`siliconflow`, `pinecone`, etc.), `retrieval.rerankApiKey` should be that provider's key.
Key storage guidance:
- Avoid committing secrets into git.
- Using `${...}` env vars is fine, but make sure the **Gateway service process** has those env vars (system services often do not inherit your interactive shell environment).
### What is the "OpenClaw workspace"?
In OpenClaw, the **agent workspace** is the agent's working directory (default: `~/.openclaw/workspace`).
According to the docs, the workspace is the **default cwd**, and **relative paths are resolved against the workspace** (unless you use an absolute path).
> Note: OpenClaw configuration typically lives under `~/.openclaw/openclaw.json` (separate from the workspace).
**Common mistake:** cloning the plugin somewhere else, while keeping a **relative path** like `plugins.load.paths: ["plugins/memory-lancedb-pro"]`. Relative paths can be resolved against different working directories depending on how the Gateway is started.
To avoid ambiguity, use an **absolute path** (Option B) or clone into `/plugins/` (Option A) and keep your config consistent.
### Option A (recommended): clone into `plugins/` under your workspace
```bash
# 1) Go to your OpenClaw workspace (default: ~/.openclaw/workspace)
# (You can override it via agents.defaults.workspace.)
cd /path/to/your/openclaw/workspace
# 2) Clone the plugin into workspace/plugins/
git clone https://github.com/win4r/memory-lancedb-pro.git plugins/memory-lancedb-pro
# 3) Install dependencies
cd plugins/memory-lancedb-pro
npm install
```
Then reference it with a relative path in your OpenClaw config:
```json
{
"plugins": {
"load": {
"paths": ["plugins/memory-lancedb-pro"]
},
"entries": {
"memory-lancedb-pro": {
"enabled": true,
"config": {
"embedding": {
"apiKey": "${JINA_API_KEY}",
"model": "jina-embeddings-v5-text-small",
"baseURL": "https://api.jina.ai/v1",
"dimensions": 1024,
"taskQuery": "retrieval.query",
"taskPassage": "retrieval.passage",
"normalized": true
}
}
}
},
"slots": {
"memory": "memory-lancedb-pro"
}
}
}
```
### Option B: clone anywhere, but use an absolute path
```json
{
"plugins": {
"load": {
"paths": ["/absolute/path/to/memory-lancedb-pro"]
}
}
}
```
### Restart
```bash
openclaw gateway restart
```
> **Note:** If you previously used the built-in `memory-lancedb`, disable it when enabling this plugin. Only one memory plugin can be active at a time.
### Verify installation (recommended)
1) Confirm the plugin is discoverable/loaded:
```bash
openclaw plugins list
openclaw plugins info memory-lancedb-pro
```
2) If anything looks wrong, run the built-in diagnostics:
```bash
openclaw plugins doctor
```
3) Confirm the memory slot points to this plugin:
```bash
# Look for: plugins.slots.memory = "memory-lancedb-pro"
openclaw config get plugins.slots.memory
```
---
## Configuration
Full Configuration Example (click to expand)
```json
{
"embedding": {
"apiKey": "${JINA_API_KEY}",
"model": "jina-embeddings-v5-text-small",
"baseURL": "https://api.jina.ai/v1",
"dimensions": 1024,
"taskQuery": "retrieval.query",
"taskPassage": "retrieval.passage",
"normalized": true
},
"dbPath": "~/.openclaw/memory/lancedb-pro",
"autoCapture": true,
"autoRecall": false,
"autoRecallMinLength": 8,
"autoRecallTopK": 3,
"autoRecallSelectionMode": "mmr",
"autoRecallCategories": ["preference", "fact", "decision", "entity", "other"],
"autoRecallExcludeReflection": true,
"autoRecallMaxAgeDays": 30,
"autoRecallMaxEntriesPerKey": 10,
"retrieval": {
"mode": "hybrid",
"vectorWeight": 0.7,
"bm25Weight": 0.3,
"minScore": 0.45,
"rerank": "cross-encoder",
"rerankApiKey": "${JINA_API_KEY}",
"rerankModel": "jina-reranker-v3",
"rerankEndpoint": "https://api.jina.ai/v1/rerank",
"rerankProvider": "jina",
"candidatePoolSize": 20,
"recencyHalfLifeDays": 14,
"recencyWeight": 0.1,
"filterNoise": true,
"lengthNormAnchor": 500,
"hardMinScore": 0.55,
"timeDecayHalfLifeDays": 60,
"reinforcementFactor": 0.5,
"maxHalfLifeMultiplier": 3
},
"enableManagementTools": false,
"sessionStrategy": "systemSessionMemory",
"scopes": {
"default": "global",
"definitions": {
"global": { "description": "Shared knowledge" },
"agent:discord-bot": { "description": "Discord bot private" }
},
"agentAccess": {
"discord-bot": ["global", "agent:discord-bot"]
}
},
"selfImprovement": {
"enabled": true,
"beforeResetNote": true,
"skipSubagentBootstrap": true,
"ensureLearningFiles": true
},
"memoryReflection": {
"storeToLanceDB": true,
"injectMode": "inheritance+derived",
"agentId": "memory-distiller",
"messageCount": 120,
"maxInputChars": 24000,
"timeoutMs": 20000,
"thinkLevel": "medium",
"errorReminderMaxEntries": 3,
"dedupeErrorSignals": true,
"recall": {
"mode": "fixed",
"topK": 6,
"includeKinds": ["invariant"],
"maxAgeDays": 45,
"maxEntriesPerKey": 10,
"minRepeated": 2,
"minScore": 0.18,
"minPromptLength": 8
}
},
"mdMirror": {
"enabled": false,
"dir": "memory-md"
}
}
```
Note: this example keeps `sessionStrategy: "systemSessionMemory"` for compatibility with existing users. The `memoryReflection.*` block documents the opt-in reflection pipeline, but it only becomes active after you explicitly switch `sessionStrategy` to `"memoryReflection"`.
### Parameter Mapping Notes (avoid common misconfig)
`memory-lancedb-pro` does **not** support `recallTopK` / `recallThreshold`.
Use these equivalents instead:
- `recallTopK` โ `retrieval.candidatePoolSize`
- `recallThreshold` โ combine `retrieval.minScore` + `retrieval.hardMinScore`
A practical starting point for Chinese chat workloads:
```json
{
"autoCapture": true,
"autoRecall": true,
"autoRecallMinLength": 8,
"autoRecallSelectionMode": "mmr",
"autoRecallExcludeReflection": true,
"retrieval": {
"candidatePoolSize": 20,
"minScore": 0.45,
"hardMinScore": 0.55
}
}
```
### Access Reinforcement (1.0.26)
To make frequently used memories decay more slowly, the retriever can extend the effective time-decay half-life based on **manual recall frequency** (spaced-repetition style).
Config keys (under `retrieval`):
- `reinforcementFactor` (range: 0-2, default: `0.5`) - set `0` to disable
- `maxHalfLifeMultiplier` (range: 1-10, default: `3`) - hard cap: effective half-life โค base ร multiplier
Notes:
- Reinforcement is **whitelisted to `source: "manual"`** (i.e. user/tool initiated recall), to avoid accidental strengthening from auto-recall.
### Embedding Providers
This plugin works with **any OpenAI-compatible embedding API**:
| Provider | Model | Base URL | Dimensions |
|----------|-------|----------|------------|
| **Jina** (recommended) | `jina-embeddings-v5-text-small` | `https://api.jina.ai/v1` | 1024 |
| **OpenAI** | `text-embedding-3-small` | `https://api.openai.com/v1` | 1536 |
| **Google Gemini** | `gemini-embedding-001` | `https://generativelanguage.googleapis.com/v1beta/openai/` | 3072 |
| **Ollama** (local) | `nomic-embed-text` | `http://localhost:11434/v1` | _provider-specific_ (set `embedding.dimensions` to match your Ollama model output) |
### Rerank Providers
Cross-encoder reranking supports multiple providers via `rerankProvider`:
| Provider | `rerankProvider` | Endpoint | Example Model |
|----------|-----------------|----------|---------------|
| **Jina** (default) | `jina` | `https://api.jina.ai/v1/rerank` | `jina-reranker-v3` |
| **SiliconFlow** (free tier available) | `siliconflow` | `https://api.siliconflow.com/v1/rerank` | `BAAI/bge-reranker-v2-m3`, `Qwen/Qwen3-Reranker-8B` |
| **Voyage AI** | `voyage` | `https://api.voyageai.com/v1/rerank` | `rerank-2.5` |
| **Pinecone** | `pinecone` | `https://api.pinecone.io/rerank` | `bge-reranker-v2-m3` |
| **vLLM / Docker Model Runner** | `vllm` | _requires custom endpoint_ | `Qwen3-Reranker` |
Notes:
- `voyage` sends `{ model, query, documents }` without `top_n`.
- Voyage responses are parsed from `data[].relevance_score`.
- `vllm` requires a custom `rerankEndpoint` (no API key needed). Only works on x86_64 NVIDIA platforms.
SiliconFlow Example
```json
{
"retrieval": {
"rerank": "cross-encoder",
"rerankProvider": "siliconflow",
"rerankEndpoint": "https://api.siliconflow.com/v1/rerank",
"rerankApiKey": "sk-xxx",
"rerankModel": "BAAI/bge-reranker-v2-m3"
}
}
```
Voyage Example
```json
{
"retrieval": {
"rerank": "cross-encoder",
"rerankProvider": "voyage",
"rerankEndpoint": "https://api.voyageai.com/v1/rerank",
"rerankApiKey": "${VOYAGE_API_KEY}",
"rerankModel": "rerank-2.5"
}
}
```
Pinecone Example
```json
{
"retrieval": {
"rerank": "cross-encoder",
"rerankProvider": "pinecone",
"rerankEndpoint": "https://api.pinecone.io/rerank",
"rerankApiKey": "pcsk_xxx",
"rerankModel": "bge-reranker-v2-m3"
}
}
```
vLLM / Docker Model Runner Example
```json
{
"retrieval": {
"rerank": "cross-encoder",
"rerankProvider": "vllm",
"rerankEndpoint": "http://host.docker.internal:12434/engines/vllm/rerank",
"rerankModel": "ai/qwen3-reranker:0.6B"
}
}
```
**Note:** vLLM reranking only works on x86_64 NVIDIA platforms. For macOS Apple Silicon, use `llama.cpp` for embeddings but not vLLM for reranking.
```
---
## Optional: JSONL Session Distillation (Auto-memories from chat logs)
OpenClaw already persists **full session transcripts** as JSONL files:
- `~/.openclaw/agents//sessions/*.jsonl`
This plugin focuses on **high-quality long-term memory**. If you dump raw transcripts into LanceDB, retrieval quality quickly degrades.
Instead, **recommended (2026-02+)** is a **non-blocking `/new` pipeline**:
- Trigger: `command:new` (you type `/new`)
- Hook: enqueue a tiny JSON task file (fast; no LLM calls inside the hook)
- Worker: a user-level systemd service watches the inbox and runs **Gemini Map-Reduce** on the session JSONL transcript
- Store: writes **0-20** high-signal, atomic lessons into LanceDB Pro via `openclaw memory-pro import`
- Keywords: each memory includes `Keywords (zh)` with a simple taxonomy (Entity + Action + Symptom). Entity keywords must be copied verbatim from the transcript (no hallucinated project names).
- Notify: optional Telegram/Discord notification (even if 0 lessons)
See the self-contained example files in:
- `examples/new-session-distill/`
---
Legacy option: an **hourly distiller** cron that:
1) Incrementally reads only the **newly appended tail** of each session JSONL (byte-offset cursor)
2) Filters noise (tool output, injected ``, logs, boilerplate)
3) Uses a dedicated agent to **distill** reusable lessons / rules / preferences into short atomic memories
4) Stores them via `memory_store` into the right **scope** (`global` or `agent:`)
### What you get
- โ
Fully automatic (cron)
- โ
Multi-agent support (main + bots)
- โ
No re-reading: cursor ensures the next run only processes new lines
- โ
Memory hygiene: quality gate + dedupe + per-run caps
### Script
This repo includes the extractor script:
- `scripts/jsonl_distill.py`
It produces a small **batch JSON** file under:
- `~/.openclaw/state/jsonl-distill/batches/`
and keeps a cursor here:
- `~/.openclaw/state/jsonl-distill/cursor.json`
The script is **safe**: it never modifies session logs.
By default it skips historical reset snapshots (`*.reset.*`), slash-command/control-note lines (for example `/note self-improvement ...`), and excludes the distiller agent itself (`memory-distiller`) to prevent self-ingestion loops.
### Optional: restrict distillation sources (allowlist)
By default, the extractor scans **all agents** (except `memory-distiller`).
If you want higher signal (e.g., only distill from your main assistant + coding bot), set:
```bash
export OPENCLAW_JSONL_DISTILL_ALLOWED_AGENT_IDS="main,code-agent"
```
- Unset / empty / `*` / `all` โ allow all agents (default)
- Comma-separated list โ only those agents are scanned
### Recommended setup (dedicated distiller agent)
#### 1) Create a dedicated agent
```bash
openclaw agents add memory-distiller \
--non-interactive \
--workspace ~/.openclaw/workspace-memory-distiller \
--model openai-codex/gpt-5.2
```
#### 2) Initialize cursor (Mode A: start from now)
This marks all existing JSONL files as "already read" by setting offsets to EOF.
```bash
# Set PLUGIN_DIR to where this plugin is installed.
# - If you cloned into your OpenClaw workspace (recommended):
# PLUGIN_DIR="$HOME/.openclaw/workspace/plugins/memory-lancedb-pro"
# - Otherwise, check: `openclaw plugins info memory-lancedb-pro` and locate the directory.
PLUGIN_DIR="/path/to/memory-lancedb-pro"
python3 "$PLUGIN_DIR/scripts/jsonl_distill.py" init
```
#### 3) Create an hourly cron job (Asia/Shanghai)
Tip: start the message with `run ...` so `memory-lancedb-pro`'s adaptive retrieval will skip auto-recall injection (saves tokens).
```bash
# IMPORTANT: replace in the template below with your actual plugin path.
MSG=$(cat <<'EOF'
run jsonl memory distill
Goal: distill NEW chat content from OpenClaw session JSONL files into high-quality LanceDB memories using memory_store.
Hard rules:
- Incremental only: call the extractor script; do NOT scan full history.
- Store only reusable memories; skip routine chatter.
- English memory text + final line: Keywords (zh): ...
- < 500 chars, atomic.
- <= 3 memories per agent per run; <= 3 global per run.
- Scope: global for broadly reusable; otherwise agent:.
Workflow:
1) exec: python3 /scripts/jsonl_distill.py run
2) If noop: stop.
3) Read batchFile (created/pending)
4) memory_store(...) for selected memories
5) exec: python3 /scripts/jsonl_distill.py commit --batch-file
EOF
)
openclaw cron add \
--agent memory-distiller \
--name "jsonl-memory-distill (hourly)" \
--cron "0 * * * *" \
--tz "Asia/Shanghai" \
--session isolated \
--wake now \
--timeout-seconds 420 \
--stagger 5m \
--no-deliver \
--message "$MSG"
```
#### 4) Debug run
```bash
openclaw cron run --expect-final --timeout 180000
openclaw cron runs --id --limit 5
```
### Scope strategy (recommended)
When distilling **all agents**, always set `scope` explicitly when calling `memory_store`:
- Broadly reusable โ `scope=global`
- Agent-specific โ `scope=agent:`
This prevents cross-bot memory pollution.
### Rollback
- Disable/remove cron job: `openclaw cron disable ` / `openclaw cron rm `
- Delete agent: `openclaw agents delete memory-distiller`
- Remove cursor state: `rm -rf ~/.openclaw/state/jsonl-distill/`
---
## CLI Commands
```bash
# List memories (output includes the memory id)
openclaw memory-pro list [--scope global] [--category fact] [--limit 20] [--json]
# Search memories
openclaw memory-pro search "query" [--scope global] [--limit 10] [--json]
# View statistics
openclaw memory-pro stats [--scope global] [--json]
# Delete a memory by ID (supports 8+ char prefix)
# Tip: copy the id shown by `memory-pro list` / `memory-pro search` (or use --json for full output)
openclaw memory-pro delete
# Bulk delete with filters
openclaw memory-pro delete-bulk --scope global [--before 2025-01-01] [--dry-run]
# Export / Import
openclaw memory-pro export [--scope global] [--output memories.json]
openclaw memory-pro import memories.json [--scope global] [--dry-run]
# Re-embed all entries with a new model
openclaw memory-pro reembed --source-db /path/to/old-db [--batch-size 32] [--skip-existing]
# Migrate from built-in memory-lancedb
openclaw memory-pro migrate check [--source /path]
openclaw memory-pro migrate run [--source /path] [--dry-run] [--skip-existing]
openclaw memory-pro migrate verify [--source /path]
```
---
## Custom Commands (e.g. `/lesson`)
This plugin provides tool-level building blocks. Slash commands such as `/lesson` are **not** built into the plugin itself; they are convenience commands you define in your Agent/system prompt, and they call the registered tools below.
### Recommended command shortcuts
- `/remember `
- call `memory_store`
- choose the most appropriate category / importance / scope
- `/lesson `
- call `memory_store` twice:
- once as `category=fact` for the lesson itself
- once as `category=decision` for the actionable rule
- `/learn `
- call `self_improvement_log` with `type=learning`
- include `category`, `area`, `priority`, `details`, `suggestedAction` when available
- `/error `
- call `self_improvement_log` with `type=error`
- include reproducible symptom, context, and prevention / fix
- `/learnings` or `/review-learnings`
- call `self_improvement_review`
- `/skill `
- call `self_improvement_extract_skill`
### Example prompt snippets
Add rules like these to your `CLAUDE.md`, `AGENTS.md`, or system prompt:
```markdown
## /lesson command
When the user sends `/lesson `:
1. Use `memory_store` to save the raw lesson as `category=fact`
2. Use `memory_store` again to save the actionable takeaway as `category=decision`
3. Confirm both saved items briefly
## /learn command
When the user sends `/learn `:
1. Use `self_improvement_log` with `type=learning`
2. Include `details`, `suggestedAction`, `category`, `area`, and `priority` if the user provided them
3. Confirm the created learning entry id
## /error command
When the user sends `/error `:
1. Use `self_improvement_log` with `type=error`
2. Capture the reproducible failure signature, context, and suggested prevention/fix
3. Confirm the created error entry id
## /review-learnings command
When the user sends `/review-learnings`:
1. Use `self_improvement_review`
2. Return the governance snapshot
## /skill command
When the user sends `/skill `:
1. Use `self_improvement_extract_skill`
2. Confirm the generated skill path
```
### Built-in Tools Reference
| Tool | Description |
|------|-------------|
| `memory_store` | Store a memory (supports category, importance, scope) |
| `memory_recall` | Search memories (hybrid vector + BM25 retrieval) |
| `memory_forget` | Delete a memory by ID or search query |
| `memory_update` | Update an existing memory in-place |
| `memory_list` | List recent memories with optional filtering |
| `memory_stats` | Show scope/category statistics |
| `self_improvement_log` | Log a structured learning/error entry into `.learnings/` |
| `self_improvement_review` | Summarize governance backlog from `.learnings/` |
| `self_improvement_extract_skill` | Create a skill scaffold from a learning entry |
> **Note**: custom commands like `/lesson`, `/learn`, `/error`, `/review-learnings`, or `/skill` are prompt-level shortcuts. The actual plugin surface is the tool set above.
---
## Database Schema
LanceDB table `memories`:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string (UUID) | Primary key |
| `text` | string | Memory text (FTS indexed) |
| `vector` | float[] | Embedding vector |
| `category` | string | `preference` / `fact` / `decision` / `entity` / `reflection` / `other` |
| `scope` | string | Scope identifier (e.g., `global`, `agent:main`) |
| `importance` | float | Importance score 0-1 |
| `timestamp` | int64 | Creation timestamp (ms) |
| `metadata` | string (JSON) | Extended metadata |
---
## Troubleshooting
### "Cannot mix BigInt and other types" (LanceDB / Apache Arrow)
On LanceDB 0.26+ (via Apache Arrow), some numeric columns may be returned as `BigInt` at runtime (commonly: `timestamp`, `importance`, `_distance`, `_score`). If you see errors like:
- `TypeError: Cannot mix BigInt and other types, use explicit conversions`
upgrade to **memory-lancedb-pro >= 1.0.14**. This plugin now coerces these values using `Number(...)` before doing arithmetic (for example, when computing scores or sorting by timestamp).
## Iron Rules for AI Agents (้ๅพ)
> **For OpenClaw users**: copy the code block below into your `AGENTS.md` so your agent enforces these rules automatically.
```markdown
## Rule 1 - ๅๅฑ่ฎฐๅฟๅญๅจ๏ผ้ๅพ๏ผ
Every pitfall/lesson learned โ IMMEDIATELY store TWO memories to LanceDB before moving on:
- **Technical layer**: Pitfall: [symptom]. Cause: [root cause]. Fix: [solution]. Prevention: [how to avoid]
(category: fact, importance โฅ 0.8)
- **Principle layer**: Decision principle ([tag]): [behavioral rule]. Trigger: [when it applies]. Action: [what to do]
(category: decision, importance โฅ 0.85)
- After each store, immediately `memory_recall` with anchor keywords to verify retrieval.
If not found, rewrite and re-store.
- Missing either layer = incomplete.
Do NOT proceed to next topic until both are stored and verified.
- Also update relevant SKILL.md files to prevent recurrence.
## Rule 2 - LanceDB ๅซ็
Entries must be short and atomic (< 500 chars). Never store raw conversation summaries, large blobs, or duplicates.
Prefer structured format with keywords for retrieval.
## Rule 3 - Recall before retry
On ANY tool failure, repeated error, or unexpected behavior, ALWAYS `memory_recall` with relevant keywords
(error message, tool name, symptom) BEFORE retrying. LanceDB likely already has the fix.
Blind retries waste time and repeat known mistakes.
## Rule 4 - ็ผ่พๅ็กฎ่ฎค็ฎๆ ไปฃ็ ๅบ
When working on memory plugins, confirm you are editing the intended package
(e.g., `memory-lancedb-pro` vs built-in `memory-lancedb`) before making changes;
use `memory_recall` + filesystem search to avoid patching the wrong repo.
## Rule 5 - ๆไปถไปฃ็ ๅๆดๅฟ
้กปๆธ
jiti ็ผๅญ๏ผMANDATORY๏ผ
After modifying ANY `.ts` file under `plugins/`, MUST run `rm -rf /tmp/jiti/` BEFORE `openclaw gateway restart`.
jiti caches compiled TS; restart alone loads STALE code. This has caused silent bugs multiple times.
Config-only changes do NOT need cache clearing.
```
---
## Dependencies
| Package | Purpose |
|---------|---------|
| `@lancedb/lancedb` โฅ0.26.2 | Vector database (ANN + FTS) |
| `openai` โฅ6.21.0 | OpenAI-compatible Embedding API client |
| `@sinclair/typebox` 0.34.48 | JSON Schema type definitions (tool parameters) |
---
## Contributors
Top contributors (from GitHub's contributors list, sorted by commit contributions; bots excluded):
- [@win4r](https://github.com/win4r) (4 commits)
- [@kctony](https://github.com/kctony) (2 commits)
- [@Akatsuki-Ryu](https://github.com/Akatsuki-Ryu) (1 commit)
- [@AliceLJY](https://github.com/AliceLJY) (1 commit)
- [@JasonSuz](https://github.com/JasonSuz) (1 commit)
- [@Minidoracat](https://github.com/Minidoracat) (1 commit)
- [@rwmjhb](https://github.com/rwmjhb) (1 commit)
- [@furedericca-lab](https://github.com/furedericca-lab) (1 commit)
- [@joe2643](https://github.com/joe2643) (1 commit)
- [@chenjiyong](https://github.com/chenjiyong) (1 commit)
Full list: https://github.com/win4r/memory-lancedb-pro/graphs/contributors
## โญ Star History
## License
MIT
---
## Buy Me a Coffee
[](https://ko-fi.com/aila)
## My WeChat Group and My WeChat QR Code
