原始内容
Rust Self-Learning Memory System
A self-learning episodic memory system with semantic pattern search, embeddings, MCP server, and a full-featured CLI.
Overview • Features • Quick Start • Documentation • Contributing • Quality Gates • License
Overview
The Rust Self-Learning Memory System provides persistent memory across agent interactions through a comprehensive MCP (Model Context Protocol) server. It captures, stores, and learns from episodic experiences to improve future performance.
Architecture:
- do-memory-core: Core memory operations, pattern extraction, and reward scoring
- do-memory-storage-turso: Primary database storage (libSQL)
- do-memory-storage-redb: Fast embedded cache layer
- do-memory-mcp: MCP server (lazy tool loading; code-exec fail-closed)
- do-memory-cli: Full-featured command-line interface (episode, pattern, storage, playbook, feedback, and more)
- do-memory-test-utils: Shared testing utilities
- do-memory-benches: Comprehensive benchmark suite
- do-memory-examples: Usage examples and demonstrations
Tech Stack: Rust 2024 edition / Tokio + Turso/libSQL + redb cache + optional embeddings (OpenAI, Mistral, local)
Features
🧠 Episodic Memory
- Complete episode lifecycle (start → execute → score → learn → retrieve)
- Detailed execution step logging with tool usage tracking
- Intelligent reward scoring with efficiency and quality bonuses
- Automatic reflection generation for learning
📚 Multiple Storage Backends
- Turso Cloud: Remote libSQL database (default)
- redb Cache: Fast embedded key-value storage
- Local SQLite: Local file-based database (fallback)
- Automatic caching with TTL-based invalidation
🎯 Pattern Recognition & Semantic Search
- Four pattern types: ToolSequence, DecisionPoint, ErrorRecovery, ContextPattern
🔄 Episode Checkpoints and Handoff
- Checkpoint episodes mid-task for long-running workflows
- Generate handoff packs to transfer state between agents or sessions
- Resume execution from saved checkpoints
- State preservation with findings and pending actions
🌊 CSM Cascading Retrieval
- 100% CPU-local retrieval via Chaotic Semantic Memory (CSM)
- 4-tier cascade (BM25 -> HDC -> ConceptGraph -> API Embeddings)
- Hyperdimensional computing (HDC) binary vectors for zero-API similarity search
- 50-70% reduction in external API embedding calls
- Semantic pattern search with natural language queries
- Intelligent pattern recommendations for tasks using multi-signal ranking
- Cross-domain pattern discovery to find analogous patterns
- Async pattern extraction with queue-based workers
- Pattern effectiveness tracking with decay over time
- Multi-signal ranking: semantic similarity, context match, effectiveness, recency, success rate
- Minimum success rate filtering (default 70%)
🔒 Agent Code Execution (fail-closed)
execute_agent_codeis not a working execution backend; calls fail closed- Prefer episode lifecycle tools and external runners for agent code
- Historical WASM/Javy sandbox paths are removed from the supported feature set
📊 Advanced Analysis
- Statistical analysis (ETS forecasting, MSTL decomposition)
- Anomaly detection and changepoint analysis
- Time series forecasting with configurable horizon
- Causal inference for pattern relationships
🔍 MCP Server
- MCP protocol implementation (v2025-11-25) with lazy tool loading
- MCP tools for memory operations, pattern search, episodes, and embeddings
search_patterns- Semantic pattern search with configurable rankingrecommend_patterns- Task-specific pattern recommendationsrecommend_playbook- Actionable step-by-step guidancecheckpoint_episode- Mid-task progress snapshots- Embedding tools: configure, test, generate, search, provider-status
- Progressive tool disclosure based on usage
- Execution monitoring and metrics tracking
🛠️ Full-Featured CLI
- Top-level command groups include: episode, pattern, storage, config, health, backup, monitor, logs, eval, embedding, completion, tag, relationship, playbook, feedback
- Episode management (create, list, view, search, log-step, complete, fail, delete, update, bulk, filter)
- Pattern analysis and effectiveness tracking (list, view, search, analyze, decay, batch)
- Discoverable config (
config init/config show-template) and documented precedence (flags → env → config → defaults) - Tag management (add, remove, search, rename, stats)
- Relationship management (add, remove, list, graph, validate)
- Storage operations (stats, sync, vacuum, health, connections) — sync is Turso↔redb only, not pattern extraction
- Backup and restore capabilities
- Multiple output formats (human, JSON, YAML)
🌐 Multi-Provider Embeddings
- OpenAI embeddings integration (text-embedding-3-small, text-embedding-3-large, ada-002)
- Mistral AI embeddings integration
- Local CPU-based embeddings
- Semantic search with cosine similarity
- Automatic embedding caching and batch processing
🛡️ Quality Assurance
- Automated quality gates (
./scripts/quality-gates.sh; seeplans/GATE_CONTRACT.md) - Large nextest suite across crates; doctests via
cargo test --doc - Property-based testing (proptest) and snapshot testing (insta)
- Mutation testing (cargo-mutants) in nightly CI
- Blocking advisory gate via
cargo deny; semver checks in CI - Zero clippy warnings policy (
-D warnings) - Skill eval contract:
./scripts/run-evals.sh(strict non-noop tests)
Quick Start
🔍 Pattern Search Example
use do_memory_core::{SelfLearningMemory, TaskContext, ComplexityLevel};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let memory = SelfLearningMemory::new();
// Search for patterns using natural language
let context = TaskContext {
domain: "web-api".to_string(),
language: Some("rust".to_string()),
framework: None,
complexity: ComplexityLevel::Moderate,
tags: vec!["rest".to_string(), "async".to_string()],
};
let results = memory.search_patterns_semantic(
"How to handle API rate limiting with retries",
context,
5 // limit
).await?;
for result in results {
println!("Pattern: {:?}", result.pattern);
println!("Relevance: {:.2}", result.relevance_score);
println!("Success Rate: {:.1}%", result.pattern.success_rate() * 100.0);
}
// Get task-specific recommendations
let recommendations = memory.recommend_patterns_for_task(
"Build async HTTP client with connection pooling",
context,
3
).await?;
for rec in recommendations {
println!("Recommended: {:?}", rec.pattern);
}
// Generate an actionable playbook
let playbooks = memory.retrieve_playbooks(
"Implement user authentication",
"security",
TaskType::CodeGeneration,
context,
1, // max playbooks
5 // max steps
).await;
if let Some(playbook) = playbooks.first() {
println!("Playbook ID: {}", playbook.playbook_id);
for step in &playbook.ordered_steps {
println!("{}. {}", step.order, step.action);
}
}
Ok(())
}
Documentation: See memory-core/PATTERN_SEARCH_FEATURE.md for complete API reference and examples.
Prerequisites
- Rust toolchain via rustup (stable, 2024 edition) — only needed if building from source
- SQLite (for local development)
- Optional: Turso CLI (for cloud database)
Install Pre-Built Binary
Pre-built binaries are available for all major platforms from GitHub Releases. No Rust toolchain required.
| Platform | Download |
|---|---|
| Linux x64 | do-memory-cli-x86_64-unknown-linux-gnu.tar.xz |
| Linux ARM64 | do-memory-cli-aarch64-unknown-linux-gnu.tar.xz |
| macOS Intel | do-memory-cli-x86_64-apple-darwin.tar.xz |
| macOS Apple Silicon | do-memory-cli-aarch64-apple-darwin.tar.xz |
| Windows x64 | do-memory-cli-x86_64-pc-windows-msvc.zip |
# Example: Linux x64
curl -sSL https://github.com/d-o-hub/rust-self-learning-memory/releases/latest/download/do-memory-cli-x86_64-unknown-linux-gnu.tar.xz | tar -xJ
# Add to PATH
export PATH="$PWD:$PATH"
Build from Source
The following system packages are required to compile from source:
| Platform | Command |
|---|---|
| Ubuntu/Debian | sudo apt-get install libssl-dev pkg-config |
| Fedora/RHEL | sudo dnf install openssl-devel pkg-config |
| Arch Linux | sudo pacman -S openssl pkg-config |
| macOS | brew install openssl pkg-config (usually pre-installed with Xcode Command Line Tools) |
Installation
# Clone the repository
git clone https://github.com/d-o-hub/rust-self-learning-memory.git
cd rust-self-learning-memory
# Build the project
cargo build --release
# Run tests (nextest recommended)
cargo nextest run --all
# Doctests separately
cargo test --doc
# Run quality gates
./scripts/quality-gates.sh
Local Database Setup (Zero Credentials)
The system supports a first-class local-only mode that requires no Turso account or cloud credentials.
# Via CLI (using auto-detected OS data directory)
do-memory-cli --storage-mode local episode list
# Via CLI (specifying a custom database path)
do-memory-cli --storage-mode local --db-path ./data/memory.db episode list
# Via Environment Variables
export MEMORY_STORAGE_MODE=local
export MEMORY_DB_PATH=./data/memory.db
do-memory-cli episode list
Note on
--db-path/MEMORY_DB_PATH: these set the local database file path. For the defaultlocal(redb) backend they set the redb cache file; for Tursoremotemode with a local file they set the Turso SQLite path. The config-file equivalent is[database].redb_path/[database].db_path. See TOML Configuration.
For programmatic usage, see the Local Development example.
Basic Usage
Setup Configuration
# Run interactive configuration wizard
do-memory-cli config wizard
# Follow the prompts to configure:
# - Database (local SQLite or remote Turso)
# - Storage (cache size, TTL, connection pool)
# - CLI (output format, progress bars, batch size)
# Generate a starter config file
do-memory-cli config init
# Print a starter config template
do-memory-cli config show-template
# Validate configuration
do-memory-cli config validate
# Check configuration status
do-memory-cli config check
Configuration Wizard provides interactive step-by-step setup with sensible defaults and validation.
CLI Interaction
# Create an episode
do-memory-cli episode create --task "Implement user authentication" --context '{"language": "rust", "domain": "auth"}'
# Log steps (needed for tool-sequence pattern extraction)
do-memory-cli episode log-step <episode-id> --tool cargo --action "run tests" --success true
# Complete with outcome (success | partial-success | failure)
do-memory-cli episode complete <episode-id> success
# Force-fail abandoned in_progress rows (operator path; ADR-075)
do-memory-cli episode fail <episode-id>
# List / search episodes
do-memory-cli episode list --limit 10
do-memory-cli episode search "authentication" --limit 5
# Patterns (populated after complete with steps; empty list explains why)
do-memory-cli pattern list --min-confidence 0.8
do-memory-cli pattern search --query "How to build REST API" --limit 5
# Tag management
do-memory-cli tag add <episode-id> "important"
do-memory-cli tag search "important"
# Health check
do-memory-cli health check
# Playbook recommendation
do-memory-cli playbook recommend "Implement JWT auth" --domain security
MCP Server
# Start the MCP server
cargo run --bin do-memory-mcp-server
# Or run with custom config
cargo run --bin do-memory-mcp-server -- --config mcp-config-memory.json
Programmatic Usage
Local Offline Development
use do_memory_core::SelfLearningMemory;
use do_memory_storage_turso::SelfLearningMemoryExt; // Required for convenience constructors
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// 1. Local file-based storage
let memory = SelfLearningMemory::with_local_storage("./data/memory.db").await?;
// 2. Or in-memory storage (ephemeral)
// let memory = SelfLearningMemory::with_in_memory_storage().await?;
Ok(())
}
Episode Recording
use do_memory_core::{SelfLearningMemory, TaskContext, TaskType, ComplexityLevel};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let memory = SelfLearningMemory::new();
let context = TaskContext {
language: Some("rust".to_string()),
framework: None,
complexity: ComplexityLevel::Moderate,
domain: "web".to_string(),
tags: vec!["api".to_string()],
};
let episode_id = memory.start_episode(
"Build REST API endpoint".to_string(),
context,
TaskType::CodeGeneration,
).await;
Ok(())
}
Documentation
| Document | Description |
|---|---|
| Configuration Wizard | Interactive setup guide |
| API Reference | Current MCP tool contract index |
| Configuration Guide | Complete configuration options |
| Database Setup | Local database configuration |
| Quality Gates | Automated quality standards |
| YAML Validation | Configuration validation strategy |
| Testing Guide | Testing infrastructure and strategies |
| Contributing | Development workflow |
| Security | Security policies and practices |
| Deployment | Deployment strategies |
| Release Engineering | Release workflow and automation |
| Playbooks & Checkpoints | Actionable memory and handoff |
Agent Documentation
| Document | Description |
|---|---|
| Building the Project | Build commands and setup |
| Running Tests | Testing strategies and coverage |
| Code Conventions | Rust idioms and patterns |
| Service Architecture | System design and components |
| Database Schema | Data structures and relationships |
| Communication Patterns | Inter-service communication |
| Agent Docs Index | Workflow docs and high-impact reference files |
Crate Documentation
| Crate | Description |
|---|---|
| do-memory-core | Core episodic learning system |
| do-memory-mcp | MCP server with secure sandbox |
| do-memory-cli | Command-line interface |
| do-memory-storage-turso | Turso/libSQL storage backend |
| do-memory-storage-redb | redb cache backend |
Quality Gates
The project maintains high quality standards through automated quality gates:
| Gate | Threshold | Description |
|---|---|---|
| Build | 0 errors | cargo build --all |
| Linting | 0 warnings | ./scripts/code-quality.sh clippy --workspace |
| Formatting | 100% | ./scripts/code-quality.sh fmt |
| Tests | All pass | cargo nextest run --all |
| File Size | ≤500 LOC | Production source files only |
| Security | 0 vulns | cargo audit in CI |
| Semver | No breaks | cargo semver-checks in CI |
Run quality gates locally:
./scripts/quality-gates.sh
For more details, see Quality Gates Documentation.
Feature Flags
Enable optional features via Cargo:
# Basic features (default)
cargo build
# All features
cargo build --all-features
# Specific features
cargo build --features openai
cargo build --features mistral
cargo build --features local-embeddings
cargo build --features embeddings-full
Available Features:
openai: OpenAI API embeddings support (do-memory-core)mistral: Mistral AI embeddings support (do-memory-core)local-embeddings: CPU-based local embeddings (do-memory-core)csm: Chaotic Semantic Memory integration for cascading retrieval (do-memory-core)embeddings-full: All embedding providers (do-memory-core)turso: Turso cloud storage with keepalive pool (do-memory-cli)redb: redb local cache layer (do-memory-cli, default)full: All features combined (do-memory-cli)oauth: Optional OAuth authorization support (do-memory-mcp)embeddings: Optional embeddings configuration support (do-memory-mcp)compression: Network compression — lz4, zstd, gzip (do-memory-storage-turso)hybrid_search: FTS5 hybrid search (do-memory-storage-turso)
Cascading Retrieval (with csm feature)
When the csm feature is enabled, semantic search uses a 4-tier cascade to minimize API calls:
| Tier | Method | CPU Cost | API Calls | When Used |
|---|---|---|---|---|
| 1 | BM25 exact match | O(n) Rayon scan | 0 | Keyword-heavy queries |
| 2 | HDC similarity | 10,240-bit SIMD | 0 | Semantic fallback |
| 3 | ConceptGraph expansion | Graph BFS | 0 | Known-domain synonyms |
| 4 | API embedding | Network call | 1 | Final fallback |
Target: 50-70% API call reduction for typical query workloads. See agent_docs/csm_integration.md for details.
Configuration
Configuration Precedence
Settings resolve highest-wins in this order (issue #846):
| Priority | Source | Examples |
|---|---|---|
| 1 (highest) | CLI flags | --db-path, --storage-mode |
| 2 | Environment variables | MEMORY_DB_PATH, MEMORY_STORAGE_MODE |
| 3 | Explicit config file | --config /path/to.toml |
| 4 | Auto-discovered config in CWD | do-memory-cli.toml, memory-cli.toml (and .memory-cli.toml / JSON / YAML variants) |
| 5 (lowest) | Built-in defaults | XDG paths, local storage defaults |
--db-path / MEMORY_DB_PATH and --storage-mode / MEMORY_STORAGE_MODE are clap options with env fallbacks: a CLI flag always beats the env var for that option, and both beat values loaded from a config file. See memory-cli/CONFIGURATION_GUIDE.md for the full reference.
Environment Variables
# Turso Cloud (default)
TURSO_URL=libsql://your-db.turso.io
TURSO_TOKEN=your-auth-token
# Local redb cache (default local backend)
LOCAL_DATABASE_URL=sqlite:./data/memory.db
REDB_PATH=./data/memory.redb
# CLI path / mode overrides (precedence levels 1–2 above)
MEMORY_DB_PATH=./data/memory.db
MEMORY_STORAGE_MODE=local
# CLI config
MEMORY_CLI_CONFIG=./memory-cli.toml
# Embeddings (CLI/MCP)
EMBEDDING_PROVIDER=openai|mistral|azure|local
OPENAI_API_KEY=sk-your-key
MISTRAL_API_KEY=your-mistral-key
AZURE_OPENAI_API_KEY=your-azure-key
OPENAI_API_KEY_ENV=OPENAI_API_KEY
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_SIMILARITY_THRESHOLD=0.7
EMBEDDING_BATCH_SIZE=32
# MCP OAuth (production HMAC verification)
# MCP_OAUTH_TOKEN_SECRET=...
TOML Configuration
[database]
turso_url = "libsql://your-db.turso.io"
turso_token = "your-auth-token"
redb_path = "memory.redb" # local (redb) cache file path
storage_mode = "local" # "remote" | "local" | "memory"
db_path = "./data/memory.db" # Turso local SQLite path (storage_mode = "local")
[storage]
max_episodes_cache = 1000
cache_ttl_seconds = 3600
pool_size = 10
[cli]
default_format = "human"
progress_bars = true
batch_size = 100
Tip: generate a valid starter config with
do-memory-cli config init(writesdo-memory-cli.toml) or print one withdo-memory-cli config show-template.
Architecture
┌─────────────────────────────────────────────────────────────┐
│ do-memory-cli │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Episode │ │ Pattern │ │ Storage │ │
│ │ Management │ │ Analysis │ │ Operations │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ do-memory-mcp │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ MCP Tools │ │ Lazy Tool │ │ Advanced │ │
│ │ Interface │ │ Loading │ │ Analysis │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ do-memory-core │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Episode │ │ Pattern │ │ Reward │ │
│ │ Management │ │ Extraction │ │ Scoring │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌──────────────────┴──────┬────────────────────────┐
│ │ │
┌───────────▼───────────┐ ┌───────────▼───────────┐ ┌───────────▼───────────┐
│do-memory-storage-turso│ │do-memory-storage-redb │ │do-memory-storage-redb │
│ │ │ │ │ │
│ libSQL/Remote │ │ Fast Access │ │ In-Memory/Temporary │
└───────────────────────┘ └───────────────────────┘ └───────────────────────┘
MCP Server Tools
The MCP server exposes tools via lazy loading (ADR-024):
- query_memory — Query episodic memory for relevant past experiences
- analyze_patterns — Identify successful strategies and recommendations
- search_patterns — Semantic pattern search with multi-signal ranking
- recommend_patterns — Task-specific pattern recommendations
- configure_embeddings / test_embeddings / generate_embedding — Embedding management
- search_by_embedding / embedding_provider_status — Semantic search and provider monitoring
- Episode lifecycle tools — create, complete, log steps, get, timeline
- Advanced analysis — Statistical analysis, forecasting, anomaly detection, causal inference
- Unavailable / fail-closed —
execute_agent_code(no working WASM execution backend)
Performance
All operations meet or exceed performance targets:
| Operation | Target (P95) | Typical Performance |
|---|---|---|
| Episode Creation | < 50ms | ~2.5 µs (19,531x faster) |
| Step Logging | < 20ms | ~1.1 µs (17,699x faster) |
| Episode Completion | < 500ms | ~3.8 µs (130,890x faster) |
| Pattern Extraction | < 1000ms | ~10.4 µs (95,880x faster) |
| Memory Retrieval | < 100ms | ~721 µs (138x faster) |
Typical performance numbers are from internal benchmarks on a warm cache; results vary by hardware and configuration. Run cargo bench for local measurements and see docs/QUALITY_GATES.md for the performance regression gate.
Benchmarks
Run cargo bench for workspace benchmarks. CLI benchmarks live in memory-cli/benches/cli_benchmarks.rs; quality gate expectations and regression checks are documented in docs/QUALITY_GATES.md.
Contributing
We welcome contributions! Please see our Contributing Guide for details.
Development Workflow
- Fork the repository
- Create a feature branch:
git checkout -b feature-name - Make your changes
- Run quality gates:
./scripts/quality-gates.sh - Submit a pull request
Code Standards
- Follow Rust idioms
- Test coverage targets per ADR-042 phases (70% → 75% → 80%)
- Run
cargo fmtandcargo clippybefore committing - Document public APIs
- Write descriptive commit messages
Quality Requirements
- All tests must pass
- No clippy warnings
- Test coverage targets per ADR-042:
- Phase 1: 70% (current focus - actual: 61.22%)
- Phase 2: 75%
- Phase 3: 80% (codecov.yml project target)
- Security audit must pass
- Performance benchmarks must not degrade >10%
License
This project is licensed under the MIT License - see the LICENSE file for details.
Support
Acknowledgments
- libSQL for the embedded database
- redb for the embedded key-value store
- tokio for asynchronous runtime
- Turso for the cloud database service