I'm Building an Open-Source AI Memory System
Every time you start a new conversation with Claude or ChatGPT, it has no idea who you are. No idea what you built last week, what decisions you made last month, what problems you've already solved. You either re-explain everything from scratch, or you paste in a wall of context and hope it fits in the window.
I got tired of that.
So I built a solution. I've been running it every day for the past two months, across every project I'm working on. It knows my architecture decisions, my past conversations, my product strategy, my learning notes. When I open a new Claude session, it recalls what I need before I even ask. It's become the second brain I didn't know I needed.
Now I'm open-sourcing it. The project is called Memory Vault.
The problem with AI memory today
Most people work around the context window problem in one of three ways. They keep a running document they paste in at the start of every session. They rely on Claude's built-in memory features, which are limited and cloud-dependent. Or they just accept the limitation and re-explain things constantly.
None of these are good enough when you're using AI seriously. When you're building products, making architecture decisions, learning new skills, and referencing work you did months ago — you need something that actually persists, actually searches, and actually understands the relationships between things you've stored.
The solutions that exist today are either too complex (Neo4j knowledge graphs that require a team to maintain), too simple (a JSON file with entities and relations), or cloud-dependent (your data lives on someone else's server). There's a gap in the middle: a self-hosted, local-first memory system that's powerful enough to be genuinely useful but simple enough for a single developer to run and own.
That's the gap Memory Vault fills.
What I built — and actually use
Memory Vault is a local-first AI memory system built on PostgreSQL and pgvector. It stores anything you want to remember — decisions, conversations, notes, project context — and makes it searchable through hybrid search (semantic + keyword combined). It connects to Claude via MCP, which means Claude can recall and store memories during any conversation without you having to do anything manually.
The core features:
- Hybrid search — semantic similarity + keyword matching combined, so you find the right memory even when you don't remember the exact words
- Knowledge graph — entities and relationships extracted automatically, so connections between things emerge over time
- MCP integration — four tools (
recall,remember,forget,memory_status) that Claude can use natively during any session - Memory spaces — separate namespaces for different projects or domains, so your work memories don't mix with your learning notes
- Local LLM chat — a dashboard where you can query your own memories through a local model (Ollama, LM Studio) without sending anything to the cloud
- Self-hosted — your data stays on your machine, always
I'm not describing a prototype. I'm describing what I run every day. The system currently holds knowledge across five memory spaces, and it's active in every Claude session I open across all my projects.
Why open-source
Because the value of a memory system grows with the community around it. More people using it means more adapters, more integrations, more feedback on what actually matters in practice. And because I genuinely believe that your AI memory should belong to you — not to a cloud platform, not to a subscription service, not to anyone else.
The free version will be MIT licensed. Everything that makes it useful as a personal memory system — hybrid search, MCP integration, knowledge graph, dashboard, local LLM chat, Docker setup — will be free forever.
There will eventually be a PRO tier for teams and advanced features. But the core will always be open.
How I'm building this
In public. Every milestone ships as code on GitHub plus a blog post here explaining what was built and why. No black box development. No big reveal. Just consistent progress, documented honestly.
The plan is nine milestones over roughly 15 weeks:
- The Announcement — this post, plus the repo going public
- The Core — hybrid search engine, ingestion pipeline, embeddings
- One Command to Start — Docker setup
- Talk to Claude — MCP server with full tool support
- The Dashboard — web UI for search, browse, ingest
- The REST API — for integrations and custom clients
- The Knowledge Graph — entity extraction and visualization
- v1.0 Release — local LLM chat, polish, full launch
- PRO Unlocked — team features, advanced analytics, paid tier
Each milestone is a working, usable increment. Not a placeholder. Not a demo. Something you can actually run.
Who this is for
Developers who use AI seriously and are frustrated by starting from zero every session. Solo builders working across multiple projects who need context to follow them around. Anyone who values privacy and wants their data to stay on their own machine.
If you've ever copy-pasted the same context block into a new conversation because the AI forgot everything again — this is for you.
What's next
The GitHub repository goes live alongside this post. Right now it has the README, the architecture overview, and the project vision. The code follows in Milestone 2.
If this sounds interesting, watch the repo. The build starts now.