I Stopped Telling My AI to Remember Things

For a while I treated AI memory like a sticky note problem. "Remember that we decided X." "Save this to memory." It worked until it didn't.

The agent would remember the last thing I asked it to save and forget everything else. Or it would confidently recall a detail that never made it out of the chat log. Or I'd start a fresh session and paste the same three paragraphs of context again, like a very tired onboarding doc.

I wanted memory that compounded. Not prompts I had to babysit.

This post is about the first system that actually worked for me: my Obsidian vault, indexed by QMD, wired into OpenClaw (my self-hosted agent). If you use Cursor instead of OpenClaw, the storage layer is the same idea. I wrote about the next layer here.

Chat is short-term memory. Files are long-term memory.

Every agent session starts cold. You can stretch that with longer context windows, pasted summaries, or clever system prompts. That buys you a day. It does not buy you a month.

The failure mode I kept hitting:

• Re-explaining. I'd already scoped PortRadar, tuned a Discord channel, or decided how cost logs should roll up. New chat, same conversation from zero.
• Lost decisions. The agent would propose an approach we'd explicitly rejected last week.
• No retrieval. Even when I did save something, I had to remember what I saved and where. The assistant didn't go find it.

The fix wasn't "remember harder." It was write things down in markdown the agent can search, and automate what goes into that store.

Obsidian was already where I kept notes. QMD made the vault searchable inside OpenClaw. Heartbeat and auto-flush made sure new context actually landed in the vault without me typing "save this" every ten minutes.

What QMD does (without the textbook)

QMD is a local search sidecar for OpenClaw. In plain terms: it reads my markdown files on a schedule and answers "what notes match this question?"

Under the hood it blends keyword search, semantic similarity, and reranking. I don't think about those layers day to day. I think about outcomes:

• I ask what we discussed about a topic.
• QMD returns the relevant note(s).
• The agent reads them and continues with actual history instead of guessing.

The feature that mattered most: QMD can index paths outside the agent's workspace. My vault lives on the host, mounted into the container. QMD re-indexes it every few minutes. The agent workspace stays operational. The vault stays the durable record.

Before and after

Before: I opened a new OpenClaw thread about my cost dashboard. The agent suggested building a SQLite + Chart.js pipeline from scratch. We'd already scoped that build, named the JSONL fields, and decided which Discord channels to track. All of that lived in a session summary I'd never written down properly. I pasted notes manually. Again.

After: I asked what we'd already decided about cost observability. QMD surfaced a session note with the channel breakdown, the spike threshold idea, and the "no third-party analytics" constraint. The agent picked up mid-stream instead of restarting.

That one moment sold me. Memory stopped being a prompt I repeated and became a system that went and looked.

It is not magic. Vague questions still miss. Stale notes still rank. I'll get to that.

What I changed in the vault (three rules, not a remodel)

I already had an Obsidian vault: Finance, Health, Records, Reviews, Travel. Fine for humans. Rough for search and agents.

I did not need a perfect PKM system. I needed three habits:

1. Keep folders relatively flat. Deep nesting makes retrieval worse. If I can't guess which folder a note lives in, search shouldn't have to either.
2. Add frontmatter and tags on anything an agent should find. Title and tags beat a clever filename buried three levels down.
3. Separate AI session notes from personal records. Session summaries and project context go in their own area. I don't want "what we decided about PortRadar" indexed next to tax docs.

I added a Work/ area for projects, an ai/ area for session summaries, and frontmatter on the files agents touch most. I skipped the fantasy of tagging all 200 notes in a weekend. I tagged what retrieval needed first and expanded over time.

How notes get in without me asking

Search only helps if the vault stays current. I did not trust myself to append notes after every session.

Two automations carried most of the load:

Heartbeat (every ~30 minutes): the agent reviews recent session transcripts, pulls out decisions, context, and learnings worth keeping, and writes them to the vault. Not everything. Just what would hurt to lose.

Auto-flush (before context compaction): when a long session gets summarized to free tokens, important context gets written to disk first. Compaction stops being amnesia by another name.

The heartbeat instruction is boring on purpose: review transcripts, identify what's worth saving, write it to Obsidian. Boring is durable. Fancy prompts drift.

This is the part most "AI memory" posts skip. Setup is a weekend. Ingestion is the product. A beautiful index of notes you never update is just an empty filing cabinet.

What still breaks

I want to be honest because overstating retrieval is how you lose trust in the whole stack.

• Stale notes win. If I changed a decision in code but not in the vault, QMD happily surfaces the old version. The agent then builds confidently on bad context. Same lesson I hit later with living docs in a repo: stale memory is worse than none.
• Vague queries miss. "What were we doing?" is weak. "What did we decide about Discord channel cost tracking?" works.
• Garbage in, garbage out. Heartbeat occasionally saves something too generic to be useful. I prune when I notice.
• Search is not identity. QMD answers "what notes match?" It does not tell the agent who I am, how I write, or which procedures to follow. That needed a different layer (see below).

None of that made me rip it out. It made me treat the vault like a product: ingest, index, prune, repeat.

Where I went next

QMD solved retrieval inside OpenClaw. One agent, one container, past context on demand.

My setup today goes further: the same Obsidian vault is the thinking backend for Cursor across every repo, with briefing files, skills, and scope rules so any session knows how to work with me without loading the whole vault. I wrote that up in Obsidian as My Thinking Backend for Cursor.

Same philosophy in both posts: file over AI. Tools rotate. Plain markdown stays. QMD was the first time that actually felt automatic instead of manual.

If you're running OpenClaw with Obsidian, QMD plus heartbeat is still the combo I'd start with. Get ingestion working before you optimize search.

Setup (if you want to try this)

Install QMD:

npm install -g @tobilu/qmd

Point it at your vault in OpenClaw config (paths live on the host mount):

{
  "memory": {
    "backend": "qmd",
    "qmd": {
      "paths": [{
        "name": "obsidian",
        "path": "/home/node/.openclaw/workspace-records/obsidian_vault",
        "pattern": "**/*.md"
      }]
    }
  }
}

Adjust the path to wherever your vault is mounted inside the container. Re-index on a short interval while you're iterating.

Then wire heartbeat and auto-flush so the vault grows without "remember this" prompts. That matters more than perfect folder taxonomy on day one.