The Agent Memory Field Guide

01The Stack at a Glance

Three pieces that fit together cleanly:

1. A memory server

A self-hosted service that stores what your agent learns. Not a raw log dump. It extracts facts, links entities, builds a knowledge graph, and surfaces what actually matters when you ask.

2. A token-saving layer

A small tool that compresses command output before it reaches the model. Most terminal output is noise. Cut it at the source and the same work costs a fraction of the tokens.

3. A packaging system

A format for bundling any of the above into self-contained deploy units: knowledge doc plus install scripts plus verification, with a manifest. Hand a folder to another machine and it sets itself up.

How they connect

Your agent --MCP (http)--> memory server :8888 |-- embedded database (data volume) |-- local embeddings + reranker (bundled) |-- an LLM you bring (cloud key or local model) `-- web UI :9999 Your agent's shell --token-saving plugin--> compact command output

The shape of the whole thing One Docker container runs the memory server with its own database. One binary hooks into your coding agent. One folder format packages it all. No cloud account required, though you can point the LLM layer at any provider you already pay for.

02Agent Memory That Learns

The memory server is Hindsight, an open-source agent memory system by Vectorize (MIT license, very active project). It solves the problem every agent has: sessions start from zero. You explain your setup, your preferences, your decisions. Next session, gone.

The core operations

Operation	What it does
`retain`	You hand it a conversation or fact. It extracts structured facts, resolves entities ("my friend" and "the friend from last week" become one person), and indexes everything.
`recall`	You ask a question. It runs several retrieval strategies in parallel (semantic search, keyword match, graph traversal, time filters) and reranks with a cross-encoder. You get the useful few, not a raw dump.
`reflect`	It reasons across the whole graph with the LLM and synthesizes an answer that connects memories from different sessions.

On top of that it keeps mental models: living summaries that refresh themselves as new memories arrive. Like a constantly updated profile of a topic or a person.

Memory banks

Banks are isolated stores. Think of each as a separate brain. One bank for personal memory, one per project, one per client if you work with clients. The isolation is structural, not a convention.

Run it

The quickest path is Docker. This runs the API, the web UI, the embedded database, and the local embedding + reranker models in one container. The only thing you must bring is an LLM key (or a local model runner).

# run the full stack locally
docker run -d --name hindsight --restart unless-stopped \
  -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_PROVIDER=openai \
  -e HINDSIGHT_API_LLM_API_KEY=sk-your-key \
  -e HINDSIGHT_API_WORKER_ID=hindsight-prod \
  -v hindsight-data:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

# API:  http://localhost:8888   UI:  http://localhost:9999

LLM lanes The server needs an LLM for fact extraction and reflection. Swap providers with env vars, no code changes: OpenAI, Anthropic, Gemini, Groq, DeepSeek, or fully local via Ollama or a built-in llama.cpp mode. Local = zero cost, zero data leaving the machine, slower per memory. Cloud = fast, uses your existing subscription. Mix them: fast lane for active work, local lane for private retention runs.

Prove it works

# create a bank
curl -s -X PUT http://localhost:8888/v1/default/banks/my-bank \
  -H "Content-Type: application/json" \
  -d '{"name":"My Bank","reflect_mission":"Remember user preferences and project facts."}'

# retain a memory (this runs the LLM extraction)
curl -s -X POST http://localhost:8888/v1/default/banks/my-bank/memories \
  -H "Content-Type: application/json" \
  -d '{"items":[{"content":"User: I prefer functional programming and dark mode."}]}'

# recall it back
curl -s -X POST http://localhost:8888/v1/default/banks/my-bank/memories/recall \
  -H "Content-Type: application/json" -d '{"query":"What do I prefer?"}'

The retain response shows a usage breakdown, which is a handy way to measure the token cost of memory. Expect a few thousand tokens per memory with a cloud LLM.

Wire it into your agent (MCP)

The server exposes a standard MCP endpoint. Any agent that speaks MCP can use memory tools natively. In Hermes, for example:

# register the server (answer: no auth, enable all tools)
printf "n\ny\n" | hermes mcp add hindsight --url http://localhost:8888/mcp/

# verify
hermes mcp test hindsight
# expect: Connected, Tools discovered: 32

Tools appear next session as mcp_hindsight_*: retain, recall, reflect, bank management, mental models, directives, and more. The multi-bank endpoint takes a bank id per call, so one agent can keep strict separation between projects.

Security notes The local MCP endpoint has no auth by design. Keep it bound to localhost. Never expose the port to your LAN without adding an access layer.

03Token Saving: the CLI Proxy Trick

In early 2026 a Reddit post went viral: "I saved 10 million tokens (89%) on my Claude Code sessions with a CLI proxy." The tool was RTK (Rust Token Killer), and it spawned a whole ecosystem.

The insight

When a coding agent runs a command, the output goes straight into the model's context window. Most of it is noise: passing test lines, verbose logs, progress bars, status spam. A CLI proxy sits between the agent and the shell. It rewrites commands so the agent runs a filtered version, or it compresses output before the model ever sees it.

# without the proxy: the full noise
cargo test  ->  155 lines of output

# with the proxy: just the signal
cargo test  ->  "pass: 21  fail: 2  ignored: 1"

RTK for Hermes

RTK ships native hooks for many coding agents, including Hermes. One command installs a plugin that rewrites terminal tool calls so Hermes receives compact output. It is fail-open: if anything goes wrong, the original command runs unchanged.

# 1. install the rtk binary (download the release for your OS into a PATH dir)
mkdir -p ~/.local/bin && cp rtk ~/.local/bin/

# 2. install the Hermes plugin
# IMPORTANT: point HERMES_HOME at your real Hermes config directory first
export HERMES_HOME="C:\\Users\\you\\AppData\\Local\\hermes"   # Windows example
rtk init --agent hermes

# 3. verify
rtk --version

The path trap The installer defaults to ~/.hermes. On Windows the real Hermes home is usually AppData\Local\hermes. If the plugin lands in the wrong place it silently never loads. Set HERMES_HOME first. This bit us; do not repeat it.

The honest numbers

Read the claims with care. The 60-90% figures are about bash output bytes, which is one contributor to input tokens, which is part of the total bill. Real savings on a full session are meaningful but smaller. The maintainers of RTK themselves corrected their marketing to say this. Treat it as a real efficiency gain, not magic.

There is also the output side: a skill that makes the model reply tersely can cut output tokens by a large margin. Both levers are worth having.

04Packets: Handing Whole Setups Between Machines

A skill is knowledge. A packet is knowledge plus machinery: a manifest, install and verify scripts, config templates, and a checksummed registry. You hand a packet to another machine or another agent, and it installs the capability and proves it works.

The anatomy of a packet

packet-name/
  packet.yaml          # manifest: id, version, privacy, what it touches
  SKILL.md             # the procedural knowledge (standard agent skill format)
  install.ps1          # idempotent installer (PowerShell on Windows)
  install.sh           # same installer in bash for POSIX
  verify.ps1           # real probes, exits non-zero on any failure
  verify.sh
  templates/           # config snippets with placeholders

The manifest carries the contract

id: my-stack
kind: stack                  # or "snapshot" for whole-system backups
version: 1.0.0
privacy: internal            # internal | public
requires:
  docker: true
installs:
  skills: [my-stack]
  mcp_servers: [my-server]
touches:
  - $HERMES_HOME/config.yaml
data_policy:
  reads: [env vars, existing auth files]
  sends: []                  # empty = no telemetry, no exfiltration
verify: [docker-health, api-openapi, mcp-connect]

Two privacy flavors

internal

For your own machines and trusted fleets. May reference your paths and conventions. Never contains actual secrets; credentials are read from the environment at install time.

public

For friends and fresh starts. Zero owner identifiers, zero telemetry, user brings their own credentials. A build-time scrubber rewrites identifiers to placeholders and FAILS the build if any survive, so a public packet physically cannot ship with internals.

Snapshots double as backups

A kind: snapshot packet captures the state of a whole system: skills, plugins, config blocks, versions. Restore rebuilds it byte-identical. Two rules make this safe:

Secrets are never captured. Auth files, env files, key material are excluded by default, and a secrets sweep fails the build if anything leaks through.
For transport, encrypt. Package the snapshot with AES-256 and keep the passphrase separate. A lost drive with an unencrypted backup is a breach waiting to happen.

# capture + encrypt in one step
powershell -File make-snapshot.ps1 -Lean -AllowlistFile reviewed.txt -Package

# on the other machine: decrypt, then restore
openssl enc -d -aes-256-cbc -pbkdf2 -pass stdin -in snapshot.enc -out snapshot.zip
unzip snapshot.zip -d snapshot

Distribution

A private git repo is the canonical library (credential-gated by collaborator access). For one-off handoffs, zip the packet and send it, or upload to private object storage and share an expiring signed URL. The URL itself is the credential; it dies on schedule.

05The Copy-Paste Kit

Prompts that worked for us. Copy, adapt the placeholders, run in your coding agent (Claude Code, Codex, or any agent that reads a brief). The full library lives in the Prompt Library.

Prompt 1: Deploy a memory server and wire it in

Read the docs at https://hindsight.vectorize.io/developer/installation first.
Then:
1. Run the Hindsight memory server in Docker (embedded database, local
   embeddings and reranker, ports 8888 API and 9999 UI).
2. Configure an LLM provider. Use my API key from the environment
   variable (never print it). Default to the cheapest fast model.
3. Create a bank named "personal" with a reflect mission.
4. Prove it works end to end with real commands:
   - create the bank
   - retain one memory
   - recall it back
   Show me the actual outputs and the token usage from the retain call.
Ground rules:
- Never write, print, or store a secret anywhere.
- No em dashes in any user-visible string.
- Keep scripts ASCII-only.
- Verify with real probes, not "it should work".

Prompt 2: Add a token-saving layer to my agent

Install RTK (Rust Token Killer) for my coding agent.
1. Download the binary for this OS from the GitHub releases page and put
   it on PATH.
2. Check which agent integrations are supported (claude, cursor, codex,
   hermes, and others). Install for the agent I actually use.
3. CRITICAL: before installing, resolve the real agent home directory
   (on Windows it is usually AppData\Local\hermes, not ~/.hermes). Set
   the HOME env var the installer expects and verify the plugin files
   landed in the right place.
4. Prove the rewrite works: run one real command through the proxy and
   show the compacted output vs the raw output.
Rules: fail open means never break the original command. No secrets.
No em dashes. Verify with real output.

Prompt 3: Build a packet (deploy unit) from an existing setup

Build a "packet" for my setup so another machine can install it.
1. Create a folder with:
   - packet.yaml manifest (id, version, kind, privacy, requires,
     installs, touches, data_policy with sends: [] and verify list)
   - SKILL.md documenting the setup end to end
   - an idempotent install script (PowerShell for Windows, bash for
     POSIX) that checks prerequisites, backs up config before touching
     it, and skips anything already installed
   - a verify script with real probes that exits non-zero on failure
2. The install script must fail loud if required values are still
   placeholders (e.g. LLM model = "your-model"), listing the options.
3. Run the verify script and show the pass/fail output.
4. Write a registry entry with SHA-256 checksums of every file.
Ground rules: no secrets anywhere, ever. Idempotent (run twice =
same state). ASCII-only scripts. No em dashes. Real probes only.

Prompt 4: Make a friend-safe public version of a packet

Take my internal packet and produce a public flavor for distribution.
1. Copy it to a clean output directory.
2. Replace every owner identifier (paths, project names, machine names,
   personal handles) with generic placeholders (e.g. $HERMES_HOME,
   your-bank-id, your-project).
3. Rewrite the manifest: privacy: public, generic author, data_policy
   sends: [] preserved.
4. Run a secrets sweep over the output; if any credential-shaped string
   survives, fail with file:line.
5. Re-scan the output for the original identifier list; if ANY owner
   identifier survives, fail. A public packet that still contains
   internals must not ship.
6. Show me the scrub report (what was replaced) and confirm exit 0.
The fail-closed guard is the deliverable. No telemetry. No em dashes.

A note on review discipline When a coding agent reports a phase done, review the diff yourself, run the one lightweight live check (one curl against the real service), and verify any security claim independently. "It printed PASS" is not proof. The fail-closed tools are the proof.

06Lessons from the Trenches

Everything below cost us real time. Steal the lesson, skip the pain.

1. Never put live credentials in docs that can move We found a root password in plaintext in a reference file inside a skills tree that syncs to a VM and can be handed to other agents. Real passwords, API keys, client secrets: they were all sitting in markdown. The fix pattern: replace with an environment variable reference, store the value in a local env file that never syncs, and run a credential-shape sweep as a build gate. Treat any doc that can be copied as public.

2. Fail-closed tools, or the guard is theater A scrubber that reports PASS while a destination folder was locked and stale files remained was worse than no scrubber. The fix: explicit error handling, then assert the post-condition (destination actually gone, file list matches exactly), then a fail guard that re-scans and exits non-zero on any surviving match. If the tool can exit 0 on failure, it is lying.

3. Windows path mangling is a silent killer MSYS (git-bash) rewrites paths when calling native Windows binaries. An anchored Docker filter like name=^/hindsight$ arrives mangled and matches nothing. A native openssl cannot open a /tmp path. Symptoms look like script bugs. Scope path-conversion overrides narrowly (a docker wrapper), never globally, and pass Windows-style paths to native tools.

4. Environment pollution between test runs is real A tool that exports a variable globally during testing leaves it in the persistent shell. Next run, a completely different tool fails mysteriously (a broken python shim, a wrong HERMES_HOME). Check your environment before debugging your scripts.

5. ASCII-only scripts on Windows Windows PowerShell 5.1 parses .ps1 files as ANSI unless there is a UTF-8 BOM. An em dash or smart quote in a string silently breaks parsing with a confusing error. Keep script files ASCII-only.

6. Idempotency is a feature, not a nicety Install scripts that skip what already exists and back up config before touching it let you run them twice with zero drift. That is what makes a "packet" safe to hand over.

7. Verify with real probes A verification script that checks the container is up, the API answers, a retain actually runs through the LLM, and the MCP client connects is worth ten times a script that prints "installation complete".

07Resources and Further Reading

Hindsight (the memory server): github.com/vectorize-io/hindsight and docs at hindsight.vectorize.io
RTK (the token saver): github.com/rtk-ai/rtk
Agent Skills standard (the portable skill format packets build on): agentskills.io
MCP (the tool protocol everything plugs into): modelcontextprotocol.io
The original viral thread that started the token-saver wave: the "10M tokens saved" post on the Claude AI subreddit.

All of the tools above are open source. The whole path in this guide costs nothing in software. Your only real costs are the LLM tokens you choose to spend and the disk space for models.