In one sentence. Persistence of Memory is a governed memory plane for websites and agents: sites send curated events in; the system embeds and stores them; later cycles retrieve only what belongs in that job’s pool — so the next action is wiser without becoming a junk drawer of every token the model ever saw.

Two audiences share this page on purpose. If you live in terminals, the stack sections name the real systems and where they sit. If you care about product economics more than process tables: this is how a site stops renting amnesia from a chat API and starts owning a compounding memory asset — isolated by purpose, auditable by operators, and hard to confuse with “we bolted a chatbot onto the footer.”

Plain English first

Most “AI websites” call a large language model, get a fluent answer, and throw away the episode. The model is rented intelligence with a short working memory (the context window). Tomorrow the site is brilliant and blank.

Persistence of Memory adds the missing layer: durable, selective memory owned by the operator. Think of three kinds of storage working together:

  • Semantic memory — “find things like this idea” (vector search). That is LanceDB.
  • Operational memory — queues, status, who deposited what, pipeline truth. That is MySQL (or SQLite locally).
  • Archival memory — the long, boring, invaluable JSON trail of what actually happened. That is the data lake on disk (S3-shaped paths; object storage later if needed).

Money guys can stop at: we separate “what feels similar,” “what the system is doing,” and “what we can prove later.” Nerds keep reading.

The operating loop

This is the product cycle, not a marketing slogan. Every AI-first site we care about runs some version of it:

  1. Constitution — voice, boundaries, north star. Version-stamped into prompts and deposits so “what we were trying to be” is reconstructable.
  2. Context — CMS copy, analytics, leads, calendars, prior run metadata. The boring world the agent must respect.
  3. Recall — semantic search inside the right memory pool. Not the whole universe. Not another tenant’s lineage.
  4. Action — publish, deliver, report, schedule, reply. Object-level work by an LLM or deterministic code.
  5. Outcome — lead, booking, open, skip, bounce, failure. Reality, not vibes.
  6. Deposit — curated event → optional consolidate → embed → synapse. Selection at the door is the gate.
  7. Better recall — the next cycle starts with evidence, not amnesia.

Stuffing every log line into the next prompt is not this loop. That is context pollution with a credit-card bill.

Components of the stack

Working software today — not a slide deck. Each piece has a job; none of them are “the AI.” The models do cognition. This stack does metamemory: what gets stored, where, and what may be retrieved.

LanceDB — semantic synapses

What it is. An open vector / multimodal database built for AI retrieval. Docs: docs.lancedb.com. We use it as the home of synapses: text (or consolidated summaries) plus dense embedding vectors, partitioned into pools (tables) per job and lineage.

Why it matters. Relational SQL is excellent at exact filters (“all events from site X last Tuesday”). It is clumsy at “find memories that mean something like this paragraph.” Vector similarity search answers the second question. LanceDB keeps that search local to our host — we are not shipping every recall query to a third-party memory SaaS as the system of record.

For operators. This is the “intuition shelf”: compact, searchable experience the agent can pull when it needs precedent — not the full legal archive.

Embeddings — turning meaning into numbers

Before LanceDB can search “by meaning,” text becomes a vector (a long list of floating-point numbers). Nearby vectors ≈ related ideas. We generate embeddings locally in the Python service (e.g. sentence-transformer style models via the Hugging Face ecosystem — see Sentence Transformers / Hugging Face) so encoding stays under our control on the memory host.

For operators. Embedding is the translation step from human language into a form machines can index by similarity. Bad deposits in → noisy recall out. Gating matters more than model fashion.

MySQL — orchestration truth

What it is. A battle-tested relational database. Locally we may use SQLite for the same schema while developing; production prefers MySQL with fail-hard configuration so we never silently drift onto a toy store.

Why it matters. Queues, pipeline status, admin users, deposit metadata, “did this ingest finish?” — these want transactions, indexes, and boring SQL. Vectors do not replace ledgers. MySQL is the control plane’s source of process truth beside LanceDB’s semantic index.

For operators. When something breaks at 2 a.m., you want rows and statuses — not a mysterious similarity score.

JSON data lake — long-term evidence

What it is. Filesystem (and later, if needed, object storage) archives with S3-shaped paths such as archive/YYYY/MM/DD/{uuid}.json. Env names still say S3_LAKE_PATH / DATA_LAKE_PATH because the layout is intentionally cloud-portable.

Why it matters. Synapses are curated and compressible. The lake keeps fuller episode evidence for audit, reprocessing, training later, or “what exactly did we store?” without bloating the hot vector index. On a Linode, growth path is local disk → Block Storage; Object Storage when the economics say so.

For operators. This is the filing cabinet. Cheap to keep. Expensive to lack when compliance or debugging shows up.

Python + FastAPI — memory service

What it is. A Python HTTP API (FastAPI) that owns ingest, embed, LanceDB insert/search, and lake writes. Bearer-token authenticated. In production it binds to loopback (127.0.0.1) — not the public internet.

Why it matters. The ML-adjacent work (embeddings, LanceDB SDK, pipeline glue) lives naturally in Python. FastAPI gives typed routes, OpenAPI when debug is on, and a clean service boundary so PHP and other site stacks do not embed vector math.

Key surfaces (conceptual). POST /events/ingest for deposits; table search/insert for pools; health for operators. Public docs stay off unless debugging.

For operators. Think “database appliance with an HTTP face,” not a public product API. Sites talk to the orchestrated edge over HTTPS; the vector engine stays behind the door.

PHP orchestrator — control plane & console

What it is. A PHP application (Slim-style routing, Twig templates, nginx in front) that operators actually use: login, admin overview, system metrics, pipeline visibility, public vision pages. Official PHP: php.net. Web server: nginx.

Why it matters. Portfolio sites (and many production CMS worlds) already speak PHP. The orchestrator is the human-facing and site-facing control plane: auth, rate limits, security headers, calling the FastAPI service with the synapse token, rendering dashboards. It is also honest about ops — disk, DB size, uptime — because memory systems fail as sysadmin problems before they fail as philosophy problems.

For operators. PHP here is not nostalgia. It is the grown-up front office: accounts, pages, admin, HTTPS. Python is the back office that touches vectors.

Object-level cognition — Grok, Claude, and friends

What it is. Large language models do the fluent work: consolidate a noisy event into a deposit-worthy summary, draft copy inside a constitution, classify an outcome. We use cloud models where they fit — e.g. xAI / Grok, Anthropic / Claude — and keep local models optional for hosts that can afford them. A small PoM box does not need to run a full local LLM to be useful; memory and gating still compound.

Why it matters. Models are interchangeable cognition. Memory pools and constitutions are the scarce, owned advantage. Swap the brain; keep the metamemory.

For operators. You rent the eloquent intern. You own the filing system that makes the intern less amnesiac next week.

Cursor — agentic build loop

What it is. An AI-native development environment used to design, implement, and harden this stack with a human still owning gates, constitutions, and production judgment.

Why it matters. PoM is not only a runtime for sites; it is built the way AI-first software is increasingly built: tight human–agent loops, real repos, real deploys. The same philosophy applies: persist what matters, select what enters the next context, do not confuse fluency with architecture.

Write path

Depositing memory is deliberately gated. Rough flow:

  1. A site or agent prepares a curated event (not a raw firehose of PII).
  2. HTTPS to the orchestrator / allowed ingest edge → internal POST /events/ingest on the FastAPI service.
  3. MySQL records queue / pipeline state.
  4. Optional LLM consolidate: compress and shape the deposit under the pool’s constitution.
  5. Embed → insert synapse into the correct LanceDB table / pool.
  6. Archive a JSON artifact into the data lake for later proof and reprocess.

Selection at the door is metacognitive gating in code: should this become a synapse at all? Dumping everything is cheaper to implement and more expensive forever.

Read path

Recall is pool-scoped on purpose:

  1. Caller issues search against a named pool (conceptually POST /tables/{pool}/search with query text).
  2. Query text is embedded; LanceDB returns nearest neighbors in that table.
  3. Compact recall snippets are injected into the next prompt cycle — enough to orient, not enough to drown.

There is no “search the entire company brain by default.” That is how marketing memory contaminates recovery voice, and how multi-tenant dreams become liability.

Isolation & constitutions

One engine. Many lineages. Pools are separated by LanceDB table and source_db (and related metadata). A daily-recovery pool does not share synapses with a marketing conversion pool. Humans still approve what must stay sacred; machines ship and remember what policy allows.

A constitution is the versioned goal-and-boundary document for a pool: tone, do-nots, success definition. It is stamped into cognition and into deposits so later recall knows under which charter a memory was formed. Without constitutions, “memory” is just a smarter junk drawer.

Security boundary

Short version for both audiences:

  • Public HTTPS terminates at nginx → PHP.
  • FastAPI / LanceDB stay on loopback with a shared secret (Bearer token); not exposed as an open port on the internet.
  • Admin console is authenticated; public pages are marketing and thesis, not the database.
  • Production prefers MySQL with explicit require flags — no silent fallback to an empty local file pretending to be prod.

Deeper diagrams and research framing live on Architecture and Vision. The psychological map is on Metacognition.

Deep architecture Vision essay Metacognition Essays