Persistence of Memory
Architecture deep dive
Mission: turn every enterprise event into AI-curated semantic memory — indexed, archived, and searchable for a lifetime.
-
Health Status (JSON) ↗
Returns the status and embedding provider.
-
Interactive Swagger API Docs ↗
Test endpoints (search, tables, ingest) from your browser.
- ReDoc Documentation ↗
Data Lake + Persistence of Memory
Enterprise AI Memory Fabric — how operational events become durable AI memory and auditable archive.
Architecture diagram
Converted from 2026.06.16.Data Lake Persistence of Memory Diagram.pdf
End-to-end flow: transactional ingest → LLM consolidation → dual-write to hot synapses and cold archive → retrieval for agents and humans.
Diagram — flow summary
Ingest
Curate
Dual-write: consolidation output is written to both stores simultaneously.
Store + use
The vision
Extracted from 2026.06.15.Data Lake + Persistence of Memory- The Vision.pdf
Persistence of Memory is the AI-first infrastructure for giving websites, internal systems, and applications long-term memory, stored in the Data Lake and intelligently cataloged by a vector database.
It enables AI systems to remember, learn from experience, and continuously improve — with much of that knowledge being self-generated by the AI itself.
Core architecture
MySQL Aurora
The transactional source of truth. Maintains three to five years of current, useful business data and houses the Heartbeat, marketing site, and Pulse databases.
LanceDB (on EFS)
The persistent, intelligent “synapses” of the system. Stores Master Contexts, semantic relationships, embeddings for similarity search, and rich derived insights the AI has generated over time. This is where the real intelligence lives.
S3 Data Lake
The long-term governed archive: one JSON document per event (transactional payload + AI-curated memory), partitioned under archive/YYYY/MM/DD/. Operational audit lives in MySQL (pipeline_log, pipeline_error_log, data_lake_queue).
The orchestrator
A PHP-based layer sits in the middle and intelligently coordinates:
- Receives events and data from applications
- Submits relevant information to LLMs (Grok, Claude, etc.) for cataloging, summarization, and insight generation
- Updates LanceDB with curated memories
- Provides clean APIs back to applications (e.g., briefing, next best action)
This creates a living, self-improving system that gets smarter over time.
Audit trail (logs for everything)
Every event leaves a trace in MySQL before it ever hits cold storage:
data_lake_queue— inbound deposit, status (pending → picked → done/dead)pipeline_log— full pipeline run: payload, status, LLM provider, timing, summarypipeline_error_log— tier failures and hard errors with provider/model context
The Python pipeline service also writes structured stage logs (retrieval, LLM, LanceDB, archive) to stdout for ops debugging.
Data flow & archiving
New data starts in MySQL Aurora, flows through the Orchestrator for intelligent LLM processing, and lands in LanceDB (active semantic memory) and the S3 Data Lake (long-term archive). Older data is automatically archived from Aurora into the S3 Data Lake based on defined policies.
This architecture provides fast transactional access, rich semantic memory for AI, and cost-effective, vast, long-term storage.
POC mapping
How this demo site maps to the vision documents
Inbound Queue(s)
Classic async deposit queue (data_lake_queue)
Data Pipeline (Python, LLM)
Factory pipeline log + transactional ingest
Data Store (LanceDB)
LanceDB vector memories (hot recall)
Data Lake (S3)
Partitioned JSON data lake (cold truth)