Teaching Himiko to remember things that aren't text — videos, images, audio, and documents.
Ogham stores text memories brilliantly. But Himiko's world isn't just text:
• Client meetings are recorded as video
• Designs and mockups are images
• Contracts and proposals are PDFs
• Voice memos from Telegram are audio
• Tutorial videos from the YouTube research library are video
Without multimodal memory, these are invisible. Himiko can't search them, can't reference them, can't connect them to text memories. With it, one search finds everything — the text note AND the meeting recording AND the PDF clause.
What happens when you drop a file into the system:
You drop a video, image, PDF, or audio file. Could be a local file, a URL, or pasted content.
For video: detects scene changes, extracts key frames, chunks long recordings into 2-minute segments with overlap. For images: normalises quality, crops if needed. Text and audio skip this step.
Why OpenCV? It's free, runs locally (no API calls), and is the industry standard for video/image processing. Deterministic — same input always gives same output.
OpenCV 4.13 FFmpeg 8.0 Local processingTwo things happen:
Gemini Embedding 2 turns the content into a 768-dimension vector — its mathematical fingerprint. This is what makes it searchable. A video about "client pricing discussion" will be near other memories about pricing, even if the words are different.
Gemini Flash generates a text description: "2-minute clip from Meridian meeting. Discussion about Q3 pricing tier changes. Jo proposed 15% increase, client pushed back on timeline." This description is what Himiko reads when reasoning — she doesn't re-watch the video every time.
gemini-embedding-2-preview gemini-flashThe vector + text description + file reference go into the media_memories table in Supabase (same database as text memories, separate table). Ogham's knowledge graph auto-links it to related memories — a meeting recording auto-links to notes about the same client.
When patterns emerge — "the last 3 client meetings all raised the same concern" — the insight gets promoted to a wiki page. The wiki cross-references it with existing knowledge. Over time, the system doesn't just store files; it builds understanding.
Karpathy Wiki /wiki promoteThe naive approach: embed a video, store the vector, search retrieves the video, hand Himiko an MP4. Problem: Himiko can't reason about raw video. She'd need to re-process it every time.
Our approach: at ingestion time, Gemini Flash watches the video once and writes a detailed text description. That description lives permanently alongside the vector. When Himiko finds the memory, she reads the text description to think, and references the video clip as supporting evidence.
The expensive work (video → text) happens once. Every future search is cheap — just finding the vector and reading the cached description.
Text memories stay on Voyage AI (512 dimensions) in the existing memories table. Multimodal content goes into a new media_memories table with Gemini embeddings (768 dimensions). Same Supabase database, separate tables.
Why: Zero risk to existing text memories. No migration needed. Ogham keeps working exactly as it does. We prove multimodal works, then merge everything into one table later when the value is clear.
What "merge later" means: Re-embed all existing text memories with Gemini (one-time batch job), switch Ogham's embedding provider, and put everything in one table. One search then finds text AND media together. But only when we're confident the multimodal pipeline is solid.
| # | Step | Status |
|---|---|---|
| 1 | Gemini API key | Done ✓ |
| 2 | OpenCV installed | Done ✓ |
| 3 | FFmpeg installed | Done ✓ |
| 4 | Create media_memories table in Supabase | Next → |
| 5 | Build ingestion tool (tools/ingest_media.py) | Pending |
| 6 | Build search tool (tools/search_media.py) | Pending |
| 7 | Wire into Ogham or as parallel MCP tool | Pending |
| 8 | Test with real content | Pending |
| 9 | Wiki promotion for multimedia insights | Pending |