I was walking the dog and listening to a podcast (as you do) when the topic of Anthropic’s recent entanglements with the Pentagon came up. I half-remembered something Dario Amodei said in a recent Dwarkesh Patel episode, drawing an equivalence between AI capabilities and nuclear weapons. I couldn’t remember the details. I remembered his gestures. Of course, there’s no way to search for that moment unless you go back to YouTube and scrub through a 2.5-hour video. You can search a transcript, but a transcript doesn’t know about body language.
Google released Gemini Embedding 2 this week, and it might change that.
So here’s the background. Most AI applications solve the recall problem using RAG (Retrieval Augmented Generation): index your data, let an LLM answer questions about it. We’ve built dozens of these pipelines at Jeavio. They work well for text. But if you want to search a podcast or a video, you first have to transcribe it, then index the transcription. And transcription is lossy. Tone, facial expressions, posture, the visual context of a conversation: none of that survives the conversion.
This is the constraint we’ve been designing around without really questioning it. Most embedding models only understand text.
Gemini Embedding 2 is Google’s first natively multimodal embedding model. It maps text, images, video, audio, and documents into a single embedding space, meaning a text query and a video frame can be compared directly because they live in the same mathematical coordinate system. Multimodal embeddings aren’t new. OpenAI’s CLIP has been around since 2021, and Meta’s ImageBind handles six modalities. But those approaches pair separate encoders (one for vision, one for text) and align them after the fact. Gemini Embedding 2 is built on the Gemini foundation model itself: the cross-modal understanding happens inside the network’s intermediate layers rather than being stitched together at the end. The difference is architectural, and it matters for retrieval quality.
Back to that Dario Amodei moment. Today, I can ask a RAG pipeline “What is Amodei’s opinion on AI job losses?” and get a solid answer from the transcript. But I can’t ask “Was he nervous when the Pentagon question came up?” A grimace, a stiff posture, a long pause before answering: these are data points that a text-only embedding simply can’t represent. A natively multimodal embedding can, because it processes video and audio directly. (The practical constraint: video input is currently limited to 120 seconds per request, so a three-hour podcast needs to be chunked. The use case holds, but the plumbing isn’t trivial.)
The applications stretch well beyond podcast search. Voice queries against video libraries. Finding the moment in a deposition where a witness’s tone shifts even though their words stay measured. Correlating images, audio, and text in a single index. And then … the uncomfortable ones. Surveillance systems that match faces, voices, and written communications in a unified semantic space. Personal photos correlated with social media posts and location data. When all modalities live in the same mathematical neighborhood, the distance between “powerful search” and “invasive profiling” gets very thin.
Embeddings are the load-bearing infrastructure of most AI experiences. We’ve been building around a text-only constraint for so long that it felt permanent. It isn’t. The applications and the policy questions are going to arrive together, and I’m not sure most teams are ready for either.