Multimodal Agentic RAG

This is a multimodal RAG app built with Gemini Embedding 2 and Google ADK. Add text, URLs, PDFs, images, audio, or video; ask a question; and get a grounded answer with clear citations.

The UI includes a 3D embedding view for inspecting the search space. Each source appears as one point. When you ask a question, the query is projected into the same space and the cited sources are highlighted.

What It Does

• Adds and removes multimodal sources from a local in-memory index.
• Uses Gemini Embedding 2 for source and query embeddings.
• Requires GOOGLE_API_KEY; the app does not use local vector or answer fallbacks.
• Retrieves evidence with cosine similarity over the stored embeddings.
• Runs a Google ADK agent to coordinate answer generation from the retrieved context.
• Shows citations separately from the answer text so citation IDs do not clutter the response.
• Projects source and query vectors into a 3D PCA view for inspection.

Architecture

The important implementation detail is that /ask performs retrieval once and passes that same retrieval packet into the ADK answer flow. The answer and the citation panel are therefore based on the same ranked evidence.

Project Structure

rag_tutorials/multimodal_agentic_rag/
|-- README.md
|-- assets/
|   `-- multimodal-agentic-rag-architecture.png
|-- backend/
|   |-- app_state.py
|   |-- rag_store.py
|   |-- requirements.txt
|   |-- server.py
|   `-- agentic_rag_agent/
|       |-- __init__.py
|       `-- agent.py
`-- frontend/
    |-- index.html
    |-- package.json
    |-- src/
    |   |-- App.tsx
    |   |-- main.tsx
    |   `-- styles.css
    |-- tsconfig.json
    `-- vite.config.ts

Run Locally

Start the backend:

cd rag_tutorials/multimodal_agentic_rag/backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
export GOOGLE_API_KEY="your-google-ai-studio-key"
python server.py

The backend runs at:

http://localhost:8897

Start the frontend in another terminal:

cd rag_tutorials/multimodal_agentic_rag/frontend
npm install
npm run dev -- --port 5177

The frontend runs at:

http://localhost:5177

If the backend is on a different port:

VITE_API_URL=http://localhost:8897 npm run dev -- --port 5177

Try It

1. Open http://localhost:5177.
2. Add a text, URL, PDF, image, audio, or video source.
3. Ask a question in the Q&A panel.
4. Review the answer and citations.
5. Inspect the source and query points in the embedding view.

API

Notes

• Storage is in memory. Restarting the backend resets the demo index.
• URL ingestion blocks localhost and private IP ranges unless ALLOW_PRIVATE_URLS=true is set.
• Media files uploaded through the Gemini File API are cleaned up after embedding.
• Blocking media processing runs in a threadpool so the FastAPI event loop is not held.
• For production, replace the in-memory store with durable storage and add authentication, background ingestion, evals, observability, and a managed vector database.