Wally - Virtual Tour Guide

Project Description — Wally: An Agentic Virtual Tour Guide for the Louvre

Wally is an AI museum tour guide that lets visitors learn about the Louvre items through a natural, conversational experience. Using an ElevenLabs real-time voice agent, RAG-augmented knowledge retrieval, and an interactive Anam avatar, Wally automatically detects BLE sensors at each exhibit and responds to spoken questions, automatically navigates through artworks, and delivers personalized descriptions, and stories for the various demographics and user types. Visitors can ask anything—from “Who painted this?” to “Take me to a Romanticism tour”—and Wally dynamically orchestrates tool calls to fetch general louvre information, artwork metadata, thematic insights
The agent solves the problem of museum overwhelm—most visitors lack context, and understanding when approaching world-class art. Wally provides a hands-free, multimodal, real-time assistant that explains, narrates, and guides like a trained curator.

Core flow:
1.⁠ ⁠User speaks to Wally through an ElevenLabs voice agent, with an optional Anam Avatar.
2.⁠ ⁠Wally parses the request, retrieves real artwork data from a custom RAG index, and calls the relevant agent.
3.⁠ ⁠Wally responds with natural speech + avatar animation, optionally recommending nearby artworks.
4.⁠ ⁠End-to-end loop runs in real time, demonstrating multimodality, and agentic planning.

The submission runs fully in the browser: load the webpage → the Anam avatar appears → ElevenLabs’ conversational agent begins streaming → user prompts → RAG lookup + tool calls → spoken curator-style guidance.

1.⁠ ⁠Working Prototype
•⁠ ⁠Real-time voice conversation via ElevenLabs agent API
•⁠ ⁠Live RAG retrieval from a curated Louvre knowledge base
•⁠ ⁠Tool calls to transfer across multiple agents on ElevenLabs (Wally - art expert, James- Museum manager)
•⁠ ⁠Anam avatar streaming responses, lip-synced to generated voice
•⁠ ⁠Browser-based demo with no manual steps required, hosted on Cloudflare with a live demo/link

2.⁠ ⁠Technical Complexity & Integration
•⁠ ⁠ElevenLabs real-time conversational agent (streaming ASR + TTS)

• ElevenLabs knowledge base for RAG
  •⁠ ⁠Anam avatar for animated visual presence
  •⁠ ⁠Real-time browser event handling
  •⁠ Gemini Flash model and ElevenLab Turbo model for low latency response
  3.⁠ ⁠Innovation & Creativity
  •⁠ ⁠A virtual “personality” that guides, narrates, explains, and adapts
  •⁠ ⁠Agentic reasoning to offer personalized tours
  •⁠ ⁠Blends voice, avatar presence, and knowledge retrieval in a novel way
  •⁠ ⁠Provides museum visitors with an experience previously available only with expert human guides
  4.⁠ ⁠Real-World Impact
  •⁠ ⁠Makes world-class art accessible, even to people with no art background
  •⁠ ⁠Removes barriers for low-vision or language-diverse visitors through voice interaction
  •⁠ ⁠Enables scalable museum engagement without human staffing requirements
  •⁠ ⁠Could power similar guides for global institutions (Met, British Museum, V&A, etc.)
  5.⁠ ⁠Theme Alignment
  •⁠ ⁠A conversational agent, embodied via voice + avatar
  •⁠ ⁠Multi-agent reasoning (navigation + interpretation + RAG retrieval)
  •⁠ ⁠Deep integration with partner technologies (ElevenLabs + browser + LLM tools)
  •⁠ ⁠Autonomous yet controlled flow demonstrating tool use and orchestration
  ______________
  Technologies, Frameworks, APIs, and Model Providers
  Core
  •⁠ ⁠Gemini Flash models for agent reasoning and tool orchestration
  •⁠ ⁠ElevenLabs Conversational AI API (Streaming ASR + TTS)
  •⁠ ⁠Anam Avatar SDK for real-time animated guide
  •⁠ ⁠Node.js
  •⁠ ⁠Supabase image storage
  •⁠ ⁠Vite frontend for browser interactions
  •⁠ ⁠RAG pipeline

N/A