Virtual Human Toolkit
Any Character, Any Technology, Anywhere
Create conversational 3D characters for web, mobile, AR/VR, and desktop
The Virtual Human Toolkit (VHToolkit) is a research and development platform from the USC Institute for Creative Technologies for creating embodied conversational agents / socially intelligent agents: real-time, interactive characters that perceive real humans and respond both verbally and nonverbally. The VHToolkit is available on GitHub under a non-commercial license.
Try it out!
Go to the live web demo and talk to a virtual human in your browser right now.
Download the source code from GitHub, and binaries from its Releases page.
Features
Integrated system
Clone, open in Unity, add your API keys, and press Play to converse with a character
Vendor agnostic
Native support for 20+ vendors and technologies across audio-visual sensing, speech recognition, language models, text-to-speech, and nonverbal behavior generation
Knowledge grounding
Plain text files become reference material through RAG (Retrieval-Augmented Generation). Keyword retrieval requires no API key or network access; optional OpenAI embeddings add semantic matching.
Cross-platform deployment
VHToolkit supports Windows, macOS, Linux, WebGL (browser), Android, iOS, and Quest (AR/VR)
Custom characters
In addition to the included character library, you can author your own characters and personalized avatars with Reallusion Character Creator (requires a separate license)
Cloud, local, and custom providers
Select between cloud and local AI providers during setup or runtime; implement the principled API to add your own technology
Supported AI Services
| Capability | Cloud | Local / on-prem |
| Language models (NLP) | Anthropic Claude, AWS Lex, OpenAI ChatGPT, Google Gemini | Ollama, Rasa, vLLM (any OpenAI-compatible endpoint) |
| Speech recognition (ASR) | Azure Speech, Google Gemini, OpenAI Realtime | Faster-Whisper, platform-native (Windows/Android/iOS) |
| Speech synthesis (TTS) | ElevenLabs (incl. voice cloning), AWS Polly, Azure, Google Gemini | Kokoro, Piper, XTTS v2 |
| Knowledge grounding (RAG) | OpenAI semantic embeddings | Unity-embedded lexical, hybrid |
| Sensing | AWS Rekognition, Azure Face (requires MS approval) | DeepFace, OpenFace |
All cloud services use your own accounts and API keys. Note that not all technologies are supported for all hardware platforms.
Mobile development requires 3rd party Unity packages. WebGL development requires custom AWS Lambdas due to CORS requirements.
Why Us
Built on RIDE
The VHToolkit is powered by RIDE (Rapid Integration & Development Environment), USC ICT's rapid prototyping modeling and simulation middleware platform. RIDE provides the system architecture, provider abstractions, web and local service integrations, and APIs. The VHToolkit adds the virtual human layer: characters, the conversation loop, and examples for all hardware platforms.
Publications
- Hartholt, A., Fast, E., Leeds, A., Mozgai, S. (2026). "Demonstrating the Open Virtual Human Toolkit: Any Character, Any Technology, Anywhere." ACM International Conference on Intelligent Virtual Agents (IVA). (To be published)
- Hartholt, A., Fast, E., Li, Z., Kim, K., Leeds, A., Mozgai, S. (2022). "Re-architecting the Virtual Human Toolkit: Towards an Interoperable Platform for Embodied Conversational Agent Research and Development." 22nd ACM International Conference on Intelligent Virtual Agents (IVA). https://dl.acm.org/doi/10.1145/3514197.3549671
- Hartholt, A., Traum, D., Marsella, S. C., Shapiro, A., Stratou, G., Leuski, A., Morency, L.-P., Gratch, J. (2013). "All Together Now: Introducing the Virtual Human Toolkit." International Workshop on Intelligent Virtual Agents (IVA). https://link.springer.com/chapter/10.1007/978-3-642-40415-3_33