Main Page
From ICT Virtual Human Toolkit
The ICT Virtual Human Toolkit is a collection of modules, tools and libraries that allow users, authors and developers to create their own virtual humans. This software is being developed at the University of Southern California Institute for Creative Technologies and can be licensed without cost for academic research purposes.
Contents |
Download
Click here to download the Virtual Human Toolkit.
News
- Mar 2 2012 - A minor release of the toolkit is now available, updating the Unity version to 3.5 as well as providing incremental changes to the Unity/SmartBody debug tools in the Unity Editor (VH menu in Unity).
- Dec 22 2011 - Happy holidays! The latest release of the toolkit includes the ability to interrupt Brad, improved support for higher resolutions, and a fix for text-to-speech not working properly. See Release Notes for details.
- Aug 10 2011 - A new version of the toolkit has been released, offering support for the free version of Unity 3D, meaning you can now create your own scenes for Brad. You can download Unity here: http://www.unity3d.com. For instructions on how to use the Unity 3D Editor with the toolkit, see the Unity section. In addition, the user interaction has been improved; Unity now launches in full-screen automatically and users get visual feedback when talking to Brad. To directly talk to Brad, first make sure you have a microphone plugged in, wait for Brad to finish his introduction, and close the tips window. Now click and hold the left mouse button while asking Brad a question; release the mouse button when you are done talking. The recognized result will be displayed above Brad in white font (toggle on/off with the O key), and Brad will answer your question. It is adviced to update Java and ActiveMQ, which are provided with the 3rd party installer versions.For a full list of improvements, changes and fixes, see the Release Notes.
- May 4 2011 - We have released a new version of the toolkit, which adds prerecorded speech for Brad. This means Brad now has a human voice, instead of one of the various text-to-speech voices. You can now also interact with Brad through the Unity 3D renderer directly, by either typing in text or by speaking through a microphone. Please note that as of this release, Unity 3D is the default and only supported renderer. Gamebryo is deprecated and will not receive future updates. Ogre is an open source renderer which is provided as a basic example only. For more details on this release, see the Release Notes. If you have any questions and or comments, please use the toolkit Google Groups.
- Mar 28 2011 - Version 0.9.12.258 has been released, adding, amongst other things, subtitles, the capability for Brad to walk around, and smaller installers. If you are upgrading from an older version, please see Release Notes in order to ensure correct functionality.
- Feb 25 2011 - Release 0.9.10.201 of the toolkit is now available here. This release adds a Unity 3D renderer version, as well as a different default voice for Brad. For more details, see Release Notes. For feedback, see the toolkit Google Groups.
- Oct 8 2010 - We just released version 0.9.9.137 of the toolkit, which adds the capability to create custom language models. Several critical bugs have been fixed as well.
- Sep 17 2010 - Version 0.9.7.107 of the toolkit has been released, adding support for the Festival text to speech engine.
Toolkit Overview
Goal
The University of Southern California Institute for Creative Technologies (ICT) has created a Virtual Human Toolkit with the goal of reducing some the complexity inherent in creating virtual humans. Our toolkit is an ever-growing collection of innovative technologies, fueled by basic research performed at the ICT and its partners. The toolkit provides a solid technical foundation and modularity that allows a relatively easy way of mixing and matching toolkit technology with a research project's proprietary or 3rd-party software. Through this toolkit, ICT hopes to provide the virtual humans research community with a widely accepted platform on which new technologies can be built.
What Is It
The ICT Virtual Human Toolkit is a collection of modules, tools and libraries that supports the creation of virtual human conversational characters. At the core of the toolkit lies innovative, research-driven technologies which are combined with other software components in order to provide a complete embodied conversational agent. Since all ICT virtual human software is built on top of a common framework, as part of a modular architecture, researchers using the toolkit can do any of the following:
- utilize all components or a subset thereof;
- utilize certain components while replacing others with non-toolkit components;
- utilize certain components in other existing systems.
The technology emphasizes natural language interaction, nonverbal behavior and visual recognition. The main modules are:
- Non Player Character Editor (NPCEditor), a package for creating dialogue responses to inputs for one or more characters. It contains a text classifier based on cross-language relevance models that selects a character's response based on the user's text input, as well as an authoring interface to input and relate questions and answers, and a simple dialogue manager to control aspects of output behavior.
- Nonverbal Behavior Generator (NVBG), a rule-based behavior planner that generates behaviors by inferring communicative functions from a surface text and selects behaviors to augment and complement the expression of those functions.
- SmartBody, a character animation platform that provides locomotion, steering, object manipulation, lip syncing, gazing and nonverbal behavior in real time, using the Behavior Markup Language.
- Watson, a real-time visual feedback recognition library for interactive interfaces that can recognize head gaze, head gestures, eye gaze and eye gestures using the images of a monocular or stereo camera.
- Speech Client (AcquireSpeech), a tool that can send audio to one or more speech recognizers and relay the information to the rest of the system. It also allows text to be typed into the system, simulating speech input. The toolkit uses PocketSphinx as a 3rd party speech recognition solution.
The target platform for the overall toolkit is Microsoft Windows, although some components are multi-platform.
What Is It Not
Although the toolkit supports virtual humans development, some components are prototypes rather than state-of-the-art technologies. The Components section lists several potential alternatives for some components.
The toolkit does not contain many of the current basic research technologies at ICT, such as the reasoning SASO agents. Most of the toolkit technology, however, is the result of basic research, which is continually evaluated for potential use in future releases.
Currently, we are not at liberty to publicly distribute any project-specific data. However, interested parties are encouraged to contact us directly. In addition, we are considering creating a forum where users can share their creations.
Who Can Use It
The toolkit has three target audiences:
- Users, who can run any of the modules without any modifications. A simple example character, Brad, is included for everyone to interact with.
- Authors, who can create their own virtual human characters using the provided software. Authors can modify the provided Brad character, or create their own virtual human completely from scratch.
- Developers, who can either use the provided modules, tools and libraries in their own system or who can extend the components in those cases where they are open source.
All toolkit software can be used without cost for academic research purposes provided all associated licenses are being honored. If you are using the toolkit or any of its components for published research, please cite us appropriately, as per clause 3 of the license. See the Papers & Links section for more details. Please contact us if you are interested in a commercial license.
Where To Get It
Please see the Support section for instructions on how to obtain the ICT Virtual Human Toolkit. The Getting Started section below will guide you through the first steps of using the software.
License and Disclaimers
The complete License Agreement and supporting documentation can be read in the License section. The License Agreement states, but is not limited to:
- The toolkit and any of its components can only be used for academic research purposes.
- If you are using the toolkit or any of its components for published research, please cite us appropriately. See Papers & Links for details.
- Toolkit users are required to honor all licenses of components and supporting software as defined in Exhibit A of the License Agreement.
Please contact us if you are interested in a commercial license.
Please be aware that the toolkit consists of research software for which documentation and support is limited. However, both the software as well as the accompanying documentation are actively being developed and updated.
Current Toolkit Users
There are many ICT projects that use a subset of the technology provided with the toolkit. Below is a list of some examples:
- SASO
- Virtual Patient
- Sergeant Star
- Gunslinger
- Elect BiLat
- InterFaces Project (with Boston Museum of Science)
- Tactical Questioning
In addition, many groups outside of ICT use some of the toolkit components, most notably SmartBody and Watson:
- University of Reykjavik
- German Research Center for Artificial Intelligence
- ArticuLab at Northwestern University
- Telecom Paris Tech
- Affective Computing Research group at MIT Media Lab
- ICSI/UCB Vision Group at UC Berkeley
- Human-Centered, Intelligent, Human-Computer Interaction group at Imperial College
- Worcester Polytechnic Institute
- Microsoft Research
- Relational Agents group at Northeastern University
- Component Analysis Lab at Carnegie Mellon University
Getting Started
Please go to the Support page for instructions on how to obtain the toolkit. This page will also give you further guidance on how to install and run the provided scenario.
For navigation on this website, please use the menu on the left. Each of the listed sections is described below:
- Architecture - Gives an overview of the toolkit architecture, based on the ICT Virtual Human Architecture.
- Components - Lists all modules, tools and libraries that make up the toolkit and links to available documentation and third party enhancements.
- Tutorials - Lists all available tutorials, including how to run the provided examples and how to create your own virtual human.
- Projects - An overview of some projects that use technology included in the toolkit, see also below.
- FAQ - Frequently Asked Questions about the toolkit in general and all of its components in detail. Also contains a glossary for often used terms and acronyms.
- Support - An overview of available support.
- Papers & Links - List of links to related sites.
Users
Users can run all the needed components in order interact with Brad, the basic example character provided with the toolkit. After obtaining the toolkit, see the instructions on the Support page on how to install it. When the installer is done, you get the option to immediately start the Launcher. With the Launcher, it should be as simple as clicking the first Launch button (in the Run Checked row under Run It All), then quickly clicking OK on the Gamebryo settings windows, and waiting for about 30 seconds for all components to launch. Note that Gamebryo needs to be up and running before SmartBody can be launched. When using the Run All functionality, the Launcher will start Gamebryo and wait 15 seconds before loading all other modules. It this is not enough time, launch Gamebryo manually, uncheck Gamebryo in the Launcher and then Run All. When all the non-tool rows in the Launcher are green, it means you are ready to start interacting with Brad. Brad is a very basic character and shows off some, but not all, of the toolkit elements.
Many windows will have popped up, but the only ones you need right now are the Gamebryo window for the graphics, and AcquireSpeech to type in questions. In AcquireSpeech, go to the Player tab, type your question in the Text field, and click the associated Send button or hit enter. Brad should respond to your question by talking back. By default, Brad uses the Text-To-Speech voice that comes with Windows, so depending on your windows version, you might hear a very outdated computer voice, or a woman's voice. Brad is authored to answer all your general questions about the toolkit. Be aware that he only has a general overview of the toolkit; he does not serve as an interactive tutorial or tutor.
For more detailed instructions, including troubleshooting, on how to run the provided example, please see here.
Authors
Authors can create their own virtual humans. They should first get familiar with the basic example character provided with the toolkit: Brad. Brad shows off some of the toolkit elements, in particular natural language interaction and nonverbal behavior. The graphics are simple, the default voice is outdated, and vision and speech recognition are not integrated.
Once you are familiar with the Brad character, you can read up on the technology behind the Components and some of the Tutorials. The documentation of these is a work in progress.
Creating a character consists of various elements:
- Natural language, the NPCEditor. See below for some basic instructions.
- Nonverbal Behavior, the Nonverbal Behavior Generator. You can edit some of the language to behavior rules in C:\vhtoolkit\core\nvb_generator\NVBGenerator\xslt\rule_input_brad.xml. In this file, keywords are associated with certain animations.
- Animations, using a third party application like Maya or 3D Studio Max, in combination with SmartBody exporters.
- Character and background models, using a third party application like Maya or 3D Studio Max, in combination with Ogre exporters.
- Textures, using a third party application like Photoshop or Gimp.
Of these, the NPCEditor is the easiest to start with. In the Utterance tab you see questions a human user can ask on the left, and the possible answers on the right. Questions are linked to answers with a certain value from 1 to 6. Usually there are only links with value 6. When you select a question, all answers linked to that question turn green. You can select multiple questions and answers by holding control, thus creating sets of links. If sets are not mapped completely 1 to 1, elements from one set that are only partially linked to elements in the other set will be displayed yellow. If you want to create a 1 to 1 mapping between the sets, select all yellow rows on both sides and set the link value to 6, either by selecting it at the bottom or by using CTRL + 6. Similarly, you can make new links between questions and answers by selecting them and setting the link value. You can add either new questions or answers with the Add button. A new, empty line appears, which you have to select in order to fill in the text field. Be aware that on the answer side, the Compile / Script section might hide the text field. You can drag the compile section down and ignore it.
Try first to extend the Brad data file (called a plist), before you create a completely new character on your own. First, think of a question Brad cannot currently answer. Try out that question and confirm that Brad says something like "I don't know.". Then, add the new question on the left. All components can stay up for this; you can edit the plist in real time. Also create the answer on the right. Be sure to set the Speaker value to Brad. Now, select both question and answer, and make sure no other elements are selected. Hit CTRL + 6, to link them together, and save your file. Ask the same question again, and Brad should now give the answer you just created. From here on you can let your creativity flow. Have fun!
Developers
Developers can extend toolkit components, use toolkit components in their own systems, or use their own components within the toolkit. First, read up on the Architecture and Components, and read the tutorial on how to develop a new module. The documentation of these is a work in progress.
The toolkit architecture consists of modules communicating by message passing. The Messages section defines some of these. You can use the Logger to get more detailed and practical information. Replacing or using a module means adhering to the existing messaging interface.
If you are interested in extending the components within the toolkit, we would love to hear from you. See the Support page on how to contact us. Note that not all provided software is open source, and that not all open source software is accessible from a repository. SmartBody is a SourceForge project, accessible here.
