Modules

The ICT Virtual Human Toolkit is built upon a common modular architecture which enables users to utilize all modules as is, one or more modules coupled with proprietary components, or one or more modules in other existing systems. Our technology emphasizes natural language interaction, nonverbal behavior and perception. Its main modules are listed below. See Documentation for an overview of the architecture, the messaging API, and other components.

MultiSense

MultiSense is a multimodal sensing framework which is created as a platform to integrate and fuse sensor technologies and develop probabilistic models for human behavior recognition. MultiSense tracks and analyzes users’ facial expressions, body posture, acoustic features, linguistic patterns and higher-level behavior descriptors (e.g. attention, fidgeting). It uses the Perception Markup Language (PML).

NPCEditor

At the core of the NPCEditor is a statistical text classification algorithm that selects the character’s responses based on the user’s utterances. A character designer specifies a set of responses and a set of sample utterances that should produce each response through a provided authoring tool. The NPCEditor also contains a dialogue manager that specifies how to use the classifier results.

Nonverbal Behavior Generator (NVBG)

The NVBG is a rule-based system that analyzes character text and functional markup to propose nonverbal behaviors. The resulting schedule is Behavior Markup Language (BML).

SmartBody

SmartBody is a character animation library that provides synchronized locomotion, steering, object manipulation, lip syncing, gazing and nonverbal behavior in real-time. It uses Behavior Markup Language (BML) to transforms behavior descriptions into real-time animations.

vhtoolkitUnity

The Toolkit uses Unity as its main game engine, which has been extended to include a tight integration with SmartBody, a messaging protocol, debug and authoring tools, and a graphical timeline editor for creating cut-scenes.