Embodied AI for virtual characters

Note

This feature is a work in progress (WIP) and will be available in future versions of MAGES SDK.

MAGES NXT introduces a tailored assistant designed for medical simulations. This virtual character serves as a guide, leading users through the simulation.

Utilizing our proprietary RAG (Retrieval Augmented Generation) pipeline, users can furnish custom data to train the assistant, ensuring optimal guidance during the simulation.

Additionally, the assistant enhances user immersion by delivering a more human-like interaction. This is achieved through a combination of a custom voice and an avatar featuring realistic facial expressions, animations, lip-syncing, and inverse kinematics. Notably, the assistant is designed to provide fast responses, further enhancing the user experience.

../../_images/avatar.png

In-simulation example.

All this is achieved through a simple, easy-to-use API, allowing users to focus on the simulation content while we take care of the rest.

../../_images/workflow.png

RAG pipeline overview.

Making use of Microsoft’s Azure Speech SDK, the assistant seamlessly transforms into a conversational agent, facilitating a more natural interaction with the user.