Skip to main content

ElysGenAIPlugin

Version Unreal Engine License

A powerful, modular Unreal Engine 5 plugin for local, offline, real-time GenAI features built on a clean hybrid architecture.

What is ElysGenAI?​

ElysGenAIPlugin provides client-side AI capabilities for Unreal Engine games:

  • Speech-to-Text (STT) - Convert player voice to text using Whisper.cpp
  • Large Language Models (LLM) - NPC dialogue and reasoning with Phi-3-mini via llama.cpp
  • Modular Architecture - Clean separation with extensible backend system
  • Multiplayer-Safe - Client-side processing, no audio replication
  • Marketplace-Ready - Permissive MIT licenses, bundled models

Quick Navigation​

Getting Started​

  • Setup Guide - Installation, configuration, and model downloader
  • Core Concepts - Understand the hybrid architecture and design patterns

Module Guides​

  • Modules Guide - Audio capture, STT, and LLM in one unified guide

Examples & Advanced​

  • Examples - Practical recipes for voice commands, NPC dialogue, multiplayer
  • Troubleshooting - Common issues and solutions

Reference​

Architecture Overview​

ElysGenAI uses a hybrid architecture for optimal performance and Blueprint-friendliness:

Actor (Blueprint/C++)
  |
  +-- Component (UERP_STTComponent) - Blueprint API
       |
       +-- Context (FERP_STTContext) - Processing logic
            |
            +-- Subsystem (UERP_STTSubsystem) - Backend lifecycle
                 |
                 +-- Backend (UERP_WhisperBackend) - Model inference

Key Benefits:

  • Lightweight contexts (no UObject overhead for heavy processing)
  • Blueprint-friendly component wrappers
  • Single shared model (memory efficient)
  • Easy to extend with custom backends

What's Included​

Core Modules​

  • ElysGenAICore - Base interfaces, settings, type system
  • ElysGenAIAudio - Microphone capture, PTT, audio routing
  • ElysGenAISTT - Speech-to-text with Whisper.cpp backend
  • ElysGenAILLM - Language models with llama.cpp backend

Bundled Models​

  • Phi-3-mini-4k-instruct (MIT) - ~2.7GB quantized LLM for dialogue
  • Whisper base.en (MIT) - ~74MB English STT model

Third-Party Integrations​

  • llama.cpp (MIT) - Fast C++ LLM inference
  • Whisper.cpp (MIT) - Fast C++ STT inference

All components use permissive MIT licenses for marketplace compatibility.

Use Cases​

Voice Commands​

Players speak commands that trigger game actions:

// "Open inventory" -> Opens inventory UI
// "Use health potion" -> Uses item
// "Follow me" -> Commands NPC

NPC Dialogue​

NPCs use LLMs for dynamic conversation:

// Player: "Where is the blacksmith?"
// NPC: "Head north past the fountain. You can't miss the smoke!"

Multiplayer Voice Chat​

Client-side STT with server-side text processing:

// Client: Captures audio -> Transcribes -> Sends text to server
// Server: Receives text -> Processes intent -> Replicates action

System Requirements​

Unreal Engine​

  • Version: 5.6+ (compatible with 5.0+)
  • Platform: Windows, Mac, Linux
  • Network: Offline-capable (no cloud dependencies)
  • CPU: Modern multi-core processor
  • RAM: 8GB+ (4GB for models, 4GB for game)
  • GPU: Not required (CPU inference)
  • Storage: 5GB for plugin + bundled models

Getting Started in 5 Minutes​

  1. Install Plugin - Copy to YourProject/Plugins/ElysGenAIPlugin
  2. Enable in .uproject - Add to Plugins list
  3. Regenerate Project Files - Right-click .uproject -> Generate
  4. Add Component - Add UERP_STTComponent to your PlayerController
  5. Test - Press Play, speak into microphone, receive transcription

See Setup Guide for detailed walkthrough.

License​

ElysGenAIPlugin is proprietary software by Rogue Paradigm.

Third-party components (llama.cpp, Whisper.cpp, Phi-3) use MIT licenses.

Support​

Ready to get started? Head to the Setup Guide or explore Core Concepts to understand the architecture!