Core Concepts

Understand ElysGenAI's architecture and design patterns.

Hybrid Architecture Pattern

The plugin uses two types of classes that work together to provide functionality.

Core Structs (F* prefix)

These do the actual work - fast and lightweight.

// Example: Speech-to-Text core struct
struct FERP_STTContext
{
    void ProcessAudio(const TArray<float>& Audio);
    FString GetTranscription();
};

Why structs?

• No UObject overhead
• Faster allocation/deallocation
• Direct memory access
• Ideal for heavy processing

Component Wrappers (U* prefix)

These expose functionality to Blueprints.

// Example: Blueprint wrapper component
UCLASS(BlueprintType)
class UERP_STTComponent : public UActorComponent
{
    UFUNCTION(BlueprintCallable)
    void StartListening();
    
    UPROPERTY(BlueprintAssignable)
    FElysSTTDelegate OnTranscriptionComplete;
};

Why components?

• Blueprint-friendly
• Event broadcasting
• Actor lifecycle management
• Easy to use

How They Work Together

Component wrappers own and forward to core structs:

void UERP_STTComponent::StartListening()
{
    // Blueprint calls component...
    // ...which forwards to core struct
    Context->StartListening();
}

Data flow:

Blueprint → Component (U*) → Core Struct (F*) → Processing
                                                       ↓
Blueprint ← Component (U*) ← Core Struct (F*) ← Result

Which One to Use?

If you're using...	Use this
C++ code	Core structs (F*Context)
Blueprints	Components (U*Component)
Both	Components (they work everywhere)

---

Module Structure

ElysGenAI is organized into four core modules:

ElysGenAICore

Purpose: Foundation module providing shared types and interfaces.

Contents:

• IERP_GenAIService - Base interface for all AI services
• IERP_AudioConsumer - Interface for audio consumers
• UERP_GenAISettings - Project settings
• FERP_AudioFormat, FERP_STTResult, FERP_LLMResult - Data types

When to use: Import for custom backend implementations.

ElysGenAIAudio

Purpose: Microphone capture and audio routing.

Contents:

• UERP_AudioCaptureSubsystem - Captures microphone input
• FERP_AudioRingBuffer - Circular audio buffer
• Push-to-talk modes (AlwaysOn, PushToTalk, PushToMute)

When to use: Any feature requiring microphone input.

ElysGenAISTT

Purpose: Speech-to-Text using Whisper.cpp.

Contents:

• UERP_STTComponent - Blueprint API
• FERP_STTContext - Processing logic
• UERP_STTSubsystem - Backend lifecycle
• UERP_WhisperBackend - Whisper.cpp integration

When to use: Voice commands, transcription, dialogue input.

ElysGenAILLM

Purpose: Language models using llama.cpp.

Contents:

• UERP_LLMComponent - Blueprint API
• FERP_LLMContext - Processing logic
• UERP_LLMSubsystem - Backend lifecycle
• UERP_LlamaCppBackend - llama.cpp integration (Phi-3-mini)

When to use: NPC dialogue, dynamic text generation.

---

Component Hierarchy

Audio Flow

Microphone
    ↓
UERP_AudioCaptureSubsystem (captures audio)
    ↓
FERP_AudioRingBuffer (buffers audio)
    ↓
IERP_AudioConsumer (distributes to consumers)
    ↓
├─ STT Component
├─ Voice Chat Component
└─ Recording Component

Key Points:

• Single audio capture point
• Fan-out to multiple consumers
• Consumer pattern for extensibility

STT Architecture

Actor with UERP_STTComponent (Blueprint API)
    ↓
FERP_STTContext (per-actor state, audio buffering)
    ↓
UERP_STTSubsystem (backend lifecycle, context tracking)
    ↓
IERP_STTBackend (interface for models)
    ↓
UERP_WhisperBackend (Whisper.cpp model loading, inference)
    ↓
Whisper.cpp C library

Layers:

1. Component - Blueprint API, event broadcasting
2. Context - Per-actor state, audio buffering
3. Subsystem - Backend lifecycle, context tracking
4. Backend - Model loading, inference
5. Model - Whisper.cpp C library

LLM Architecture

Actor with UERP_LLMComponent (Blueprint API)
    ↓
FERP_LLMContext (conversation history, prompt formatting)
    ↓
UERP_LLMSubsystem (backend lifecycle, context tracking)
    ↓
IERP_LLMBackend (interface for models)
    ↓
UERP_LlamaCppBackend (llama.cpp model loading, inference)
    ↓
llama.cpp C library (Phi-3-mini)

Layers:

1. Component - Blueprint API, event broadcasting
2. Context - Conversation history, prompt formatting
3. Subsystem - Backend lifecycle, context tracking
4. Backend - Model loading, inference
5. Model - llama.cpp C library

---

Key Design Patterns

Subsystem Pattern

Purpose: Manage backend lifecycle and shared resources.

Benefits:

• Single model instance (memory efficient)
• Automatic initialization/cleanup
• Global access via GetGameInstance()->GetSubsystem<T>()

Example:

UERP_STTSubsystem* STTSubsystem = 
    GetGameInstance()->GetSubsystem<UERP_STTSubsystem>();

Consumer Pattern

Purpose: Distribute audio to multiple consumers.

Benefits:

• Single capture point
• Multiple consumers (STT, voice chat, recording)
• Extensible (implement IERP_AudioConsumer)

Example:

class UMyAudioConsumer : public IERP_AudioConsumer
{
    virtual void OnAudioDataReceived_Implementation(const FERP_AudioBuffer& Buffer) override
    {
        ProcessAudio(Buffer.AudioData);
    }
};

Context Pattern

Purpose: Lightweight per-actor processing state.

Benefits:

• No UObject overhead
• Fast allocation/deallocation
• Multiple contexts share single backend

Example:

TUniquePtr<FERP_STTContext> Context = MakeUnique<FERP_STTContext>();
Context->ProcessAudio(AudioData);

---

Memory Model

Single Backend, Multiple Contexts

UERP_STTSubsystem (Game Instance Subsystem)
    |
    +-- UERP_WhisperBackend (single model, ~74MB)
         |
         +-- FERP_STTContext (Actor 1, ~1KB)
         +-- FERP_STTContext (Actor 2, ~1KB)
         +-- FERP_STTContext (Actor 3, ~1KB)

Benefits:

• Model loaded once (saves memory)
• Each actor has own processing state
• Context switching is fast

Lifecycle Management

Subsystem:

• Initialized on Game Instance creation
• Loads backend and model
• Cleaned up on Game Instance destruction

Component:

• Created with Actor
• Registers context with Subsystem
• Unregisters on Actor destruction

Context:

• Created by Component
• Destroyed by Component
• Lightweight (no UObject overhead)

---

Extension Points

Custom Audio Consumers

Implement IERP_AudioConsumer to receive audio:

UCLASS()
class UMyCustomConsumer : public UObject, public IERP_AudioConsumer
{
    GENERATED_BODY()
    
    virtual void OnAudioDataReceived_Implementation(const FERP_AudioBuffer& Buffer) override;
    virtual FString GetConsumerName_Implementation() const override;
};

Custom STT Backends

Implement IERP_STTBackend for custom models:

UCLASS()
class UMySTTBackend : public UObject, public IERP_STTBackend
{
    GENERATED_BODY()
    
    virtual bool Initialize_Implementation(const FString& ModelPath) override;
    virtual void Transcribe_Implementation(const TArray<float>& Audio, FERP_STTResult& Result) override;
};

Custom LLM Backends

Implement IERP_LLMBackend for custom models:

UCLASS()
class UMyLLMBackend : public UObject, public IERP_LLMBackend
{
    GENERATED_BODY()
    
    virtual bool Initialize_Implementation(const FString& ModelPath) override;
    virtual void Generate_Implementation(const FString& Prompt, FERP_LLMResult& Result) override;
};

---

Best Practices

Performance

• Use lightweight contexts for per-actor state
• Share backend across all contexts
• Process audio on worker threads
• Use ring buffers for audio streaming

Memory

• Single model instance per type
• Destroy contexts when actors destroyed
• Use quantized models (Q4) for LLMs
• Monitor memory usage with stat commands

Multiplayer

• Process audio client-side only
• Never replicate audio data
• Replicate transcription text if needed
• Use server RPCs for command processing

Blueprint Usage

• Use components for Blueprint access
• Bind to events for async results
• Check IsListening() before starting
• Clean up bindings on component destruction

---

Next Steps

Now that you understand the architecture:

• Modules Guide - Deep dive into Audio, STT, and LLM
• Examples - Practical implementation recipes
• API Reference - Complete class documentation