Stage 1: Base MiniGPT Model

Implemented a miniature GPT-style language model from scratch using Python and PyTorch. Trained on classic literature to learn basic language patterns and generate coherent text. The objective was to fundamentally understand the internal mechanics of transformer-based models, including tokenization, attention mechanisms, and probabilistic text generation.

Training Data & Objective

Trained on public domain literature from Project Gutenberg (e.g., Frankenstein, Alice's Adventures in Wonderland) using a next-token prediction objective to learn structural grammar and narrative flow.

View Base Repository

Example Generated Output

Training Configuration

• Transformer layers: 4 | Attention heads: 4
• Embedding dimension: 128 | Context window: 128 tokens
• Training steps: ~10,000+ | Optimizer: Adam

loss = loss_fn(
    logits.view(-1, vocab_size),
    targets.view(-1)
)

optimizer.zero_grad()
loss.backward()
optimizer.step()

Stage 2: Rust NPC Dialogue Engine

The Base MiniGPT model was adapted and fine-tuned to power "Rust", a sarcastic mechanic living in a post-apocalyptic industrial city for the game The Girl the Sky Kept. The system was trained on custom dialogue datasets designed for specific NPC personality modeling and features an interactive chat interface for real-time response testing.

Core Features

• Dialogue dataset generation and training pipeline
• NPC personality modeling through structured prompt engineering
• Interactive chat interface for testing dynamic responses
• GPU accelerated training using PyTorch

View Rust Repository

Live Interaction Demo

Architecture

• Transformer blocks with multi-head self-attention
• Tokenized dialogue training dataset (~3000+ pairs)
• Text generation using top-k sampling and temperature scaling

class MiniGPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.blocks = nn.Sequential(*[Block() for _ in range(layers)])
        self.head = nn.Linear(embed_size, vocab_size)