Theo
The first open-source biblical language model family trained from scratch in Brazilian Portuguese
Open Source
Brazilian AI Milestone
The first open-source biblical language model family trained from scratch in Brazilian Portuguese
Almost every LLM that can discuss the Bible in Portuguese is either a translation of an English model, a fine-tune on top of a large multilingual base (LLaMA, Mistral, etc.), or proprietary. Theo is none of those things. It is a ground-up experiment: custom tokenizer, custom architecture, custom training pipeline — shaped entirely by the Word, from the very first weight initialization.
The project spans four model sizes — 7.4M, 27.3M, 69.3M, and 76.8M real parameters — each trained in two phases: Continual Pre-Training (CPT) on a corpus of 457 million tokens of biblical and theological content, then Supervised Fine-Tuning (SFT) on nearly 19,000 theological conversations. The result, together with the flagship Theo-500M, is 10 open-source checkpoints, all available on HuggingFace under Apache 2.0.
The 78M Chat model — the largest completed — was rated B / 7.5 in qualitative evaluation. It maintains a consistent personality (warm, curious, evangelical, Brazilian), engages naturally with questions of faith, cites biblical context, and asks questions back. For a model trained entirely from scratch at this scale, that's remarkable.
Theo also proves something broader: you don't need a team, a research lab, or millions of dollars to train a real language model. You need curiosity, a cloud GPU, and time.
Honestly — I used to think training a model from scratch was simply impossible for someone like me. Something reserved for research labs, massive teams, and budgets I'd never have. Theo proved me completely wrong, and that changes everything. This project didn't just work — it genuinely excited me about where this can go. I want to go much deeper: bigger architectures, richer corpora, models that actually push the boundary of what a small specialized LLM can do. This is only the beginning.
What went into building Theo
Five sizes, two variants each — Base (CPT) for research, Chat (SFT) for conversations
From raw text to a model that understands faith — built entirely from scratch
12+ Christian websites were scraped and processed alongside multiple Portuguese Bible translations to build a corpus of 265,000 documents — totaling 457 million tokens. Sources include theological commentaries, sermons, doctrinal articles, devotionals, and apologetics, all in Brazilian Portuguese.
A custom Byte-Pair Encoding tokenizer with 16,000 tokens was trained from scratch on the biblical corpus — not adapted from an existing tokenizer. It achieves a density of 1.26 tokens per word in Brazilian Portuguese, optimized for the theological vocabulary of the domain.
Four decoder-only Transformer architectures — 10M, 25M, 50M, and 78M — were trained from random initialization on the full corpus for 2 epochs each. No weights from any external model were used at any point. Training ran on an NVIDIA L40S 48GB via RunPod.
Each base model was fine-tuned on ~19,000 theological conversation pairs generated with a Qwen 35B teacher model — question/answer dialogues covering faith, Scripture, Christian life, and the personality of Theo. This phase transforms a text-completion model into a conversational assistant.
Not a fine-tune. Not a translation. Something built from nothing.
No pretrained weights. No transfer learning. Theo's architecture, tokenizer, and training pipeline were built independently — which means its representations are shaped entirely by biblical content, not by web-scale English data that had to be unlearned.
Every token Theo was trained on comes from the Bible, biblical commentary, theology, or Christian writing in Portuguese. The result is a model with deep domain specialization — it understands the language of Scripture at a structural level, not just superficially.
The tokenizer, corpus, and fine-tuning data are all in Brazilian Portuguese. Theo wasn't translated or adapted — it thinks in PT-BR from the ground up. In 2026, there is no other publicly available LLM like it in this language and domain.
Theo was designed, trained, evaluated, and released by a single developer — no team, no research institution, no budget beyond cloud GPU hours. It demonstrates that the barrier to entry for training real language models is lower than most people think.
The four model sizes exist to demonstrate scaling in action. Each generation shows measurable improvement in coherence, grammar, and personality. The progression from 7.4M to 76.8M parameters is a live experiment in how language emerges with scale.
All 10 model checkpoints, the tokenizer, the training code, and the inference scripts are publicly available on HuggingFace under the Apache 2.0 license. Anyone can download, fine-tune, or build on top of Theo — no restrictions, no fees.
Built with open tools, trained on open data
Python
PyTorch
NVIDIA L40S
RunPod
HuggingFace
Mac M4 (dev)
All 10 models are open source and ready to run. Clone, fine-tune, or build on top.