sudani_lm in progress
A language model for Sudanese Arabic, trained from scratch on my own two-GPU workstation. Trains its own 32k BPE tokenizer, pretrains a 110M-parameter decoder-only transformer (12 layers, 768-dim, RoPE, SwiGLU) on 5B Arabic tokens at 60% MFU, then continues pretraining on Sudanese text and finetunes on dialect chat, with each data decision settled by ablation. Configs and results for every stage are in the README.