Perplexity
Measured on held-out sets for specific dialects.
LANGUAGE MODELINGLanguage
in its own
context.
A compact language model shaped by the words, dialects, and everyday expression of Pakistan.
01 / INTRODUCTION
Made for the way
Pakistan speaks.
Proxima AI is a specialized large language model trained from the Qwen-0.5B base architecture on a corpus focused on Pakistani languages and dialects.
0.5BPARAMETERS
~500MTRAINING TOKENS
05LANGUAGE GROUPS
02 / THE MODEL
Small enough
to be focused.
The foundation is Qwen-0.5B, an efficient transformer with approximately half a billion parameters. Proxima AI inherits its core architecture and trains on regionally focused text.
MODEL FLOW
01Prepared textRegional language corpus
02TokenizationExtended Qwen tokenizer
03Transformer24 layers / 16 attention heads
04PredictionNext-token objective
- Total parameters
- 0.5 billion
- Transformer layers
- 24
- Hidden size
- 2,048
- Attention heads
- 16
- Feed-forward intermediate
- 5,632
03 / THE CORPUS
~M
TRAINING TOKENSA regional
range of voices.
The training corpus balances five language groups. Each share below is the approximate distribution reported in the technical document.
40%
30%
15%
10%
5%
01Urdu40%
02Punjabi30%
03Sindhi15%
04Pashto10%
05Balochi + regional varieties5%
04 / THE PROCESS
Prepared
with intent.
- 01 / FILTER
Deduplicate and review
Remove repeated material and apply quality filtering to the source corpus.
- 02 / NORMALIZE
Standardize the text
Normalize text and encoding so varied sources enter a consistent pipeline.
- 03 / TOKENIZE
Represent each language
Use an extended Qwen tokenizer to prepare the text for model training.
- 04 / BALANCE
Sample across dialects
Balance training samples across the language and dialect distribution.
- 05 / TRAIN
Predict the next token
Train with a standard causal language modeling objective on the prepared corpus.
05 / EVALUATION
Evaluated from
more than one angle.
Custom test sets represent realistic Pakistani language use cases. The documentation identifies three evaluation methods.
BLEU scores
Used for translation tasks between dialect pairs.
TRANSLATIONHuman review
Fluency, coherence, and cultural appropriateness rated on a five-point scale.
QUALITATIVE REVIEWMore language.
More context.
The Proxima AI LLM is a compact, regionally attuned model. The technical document identifies larger base models and broader dialect coverage as areas for future exploration.