CAFUNE: A Brazilian Portuguese Large Language Model Based on Discrete Masked Diffusion

CAFUNE is a fully locally trained bidirectional Transformer model optimized for Brazilian Portuguese, using LLaDA-style discrete masked diffusion technology to generate text. The project demonstrates the feasibility of building an approximately 5 million parameter model without external API call costs and zero data privacy risks, equipped with a complete RLAIF teacher system and ethical monitoring mechanism.

Most large language models rely on massive computing resources and expensive API calls. CAFUNE chooses the local training path, focusing on Brazilian Portuguese culture. The core idea is to prove that a fully functional model can be built without an enterprise-level budget, and 100% local training ensures zero call costs and data privacy.

The core of the model is a 5 million parameter bidirectional Transformer implemented in Julia (d_model=256, 8 attention heads, 6-layer encoder), using discrete masked diffusion technology (20 denoising steps, temperature=0.5). The Adam optimizer (learning rate=5e-6) is used for training, and the dataset contains 6000 Brazilian Portuguese sentence pairs.

The RLAIF teacher system generates MNS scores through hybrid evaluation (60% BitNet semantic coherence, 40% Flair sentiment/part-of-speech/vocabulary coverage); the sentinel monitoring includes Raegis ethical flattery detection and Guardian anomaly detection, which apply penalty scores and flag bits respectively.

A 2048-byte memory-mapped file is used to achieve efficient communication between Julia and Python components, dividing fields such as handshake area, loss area, and text buffer; a BPE tokenizer (500 tokens, including 38 Portuguese accented characters) is used, with a sequence length limit of 128 tokens.

Deployment requires configuring environment variables, installing dependencies, and starting all components via start_all_services.bat; technical highlights include discrete diffusion application, performance advantages of the Julia language, local-first paradigm, and built-in ethical monitoring mechanism.

Limitations: Portuguese-focused, limited vocabulary/data scale, 5 million parameter capacity, 128-token length constraint; Future directions: expand languages/parameters, advanced diffusion scheduling, complex teacher system; Summary: Proves that building a complete pipeline under limited resources is feasible, and it is an excellent platform for learning and experimentation.