In-depth Analysis of Large Language Models: From Architectural Principles to Efficient Fine-Tuning

This article is based on the GitHub report LLM-presentation (link: https://github.com/DanielServejeira/LLM-presentation) published by Daniel Henrique Peres Servejeira and João Gabriel de Morais Bezerra on May 27, 2026. It systematically organizes the core principles of large language models, including neural network architecture, decoding and sampling algorithms, pre-training paradigms, and parameter-efficient fine-tuning techniques, helping developers establish a complete cognitive framework for generative AI.

Mainstream architectures are divided into three categories:

• Decoder-only models: Represented by the GPT series, they generate text word by word autoregressively, excel at fluent long text generation, and are the foundation of conversational AI;
• Encoder-only models: Represented by BERT, they use bidirectional attention mechanisms and excel at understanding tasks (sentiment analysis, named entity recognition, etc.);
• Encoder-decoder hybrid models: Represented by T5 and BART, they balance understanding and generation capabilities and are suitable for sequence-to-sequence tasks such as machine translation and text summarization.

Models sample outputs from probability distributions, and strategies affect diversity and quality:

• Temperature parameter: Low temperature (e.g., 0.2) makes outputs conservative and deterministic, while high temperature (e.g.,1.5) increases randomness;
• Top-k sampling: Only considers the top k words with the highest probability (typically k=50), balancing quality and diversity;
• Top-p sampling: Dynamically selects a set of words whose cumulative probability reaches the threshold p, adaptively adjusting the number of candidates.

Pre-training consists of two stages:

• Self-supervised learning tasks: BERT uses Masked Language Modeling (MLM, predicting masked words), while GPT uses Causal Language Modeling (CLM, predicting the next word);
• Large-scale datasets: Relies on cleaned high-quality corpora such as C4 and The Pile (web pages, books, code, etc.);
• Optimization objective: Minimize cross-entropy loss, with training costs reaching millions of dollars.

LoRA solves the problem of high cost in traditional fine-tuning:

• Core idea: Add low-rank matrices BA next to the original weight W (W'=W+BA), where the rank r is much smaller than the original dimension;
• Effectiveness: Task adjustment only requires a low-dimensional subspace, reducing trainable parameters from billions to millions;
• Application value: Consumer-grade GPUs can perform fine-tuning, supporting lightweight adapters for multiple tasks and lowering deployment thresholds.

• Evaluation metrics: Perplexity measures prediction ability, and scaling laws show that performance grows in a power-law manner with the number of parameters, data volume, and computational volume;
• Social challenges: Hallucinations (generating false information), copyright (infringement of training data), harmful content (bias and discrimination), and energy consumption (high carbon footprint).

Conclusion: Large language models are an important advancement in the AI field, and understanding their principles helps in using existing tools and innovating the next generation of models; Recommendations: Establish a responsible usage framework to address challenges such as hallucinations, copyright issues, harmful content, and energy consumption.