LLM Learning Journey: A Complete Practical Guide from Word Embeddings to Transformer Architecture

Ananyagawade12's LLMs repository documents a complete learning exploration journey from basic word embeddings to modern Transformer architectures, covering classic word embedding techniques, Transformer variants, positional encoding, normalization techniques, model evaluation, and prompt engineering. It forms a structured learning system through the combination of implementations and experiments, providing a reference path for NLP/LLM learners from different backgrounds.

Positioned as a "learning journey", this repository differs from those that only provide code. It records thoughts, experiments, and comparative analyses during the learning process, covering NLP basic concepts, classic techniques, modern architectures, and optimization methods, forming a complete learning system and providing a reference roadmap for developers, researchers, or students.

Word embedding is the cornerstone of NLP, mapping discrete words to continuous vectors. The repository discusses three mainstream techniques:

• Word2Vec: Learns word vectors by predicting context/target words;
• FastText: Introduces subword information to handle rare and out-of-vocabulary words;
• GloVe: Uses global word-word co-occurrence statistics to learn word vectors. It includes comparative experiments analyzing the differences in performance on semantic tasks, semantic relationships, and computational efficiency among different techniques.

Transformer is the core of modern LLMs. The repository covers multiple variants:

• BERT: Bidirectional encoder pre-trained via masked language modeling and next sentence prediction;
• IndicBERT: BERT variant optimized for Indian languages;
• BART: Combines bidirectional encoder and autoregressive decoder, suitable for text generation. It deeply analyzes the attention mechanism, including self-attention calculation, multi-head attention design, and weight visualization.

Positional Encoding

Transformer requires positional encoding to inject sequence order:

• Absolute positional encoding: Generates unique encoding using sine/cosine functions;
• Relative positional encoding: Encodes relative relationships between positions;
• RoPE: Integrates rotation matrices into attention calculation, with better extrapolation ability.

Normalization Techniques

Deep network training requires normalization:

• LayerNorm: Normalizes across the feature dimension of samples;
• RMSNorm: Simplified version of LayerNorm, removing mean centering;
• pRMSNorm: Introduces learnable scaling parameters. It includes comparative experiments showing performance differences under different tasks and model depths.

Model Evaluation

• Sequence generation tasks: CER, WER, BLEU, chrF++, BERTScore;
• Classification tasks: Accuracy, Precision, Recall, F1 Score.

Prompt Engineering

• Zero-Shot Prompting: Direct task description;
• One-Shot Prompting: Provide one example;
• Few-Shot Prompting: Learn task patterns from multiple examples.

Learning Outcomes

1. Intuitive understanding of the complete NLP pipeline process;
2. Technical evolution trajectory from traditional embeddings to LLMs;
3. In-depth understanding of Transformer's internal mechanisms;
4. Implementation and evaluation experience of multiple model variants.

Value to Learners

• Beginners: Learn step-by-step according to the structure, consolidate theory through code and experiments;
• Experienced developers: Quickly look up specific technical details;
• Researchers: Comparative experiments and evaluation metric implementations can serve as a starting point for research.

Tech Stack Inference

Main language: Python; Deep learning frameworks: PyTorch/TensorFlow; Tokenization tools: Hugging Face Tokenizers/SentencePiece; Evaluation libraries: Possibly NLTK, JiWER, etc.