Building an NLP Pipeline from Scratch: Dual-Track Implementation of Word2Vec and Named Entity Recognition

This open-source project builds an end-to-end NLP processing pipeline, covering two core tasks: word embedding learning (Word2Vec Skip-Gram with negative sampling) and Named Entity Recognition (NER). The NER task uses a dual-track implementation: Hidden Markov Model (HMM) and feedforward neural network, providing learners with an example of combining theory and practice, helping them understand the similarities and differences between statistical methods and deep learning.

As a core field of AI, NLP requires solid theoretical knowledge and engineering skills. However, learners often struggle to translate algorithms into code or understand the connections between different technical approaches. This project addresses these issues through an end-to-end pipeline and dual-track implementation strategy, allowing learners to intuitively compare method differences and deepen their understanding of algorithms.

Word embedding is the foundation of modern NLP. The project uses the Skip-Gram architecture of Word2Vec with negative sampling optimization. The core of Skip-Gram is to predict context words using the central word, which is suitable for large-scale corpora and learning low-frequency words. Implementation details to focus on include window size (semantic scope) and negative sampling rate (balance between efficiency and quality). The trained word embeddings can be used for downstream tasks such as word similarity calculation and analogy relation discovery.

NER is a core task in information extraction. The project provides two implementations:

1. HMM: A statistical method that models NER as a sequence labeling problem, relying on initial state, transition, and emission probabilities, with decoding using the Viterbi algorithm. Its advantages are mature theory and strong interpretability; disadvantages are limited feature engineering and difficulty in capturing long-distance dependencies.
2. Feedforward Neural Network: A deep learning method that converts words into vectors using word embeddings (e.g., output from Word2Vec) and inputs them into a multi-layer network for classification. Its advantages are automatic feature learning, ability to capture complex patterns, and integration of more context information.

The highlight of the project is its end-to-end pipeline design: first learn word embeddings via Skip-Gram, then use them as input for NER. The modular design improves code reusability and aligns with best practices in machine learning engineering. Learners can gain practical skills from the code, such as data preprocessing (cleaning, tokenization, vocabulary building), model training (learning rate scheduling, early stopping), and evaluation metrics (precision, recall, F1-score).

Different readers can learn in a differentiated way:

• Beginners: Start with the HMM implementation to understand statistical methods and the Viterbi algorithm, then transition to neural networks to experience the charm of representation learning.
• Deep Learning Experienced Users: Focus on Word2Vec details (negative sampling, hierarchical Softmax, subsampling of high-frequency words).
• Interview/Review Candidates: Cover high-frequency exam points (word embedding principles, sequence labeling decoding, comparison between statistical methods and neural networks) to connect knowledge points into a systematic network.

Although Transformers and LLMs are gaining attention, classic methods like Word2Vec and HMM still have value: they help grasp the development context of NLP and play a role in resource-constrained scenarios (edge devices, low latency). This project contributes valuable resources to the NLP education community, balancing theoretical depth and engineering accessibility, making it an open-source repository worth in-depth study.