Notorch: A Neural Network Framework Rewritten in Pure C, Ditching PyTorch's 2.7GB Baggage

Notorch is a pure C neural network framework designed as an alternative to PyTorch for specific scenarios. It uses only ~3300 lines of C code (2 files: notorch.h and notorch.c), requires no Python runtime, and supports modern deep learning features like Transformer architecture, automatic differentiation, BitNet quantization, LoRA fine-tuning, and more. Key advantages include fast compilation (under 1 second), minimal memory footprint, and transparency—ideal for edge/embedded devices, teaching, or rapid prototyping.

PyTorch's widespread use comes with significant tradeoffs:

• Size: 2.7GB base package (plus dependencies like torchvision push it over 3.5GB).
• Overhead: Slow import times (tens of seconds), Python runtime GIL/GC issues, and complex build systems (CMake, CUDA Toolkit).
• Complexity: 400k lines of hidden C++ code behind the Python API.

Notorch was born to counter this over-complication. Its core idea (as noted in the header file) is to prove neural networks—at their essence, matrix operations and softmax—don’t need massive infrastructure.

Notorch’s core features include:

• Minimal Architecture: 2 files, ~3300 lines of C code; simple compile command (cc notorch.c -o notorch -lm).
• Tensor System: Up to 8D tensors, reference-counted memory management, no GC pauses.
• Auto-Differentiation: Tape-based reverse-mode AD with explicit operations (no zero_grad or no_grad context).
• Full Transformer Support: Linear layers, RMS/LayerNorm, causal/multi-head/grouped query attention, RoPE, SwiGLU/GEGLU activations, and cross-entropy loss.
• Optimizers: Adam, AdamW (weight decay decoupling), and Chuck (adaptive optimizer with 5-level gradient perception).

Notorch also offers advanced features:

• BitNet b1.58: Native support for 3-value weight quantization (-1,0,+1) and int8 activation quantization, with STE for backprop.
• LoRA Fine-Tuning: Freeze base parameters to train only adapter layers for efficient fine-tuning.
• BLAS Inference: Optimized matrix operations via BLAS/Apple Accelerate/CUDA.
• Alignment Training: DPO (Direct Preference Optimization) and GRPO (Generalized Reward Policy Optimization).
• Cross-Platform Support: Linux/macOS/Windows (MinGW/WSL), x86_64/ARM64, and CUDA (via conditional compilation).

Performance comparison with PyTorch:

Metric	PyTorch	Notorch
Installation Size	2.7GB+	<1MB
Compile Time	Tens of mins	<1 second
Startup Delay	Seconds	Milliseconds
Memory Footprint	GB-level	MB-level

Use cases:

• Embedded Deployment: Integrate neural networks into C/C++ apps without Python runtime.
• Edge Devices: Run models on resource-constrained hardware.
• Teaching: Transparent codebase to learn deep learning fundamentals.
• Rapid Prototyping: Fast compile times for iterative development.

Notorch isn’t meant to replace PyTorch but to provide an alternative for scenarios where efficiency, transparency, or resource constraints matter. Its philosophy aligns with the 'Arianna Method'—focus on patterns over parameters, engineering over emergence, and control over abstraction.

In conclusion, Notorch proves deep learning frameworks don’t have to be臃肿. For developers needing a lightweight, transparent tool, it offers a refreshing choice—empowering them to take control of their AI workflows.