Reading
MDA (Multidimensional Associative Memory) is an innovative online learning system that enables large language models (LLMs) to learn and memorize in real-time during inference, without the need for GPUs or backpropagation, opening up a new path for the continuous learning of LLMs.
Section 01
MDA (Multidimensional Associative Memory) is an innovative online learning system that enables large language models (LLMs) to learn and memorize in real-time during inference, without the need for GPUs or backpropagation, opening up a new path for the continuous learning of LLMs.
Section 02
Current large language models (LLMs) have a fundamental limitation: they "remember" knowledge during the training phase, and once deployed, they cannot truly learn new things during inference.
The so-called "In-context Learning" merely stuffs examples into prompts, which is neither persistent nor efficient.
Traditional solutions—such as fine-tuning or continuous learning—require backpropagation and large amounts of computing resources, usually GPU clusters. This is too costly for real-time adaptation in production environments.
MDA (Multidimensional Associative Memory) was created precisely to address this pain point.
Section 03
MDA is an online associative memory system implemented purely in Python, enabling LLMs to learn and update knowledge in real-time during inference, with the following revolutionary features:
Section 04
The core of MDA is based on classical associative memory theory, especially the modern evolution of Hopfield networks. Its key mechanisms include:
Section 05
MDA uses a high-dimensional vector space to store memories. Each memory item is encoded as a high-dimensional vector, and associative retrieval is achieved through the geometric relationships between vectors. This representation has natural fault tolerance and generalization capabilities.
Section 06
Unlike backpropagation, MDA adopts Oja's rule—a local update mechanism based on Hebbian learning. Oja's rule only relies on the current input and existing weights, with extremely low computational complexity, making it suitable for online updates.
Section 07
The computational graph of traditional training needs to save intermediate activation values for backpropagation, while MDA's updates are "stateless"—each update only depends on the current input and existing weights, with minimal memory usage.
Section 08
Imagine a customer service robot that can remember users' preferences and historical questions during conversations, without the need for pre-training or maintaining a complex user profile system.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.