Running Large Language Models Inside SAP S/4HANA: A Pure ABAP-Implemented LLM Inference Engine

abap-llm-engine is the world's first LLM inference engine fully implemented based on ABAP, allowing direct execution of large language models (e.g., SmolLM2-135M) inside SAP S/4HANA systems. Without external dependencies like Python, llama.cpp, or ONNX, this project leverages HANA database acceleration capabilities to bring native AI to traditional enterprise ERP systems.

Traditional SAP systems lack native AI capabilities. External LLM deployment solutions often rely on Python environments, third-party inference libraries, or external API calls, leading to issues like data security risks, high integration complexity, and incompatibility with air-gapped environments. The abap-llm-engine project aims to address these pain points by natively embedding AI capabilities inside SAP systems.

The project uses a modular class structure design, with core components including:

• ZCL_LLM_ENGINE: Inference process coordinator
• ZCL_LLM_BPE_TOKENIZER: BPE tokenizer
• ZCL_LLM_TENSOR: Tensor operation class
• ZCL_LLM_TRANSFORMER_BLOCK: Transformer layer implementation (including RMS normalization, ROPE positional encoding, grouped query attention, etc.)
• ZCL_LLM_HANA_ACCEL: HANA acceleration module

Model specifications: 135 million parameters, Llama architecture (30 layers, 576 hidden dimensions, 9 attention heads), ~250MB memory usage after INT8 quantization, 8192-token context window.

The project offers three operation modes:

1. Pure ABAP Mode: Speed of 5-30 seconds per token, no special configuration needed, suitable for development testing or scenarios without HANA acceleration.
2. HANA AMDP Acceleration Mode: Speed of 0.5-3 seconds per token, offloads matrix operations to HANA parallel engine for significant speed improvement.
3. Shared Memory Mode: 30% faster than the basic mode; weight sharing reduces memory usage and enhances concurrency.

This project features zero external dependencies and native SAP integration:

• No need for Python, third-party libraries, or external APIs
• Direct access to SAP data dictionary (e.g., DD03L) to ensure accurate generated content
• Deployable via SAP transport requests for convenient version management
• Supports air-gapped environments with no need for external networks
• Sub-second inference response achievable with HANA acceleration.

The project overcame several technical challenges:

1. ABAP Matrix Operations: Designed tensor operation classes to implement efficient matrix multiplication, activation functions, etc.
2. Memory Management: Controlled model memory to around 250MB via INT8 quantization and shared memory mechanisms.
3. HANA Acceleration: Used AMDP to offload compute-intensive operations to HANA, achieving parallel acceleration effects similar to GPUs.

Application Scenarios: Intelligent report generation, ABAP code assistant development, business process optimization, natural language data querying, automatic training document generation, etc.

Summary: abap-llm-engine represents an important direction for enterprise AI integration—natively embedding into existing business systems without architecture refactoring. For SAP enterprises, it enables access to cutting-edge AI capabilities within the familiar ABAP environment, providing a feasible path for enterprise AI transformation.

Project URL: https://github.com/cadiraca/abap-llm-engine License: Apache 2.0