Integration of Bard-VL and vLLM: A High-Throughput Inference Solution for Diffusion-Based Vision-Language Models

This project integrates the Bard-VL diffusion vision-language model into the vLLM inference engine, enabling high-throughput vision-language model inference and an OpenAI-compatible service interface.

• Original Author/Maintainer: NinoNeumann
• Source Platform: GitHub
• Original Title: Bard-VL_vLLM
• Original Link: https://github.com/NinoNeumann/Bard-VL_vLLM
• Source Publication/Update Time: 2026-06-16

Vision-Language Models (VLMs) are developing rapidly, capable of understanding both images and text simultaneously to perform functions such as image description, visual question answering, and image-text dialogue. However, these models face unique challenges in practical deployment:

Bard-VL uses a diffusion architecture to generate text outputs, which is fundamentally different from traditional autoregressive language models (such as GPT, Llama):

• Iterative Denoising: Requires multiple iterative steps to gradually remove noise and generate the final output
• Computationally Intensive: Each generation step requires a complete forward pass of the model
• Parallelization Difficulty: The generation process is difficult to batch process as efficiently as autoregressive models

• Latency Sensitivity: Users expect real-time visual interaction responses
• Throughput Bottleneck: Single-user scenarios are already challenging, and multi-user concurrency is even more difficult
• Resource Consumption: Both the visual encoder and diffusion decoder require large amounts of GPU memory
• Service Compatibility: Need to be compatible with the existing API ecosystem

The Bard-VL_vLLM project, created by developer NinoNeumann, aims to address these challenges and bring diffusion-based VLMs to production-level deployment.

vLLM is a high-performance LLM inference engine developed by the University of California, Berkeley, known for its innovative PagedAttention technology:

• PagedAttention: Manages KV cache with paging to significantly reduce memory fragmentation
• Continuous Batching: Dynamically adjusts batch size to improve GPU utilization
• Memory Efficiency: Supports higher concurrency and longer contexts

Integrating diffusion models into vLLM requires solving several key issues:

Visual Encoder Integration

Bard-VL uses a visual encoder (such as CLIP or SigLIP) to process input images:

• Image Preprocessing: Resize, normalize, and split into chunks
• Feature Extraction: Generate image embedding vectors
• Text Alignment: Map visual features to the input space of the language model

Diffusion Decoder Adaptation

The core challenge is integrating the diffusion generation process into vLLM's scheduling system:

• Multi-step Iteration Management: Map diffusion denoising steps into schedulable units
• Intermediate State Caching: Cache intermediate representations between denoising iterations
• Batch Reorganization: Dynamically reorganize batches based on denoising progress

Attention Mechanism Modification

Diffusion models typically use bidirectional attention, which requires adaptation:

• Support cross-modal attention from text to image
• Handle conditional injection of diffusion time steps
• Optimize memory access patterns for attention computation

The project provides an interface compatible with the OpenAI API for easy integration: