Summary of Changes

Hello @mickqian, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the SGLang server's handling of Vision-Language Models (VLMs) by implementing a more robust warmup procedure and improving model capability reporting. It introduces a minimal image for VLM warmup requests, ensuring that multimodal components are properly initialized. Furthermore, the server's API now provides explicit flags to indicate whether a model possesses image or audio understanding capabilities. The changes also include refactoring of vision attention mechanisms for better maintainability and accuracy, alongside the addition of temporary debug prints to aid in development.

Highlights

• VLM Warmup with Minimal Image: The PR introduces a minimal 1x1 black PNG image, base64 encoded, to be used during the warmup phase for Vision-Language Models (VLMs). This ensures that VLM-specific pathways are properly initialized and ready for multimodal requests.
• Model Capability Exposure: The get_model_info API now exposes new flags, has_image_understanding and has_audio_understanding, allowing clients to programmatically determine if a loaded model supports image or audio processing capabilities.
• Vision Attention Refactoring: The logic for converting Hugging Face attention backend configurations to SGLang's internal QKV backend types has been extracted into a dedicated utility function. Additionally, the vision attention mechanism now correctly handles and utilizes max_seqlen for mask generation, improving attention mask handling.
• Temporary Debugging Prints: Several temporary print statements (e.g., "0000", "1111", "22222", "333333") have been added across different files, likely for debugging purposes related to request handling and disaggregation modes.

Code Review

This pull request adds support for warming up Vision Language Models (VLMs) with a minimal image and exposes model capabilities for image and audio understanding via an API endpoint. The changes are well-structured, including a nice refactoring in the vision attention layer. My review includes suggestions to remove some leftover debug print statements and to simplify a redundant conditional block for better code clarity and maintainability.