VisionDesk: A Gesture-Controlled Virtual Mouse System Based on Computer Vision

VisionDesk is a gesture recognition virtual mouse system implemented using MediaPipe and OpenCV, supporting various contactless interactive operations such as cursor movement, clicking, dragging, volume and brightness adjustment. This project captures hand movements via a camera and converts them into operation commands, providing users with a new human-computer interaction method suitable for multiple scenarios like presentations, kitchens, and accessibility.

VisionDesk is a gesture-controlled virtual mouse platform based on computer vision and artificial intelligence, aiming to realize contactless computer interaction. It addresses the operational needs of users with busy hands (e.g., kitchen operations), in hygiene-sensitive environments (e.g., hospitals), or with mobility impairments, lowering the threshold of traditional input methods and providing a more natural and intuitive interaction experience.

Tech Stack

• OpenCV: Video stream capture and image preprocessing
• MediaPipe: High-precision hand key point detection
• PyAutoGUI: Simulate mouse and keyboard operations
• PyCAW: Windows volume adjustment

Gesture Command Mapping

• V gesture: Control cursor movement
• Index finger up: Right-click
• Middle finger up: Left-click
• Two fingers together: Double-click
• Fist: Drag mode
• Pinch gesture: Adjust volume/brightness (main hand), page scrolling (auxiliary hand)

Advantages of the tech combination: MediaPipe lowers the development threshold, OpenCV is cross-platform, and PyAutoGUI enables system interaction.

VisionDesk has a wide range of application scenarios:

1. Presentations: Speakers can control slides without touching the computer
2. Kitchens/Laboratories: Operate devices even when hands are greasy or covered with materials
3. Accessibility Assistance: Provide alternative operation solutions for users with mobility impairments or RSI (Repetitive Strain Injury)
4. Hygiene-sensitive environments: Reduce cross-contamination risks in hospitals and food factories

These scenarios reflect the practical value of contactless interaction.

Core challenges during development and their solutions:

• Latency optimization: Rely on MediaPipe's lightweight design to ensure real-time interaction
• Mis-touch prevention: Use specific gestures (e.g., V gesture) as trigger conditions to reduce misoperations
• Environmental adaptability: OpenCV image preprocessing standardizes input to improve robustness

These measures ensure the system's stability and user experience.

The project plans to add the following in the future:

• Voice command integration: Gesture + voice hybrid interaction
• Custom gestures: Allow users to define exclusive commands
• Multi-monitor support: Cross-screen cursor movement
• AI gesture training: User-defined gesture training interface

These features will further enhance personalization and scalability.

VisionDesk is an epitome of human-computer interaction moving towards natural and intuitive directions, integrating mature frameworks to form a usable product with intuitive interaction logic. It is not only a practical tool but also an excellent case for computer vision application development. With technological progress, contactless interaction solutions are expected to become a strong complement to traditional input methods and be applied in more scenarios.