LeKiwi Object: Local-First Multi-Agent Workflow for Mobile Manipulation Robots

LeKiwi Object is a course project for the LeKiwi mobile manipulation robot, aiming to build a local-first multi-agent workflow system. Its core philosophy prioritizes on-robot computation to reduce reliance on cloud servers, offering advantages like low latency (critical for real-time control), privacy security (no sensitive data upload), offline availability, and cost control. This project covers multi-agent architecture design, local-first technical implementation, typical workflow examples, educational value, limitations, and future directions.

Mobile manipulation robots combine mobile chassis flexibility with robotic arm precision, enabling tasks like navigation and fine operations. LeKiwi is an open-source, low-cost platform for education/research, integrating a wheeled chassis and lightweight arm. However, traditional centralized control struggles to coordinate interdependent tasks (navigation, perception, planning, control) that require real-time synchronization.

The project uses a multi-agent architecture:

1. Navigation Agent: Handles path planning, obstacle avoidance, SLAM (map maintenance) via ROS navigation stack.
2. Perception Agent: Processes sensor data (images/point clouds) for object detection/pose estimation using lightweight, optimized AI models.
3. Manipulation Agent: Computes grasp poses, plans arm trajectories, controls gripper actions.
4. Orchestration Agent: Parses user commands, decomposes tasks, schedules agents, handles exceptions.

Local-first implementation details:

• Hardware: NVIDIA Jetson (GPU acceleration), Raspberry Pi + Coral TPU (low-cost), Intel NUC (x86 compatibility).
• Model Optimization: Quantization (32-bit →8-bit), pruning, knowledge distillation, lightweight models (MobileNet, EfficientNet-Lite).
• Communication: ROS topics (async data flow), services (sync requests), actions (long tasks with feedback).

Scenario: User command 'Put the red block on the table into the left box'. Steps:

1. Instruction Parsing: Orchestration agent extracts target object (red block), action (grab/place), destination (left box).
2. Environment Exploration: Navigation agent moves robot to optimal observation position near the table.
3. Object Detection: Perception agent identifies red block and returns its pose.
4. Grasp Planning: Manipulation agent calculates optimal grasp pose and arm trajectory.
5. Grasp Execution: Manipulation agent controls arm to grab the block and verifies success.
6. Navigate to Destination: Navigation agent moves robot to the left box.
7. Place Object: Manipulation agent places the block into the box.
8. Task Confirmation: Orchestration agent verifies completion and reports to user.

As a course project, LeKiwi Object offers key learning opportunities:

• System Integration: Combine mechanical, electronic, software knowledge to enable collaborative work.
• Distributed System Design: Understand concurrency control, fault tolerance, message synchronization, and deadlock avoidance.
• Resource-Constrained Optimization: Learn model optimization, algorithm selection, and performance tuning under limited resources.
• Open Source Participation: Use ROS and contribute to open-source communities, understanding modern collaborative development.

Current Limitations:

• Simple agent coordination (struggles with complex concurrent scenarios).
• Less robust error recovery mechanisms.
• Perception limited by edge device computing resources.

Future Improvements:

• Integrate reinforcement learning for better task scheduling.
• Add natural language interfaces for flexible human-robot collaboration.
• Create digital twins for offline testing/simulation.
• Extend to multi-robot collaboration scenarios.

LeKiwi Object is an excellent educational example integrating multi-agent systems, edge AI, and mobile manipulation. Key insights for robot development:

1. Modular Design: Separate concerns via agents for maintainability and scalability.
2. Local-First: Prioritize local computation for reliability and responsiveness.
3. Progressive Complexity: Start with simple workflows, then add features.
4. Open Source Collaboration: Leverage open-source tools to accelerate development.

With advancing edge computing and AI optimization, local-first robot architectures will become more practical for diverse applications.