Development of an Autonomous Driving System Based on the CARLA Simulator: A Complete Technical Pipeline from Perception to Decision-Making

This project aims to build a complete autonomous driving system based on the CARLA simulator, covering an end-to-end technical pipeline from environmental perception to vehicle control. The project addresses the issues of high cost and significant safety risks in real-road testing by providing a safe and controllable experimental environment through simulation. Currently, the basic simulation framework has been built; subsequent steps will integrate core modules such as YOLO object detection and decision-making logic, ultimately achieving autonomous driving behavior.

Autonomous driving technology research and development faces challenges of high real-road testing costs and significant safety risks, making simulation platforms a key solution. As an open-source simulator, CARLA has the following advantages:

1. High-fidelity environment rendering: Generated using Unreal Engine to produce near-real visual effects, supporting training of visual perception models;
2. Rich sensor support: Built-in interfaces for multiple sensors such as RGB cameras, LiDAR, and radar;
3. Programmable traffic scenarios: Dynamically generate vehicles, pedestrians, and traffic lights via Python API;
4. Open-source and extensible: Provides Python API and C++ source code for easy customization and algorithm integration.

The project has completed the construction of the basic simulation framework, including:

1. Simulation connection and environment loading: Establish CARLA client-server connection and load city maps;
2. Dynamic vehicle generation: Generate vehicles via blueprint library, using collision detection to ensure safety;
3. Multi-view camera system: Supports driver, dashboard, hood, and other views, with real-time switching via spectator camera;
4. Vehicle control interface: Implements basic controls such as throttle, steering, and braking, following real vehicle protocols. In addition, core technical concepts include coordinate transforms (Transforms), the spectator camera system, and safe generation logic (try_spawn_actor method).

The project will develop the following core modules in subsequent phases:

1. RGB camera sensor integration: Install physical RGB cameras to obtain raw image data;
2. Real-time video stream processing: Establish image stream transmission from CARLA to Python and implement format conversion;
3. YOLO object detection integration: Use YOLO algorithm to detect objects such as pedestrians, vehicles, and traffic lights;
4. Decision-making logic development: Build rule-based or learning-based decision models to convert perception results into control commands;
5. Autonomous driving behavior implementation: Integrate modules to achieve autonomous driving functions such as lane keeping and adaptive cruise control.

The project's planned expansion directions include:

1. Real-time dashboard: Display information such as vehicle speed, obstacle distance, and decision status;
2. Lane detection and traffic light recognition: Enhance the ability to detect road elements;
3. Reinforcement learning and neural network decision models: Explore data-driven driving strategies;
4. ROS2 and SLAM integration: Connect to the Robot Operating System to implement simultaneous localization and mapping (SLAM).

The project uses a technology stack including Python (programming language), CARLA (simulation platform), NumPy (scientific computing), OpenCV (computer vision), and YOLO (object detection). Project Value:

• Learners: Understand practical cases of autonomous driving technology stacks;
• Researchers: A low-cost testing platform for verifying new algorithms;
• Engineers: A transition bridge for migrating to real systems;
• Industry: Aligns with the standard practice trend of simulation verification → real-world migration.