Core Analysis of the ETS2 Autonomous Driving AI Project

The ETS2-Driving-AI project analyzed in this article is based on the Euro Truck Simulator 2 (ETS2) simulator, using multimodal deep learning (integration of computer vision + vehicle telemetry data) and an end-to-end learning paradigm to achieve autonomous driving in a simulated environment. Built on a low-cost, high-fidelity virtual platform, the project demonstrates the application potential of deep learning in the field of autonomous driving, with both educational research value and the possibility of migration to the real world.

Project Background and Motivation

Autonomous driving research and development require expensive hardware and complex testing environments. As a highly realistic simulation game, ETS2 provides a low-cost, high-fidelity virtual testing platform. The unique feature of this project is its use of an end-to-end learning approach: the model directly outputs control signals from raw inputs (screen images + vehicle data), replacing the traditional multi-module pipeline architecture.

Detailed Technical Architecture

Multimodal Input Fusion

• Visual Input: Use MobileNetV3 (lightweight CNN) to process screen captures and extract visual features such as road boundaries and lane lines;
• Telemetry Data: Process vehicle status data (e.g., speed, speed limit, cargo weight) via MLP;
• Feature Fusion: Integrate CNN and MLP features to output three continuous control signals: steering wheel angle, throttle, and brake.

End-to-End Learning Paradigm

Advantages: Automatically learns human driving habits, avoids the limitations of manual feature engineering, and captures intuitive driving behaviors; Challenges: Poor interpretability and high requirements for the quality of training data.

Data Collection and Training Process

Data Collection

Recorded via scripts: game screen frames, vehicle data obtained from the ETS2 telemetry API, and real control signals read from the game's physics engine (to eliminate noise from human-computer interaction delays).

Training and Evaluation

• Training: Supervised learning regression framework to optimize the proximity between predicted control signals and human driving behavior, supporting flexible parameter configuration;
• Evaluation: Use metrics such as MAE, RMSE (prediction accuracy), R² (explanatory power), and error quantile analysis.

Real-Time Inference System

Workflow:

1. Screen capture: Continuously capture game images;
2. Telemetry reading: Obtain vehicle status via API;
3. Model inference: Get control signal predictions from input data;
4. Control execution: Send signals to the game via a virtual Xbox controller.

Features: Supports manual override (switch to autonomous driving via physical controller), human-machine collaboration mode (transmit human input for AI assistance).

Technical Innovations and Practical Significance

Virtual Controller Solution

Using a virtual Xbox controller to execute controls, achieving continuous output, smooth driving, and close-to-real manipulation.

Migration Potential

Although targeted at a game environment, methods such as multimodal fusion and end-to-end learning are applicable to real autonomous driving systems (e.g., Waymo and Tesla use simulators to verify algorithms).

Educational Research Value

Provides a complete, runnable reference implementation with a clear process from data collection to deployment, suitable as learning material or a research prototype.

Limitations and Future Directions

Current limitations: Only focuses on lane keeping and speed control in highway scenarios, not involving complex urban road conditions, traffic light recognition, etc.

Future directions:

• Introduce temporal modeling (LSTM, Transformer) to capture dynamic driving behaviors;
• Add semantic recognition of traffic signs and signals;
• Explore reinforcement learning to allow the model to evolve on its own;
• Research domain migration technology from simulation to reality.

Conclusion

The ETS2-Driving-AI project demonstrates the strong potential of deep learning in the field of autonomous driving, achieving smooth simulated driving through a multimodal architecture and end-to-end learning. For autonomous driving beginners, it is an excellent learning case and practice platform.