Collaborative Perception for Autonomous Driving: From Virtual To Real Datasets

Paper submitted to the Second Workshop On Learning Beyond Deep Learning (LBDL II), part of ICIP 2026 Satellite Workshops.

Cooperative Perception (CP) supports autonomous driving by enabling connected autonomous vehicles (CAVs) to share sensor data, thereby increasing situational awareness and safety. However, training CP models requires diverse traffic scenarios, and real-world datasets are limited due to the high cost of multi-agent setups and the labor-intensive nature of labeling. Synthetic datasets offer a scalable alternative, but models trained exclusively on virtual data often face domain adaptation challenges. This paper presents a systematic benchmark that evaluates two advanced 3D LiDAR-based object detectors on synthetic and real datasets using three strategies: 

• training from scratch,
• transferring from synthetic to real data,
• and training with mixed datasets.

The results show that pre-training on synthetic data followed by fine-tuning on real datasets yields the best performance (+2% AP@0.5, +2-4% AP@0.7 for car detection), while mixed training improves cross-domain generalization. These findings underscore the value of synthetic data as a complementary resource for developing robust CP systems.