Planned intervention: On Thursday 19/09 between 05:30-06:30 (UTC), Zenodo will be unavailable because of a scheduled upgrade in our storage cluster.
Published October 18, 2022 | Version 1.0
Software Open

Multimodal-fusion-driven scene analysis and understanding

Creators

  • 1. Industrial Systems Institute Athena Research Center

Description

Multimodal fusion driven scene understanding

This repository provides a plugin for the OpenPCDet object detection framework that facilitates fusion of 2D and 3D object detection.

Instructions

  1. Please download, and install all the requirements of OpenPCDet
  2. Download folder named "fusion" into the root directory of OpenPCDet
  3. Structure should be the following
    |
    |
    |-- data/
    |
    |-- docker/
    |
    |-- docs/
    |
    |-- pcdet/
    |
    |-- fusion/
    |
    |-- tools/
    |
    |-- LICENCE
    |
    |-- README.md
    |
    |-- requirements.txt
    |
    |-- setup.py

Limitations

  1. Non reported
  2. Successfully tested with commit b345b08c5d3e49ff82a5374d033ddd2b5e66253e [2022-09-25]

Requirements

Extra requirements for cropping pdf report in evalution script:

sudo apt-get install texlive-extra-utils

Usage

Configuration file:

fusion/cfgs_custom/multimodal/config.json

{
  "multimodalv2": {
    "root": "/home/<HOME_DIR>/Workspace/Automotive/OpenPCDet/",
    "path_to_data": "data/kitti/training/",
    "path_to_calibration_for_tracking": "calib.txt",
    "path_to_groundtruth_for_tracking": "groundtruth.txt",
    "path_to_image": "image_2/",
    "path_to_image_right": "image_3/",
    "path_to_lidar": "velodyne/",
    "path_to_labels": "label_2/",
    "deeplab_root": "",
    "save_path_root": "fusion/results/dump/",
    "save_path_came": "fusion/results/dump/image/",
    "save_path_image_from_lidar": "fusion/results/dump/image_lidar/",
    "save_path_meta_data": "fusion/results/dump/meta_data/",
    "save_path_lidar": "fusion/results/dump/lidar/",
    "cut_off_percentage": 0.8,
    "cut_off_2D": 0.8,
    "nms_fusion_threshold": 0.5,
    "segmentation_model": "",
    "image_detection_model": "fusion/imagedet/models/squeezedet_kitti_epoch280.pth",
    "lidar_detection_cfg": "fusion/cfgs/kitti_models/pv_rcnn.yaml",
    "lidar_detection_model": "fusion/trained/pv_rcnn_8369.pth",
    "start_frame": 0,
    "denoise": 0,
    "meta_data": 0
  },
  "comment": {

  }
}

Run fusion:

cd fusion

python runFusion.py

Evaluate fusion outcomes,

The script compares fusion with image-only and LIDAR-only detection:

cd fusion

python runEvaluate.py

 

Files

Multimodal-fusion-driven-scene-understanding.zip

Files (1.9 GB)

Additional details

Funding

CPSoSaware – Cross-layer cognitive optimization tools & methods for the lifecycle support of dependable CPSoS 871738
European Commission

References

  • OD Team. "Openpcdet: An open-source toolbox for 3d object detection from point clouds." (2020).
  • Wu, Bichen, et al. "Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.
  • Pikoulis, E. V., Mavrokefalidis, C., Nousias, S., & Lalos, A. S. (2022). A new clustering-based technique for the acceleration of deep convolutional networks. In Deep Learning Applications, Volume 3 (pp. 123-150). Springer, Singapore.