Info: Zenodo’s user support line is staffed on regular business days between Dec 23 and Jan 5. Response times may be slightly longer than normal.

Published October 18, 2022 | Version 1.0
Software Open

Multimodal-fusion-driven scene analysis and understanding

Creators

  • 1. Industrial Systems Institute Athena Research Center

Description

Multimodal fusion driven scene understanding

This repository provides a plugin for the OpenPCDet object detection framework that facilitates fusion of 2D and 3D object detection.

Instructions

  1. Please download, and install all the requirements of OpenPCDet
  2. Download folder named "fusion" into the root directory of OpenPCDet
  3. Structure should be the following
    |
    |
    |-- data/
    |
    |-- docker/
    |
    |-- docs/
    |
    |-- pcdet/
    |
    |-- fusion/
    |
    |-- tools/
    |
    |-- LICENCE
    |
    |-- README.md
    |
    |-- requirements.txt
    |
    |-- setup.py

Limitations

  1. Non reported
  2. Successfully tested with commit b345b08c5d3e49ff82a5374d033ddd2b5e66253e [2022-09-25]

Requirements

Extra requirements for cropping pdf report in evalution script:

sudo apt-get install texlive-extra-utils

Usage

Configuration file:

fusion/cfgs_custom/multimodal/config.json

{
  "multimodalv2": {
    "root": "/home/<HOME_DIR>/Workspace/Automotive/OpenPCDet/",
    "path_to_data": "data/kitti/training/",
    "path_to_calibration_for_tracking": "calib.txt",
    "path_to_groundtruth_for_tracking": "groundtruth.txt",
    "path_to_image": "image_2/",
    "path_to_image_right": "image_3/",
    "path_to_lidar": "velodyne/",
    "path_to_labels": "label_2/",
    "deeplab_root": "",
    "save_path_root": "fusion/results/dump/",
    "save_path_came": "fusion/results/dump/image/",
    "save_path_image_from_lidar": "fusion/results/dump/image_lidar/",
    "save_path_meta_data": "fusion/results/dump/meta_data/",
    "save_path_lidar": "fusion/results/dump/lidar/",
    "cut_off_percentage": 0.8,
    "cut_off_2D": 0.8,
    "nms_fusion_threshold": 0.5,
    "segmentation_model": "",
    "image_detection_model": "fusion/imagedet/models/squeezedet_kitti_epoch280.pth",
    "lidar_detection_cfg": "fusion/cfgs/kitti_models/pv_rcnn.yaml",
    "lidar_detection_model": "fusion/trained/pv_rcnn_8369.pth",
    "start_frame": 0,
    "denoise": 0,
    "meta_data": 0
  },
  "comment": {

  }
}

Run fusion:

cd fusion

python runFusion.py

Evaluate fusion outcomes,

The script compares fusion with image-only and LIDAR-only detection:

cd fusion

python runEvaluate.py

 

Files

Multimodal-fusion-driven-scene-understanding.zip

Files (1.9 GB)

Additional details

Funding

CPSoSaware – Cross-layer cognitive optimization tools & methods for the lifecycle support of dependable CPSoS 871738
European Commission

References

  • OD Team. "Openpcdet: An open-source toolbox for 3d object detection from point clouds." (2020).
  • Wu, Bichen, et al. "Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.
  • Pikoulis, E. V., Mavrokefalidis, C., Nousias, S., & Lalos, A. S. (2022). A new clustering-based technique for the acceleration of deep convolutional networks. In Deep Learning Applications, Volume 3 (pp. 123-150). Springer, Singapore.