This repository implements a tool based on our work "Software-Only Semantic Diverse Redundancy for High-Integrity AI-Based Functionalities" presented at the 12th Edition of the European Congress on Embedded Real Time Systems (ERTS 2024), on June 11-12, 2024. The paper can be accessed at: https://hal.science/hal-04614881

This tool allows the execution of YOLOv4 with different redundancy modes (Dual Modular Redundancy, or Triple Modular Redundancy), and it has been implemented on top of the Darknet framework (https://github.com/AlexeyAB/darknet). Note that the Darknet framework supports the execution of other CNNs, and our implementation could be modified, if needed, to support other CNNs.

Our approach builds on realizing a redundant scheme for the inference with a given AI model by altering the inputs across redundant instances, and merging them conveniently.

Note that to achieve maximum performance, three parallel compute units are needed. However, in this repository, we provide a software-only implementation where all redundant images are executed simultaneously on a single GPU.

The outputs from the different redundant components are then merged conveniently using different merging algorithms (Voting, Averaging, or Maximum).

Please refer to the aforementioned publication for in-depth details, and the experimental evaluation of our solution. This explanation will focus on the description and usage of the tool.

This tool allows for both fault-free executions and fault injections to evaluate the robustness of the model under random faults.

Tool parameters

List of all image transformations implemented:

• none: This correspond to the baseline without any image transformation
• hflip
• vflip
• equalize
• gamma
• gaussian_blur 
• gaussian_noise
• median_blur
• raise_value
• salt_pepper
• sharpen
• right_shift
• left_shift
• top_shift
• bottom_shift
• anticlockwise_rotate
• clockwise_rotate
• dropout

List of all added parameters to call Darknet (check the Darknet GitHub for other parameters if you need it: https://github.com/AlexeyAB/darknet):

Execution modes:

• '-single_execution': execute YOLO with a single component
• '-dmr_execution': execute YOLO with 2 components
• '-tmr_execution': execute YOLO with 3 components

Specify which image transformations to use:

• '-img_transformations X (X) (X)': specify 1, 2 or 3 image transformations accordingly, depending on which execution mode you have selected. If you want to execute it without any image modification, specify "none"

Modify the image transformation parameters (e.g., number of pixels to shift): 

• '-img_transformations_cfg img_cfg.txt': specify the file where all the image parameters are stored
• The format of the img_cfg.txt file is as follows:
  • 1.5 // gamma: used for gamma correction (cv::Mat gamma_image(cv::Mat img, float gamma))
  • 5 // kernel_size: used for gaussian blur (cv::Mat gaussian_blur_image(cv::Mat img, int kernel_size)) and median blur (cv::Mat median_blur_image(cv::Mat img, int kernel_size))
  • 0 // mean: used for gaussian noise (cv::Mat gaussian_noise_image(cv::Mat img, float mean, float sigma))
  • 10 // sigma: used for gaussian noise (cv::Mat gaussian_noise_image(cv::Mat img, float mean, float sigma))
  • 0.9 // power: used for raise value (cv::Mat raise_value_image(cv::Mat img, float power))
  • 0 // low: used for salt pepper (cv::Mat salt_and_pepper_image(cv::Mat img, int low, int up))
  • 25 // up: used for salt pepper (cv::Mat salt_and_pepper_image(cv::Mat img, int low, int up))
  • 3 // kernel_rows: used for sharpen (cv::Mat sharpen_image(cv::Mat img, cv::Mat kernel))
  • 3 // kernel_cols: used for sharpen (cv::Mat sharpen_image(cv::Mat img, cv::Mat kernel))
  • 0 -1 0 // specify the kernel based on kernel_rows and kernel_cols: used for sharpen (cv::Mat sharpen_image(cv::Mat img, cv::Mat kernel))
  • -1 5 -1 // specify the kernel based on kernel_rows and kernel_cols: used for sharpen (cv::Mat sharpen_image(cv::Mat img, cv::Mat kernel))
  • 0 -1 0 // specify the kernel based on kernel_rows and kernel_cols: used for sharpen (cv::Mat sharpen_image(cv::Mat img, cv::Mat kernel))
  • 5 // pixel_shift: used for shifting pixels (cv::Mat shift_image(cv::Mat img, int shift_type, int pixel_shift))
  • 0.8 // angle: used for rotating the image (cv::Mat rotate_image_angle(cv::Mat img, float angle))
  • 0.05 // percentage_drop: used for dropout (cv::Mat dropout_image(cv::Mat image, float percentage))

Specify the merging algorithm:

• '-vot_merge': exectue the Voting merging algorithm
• '-avg_merge': exectue the Averaging merging algorithm
• '-max_merge': exectue the Maximum merging algorithm

Specify a directory with the images or the video to process:

• '-in_dir images/'
• '-in_dir video_file.avi'

 Specify the output directory to store the detection files (.txt files and images):

• '-save_dets out_dir/'

Specify which fault injection mechanism to use:

• '-fault_injection (-simultaneous_faults)': if '-simultaneous_faults' is specified, all components will have a fault injected in the same operation, otherwise, only one component will have the fault injection

 Specify which dataset to use (only COCO and KITTI datasets are fully supported, as evaluated in the paper):

 '-kitti_dataset': specifies the use of the KITTI dataset. Only Car and Truck classes will be detected, and these will be merged into a single class. Therefore, it will detect only one class: Vehicle

otherwise: it will assume the use of the COCO dataset, where it will only detect Person, Car, Bus, Motorbike and Truck. The vehicle classes will be merged into a single class. Therefore, it will detect the following classes: Person, Vehicle

Default thresholds used:

• Confidence threshold: 0.5
• NMS threshold: 0.5

Downloading the pretrained weights

We have used the pretrained YOLOv4 weights that were trained by the YOLO authors. The weights can be downloaded from the Darknet repository: https://github.com/AlexeyAB/darknet

Compilation steps

Simply navigate to the main folder and run make. You may change some parameters in the Makefile, but do not change the first section, as indicated in the comment. It should be compiled with GPU and OpenCV support.

Datasets evaluated

We have evaluated our solution with two datasets: (i) COCO validation subset, and (ii) KITTI training subset. We have only evaluated those images from COCO that contain Person and Vehicle objects. You can find the COCO images and groundtruths that we have evaluated in the images_COCO.zip and groundtruth_COCO.zip, respectively. You can find the KITTI groudntruths that we have evaluated in the groundtruths_KITTI.zip. You can download the KITTI images from the official website (~12GB): https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d The images evaluated can be downloaded in the Download left color images of object data set (12 GB) section. The zip file from the KITTI website contains the training and test images. We have used the training subset of images since the YOLO model was trained with a different dataset, and the training images have their associated groundtruths that can also be downloaded from their website to calculate the accuracy metrics.

Examples to execute our tool

Examples that show how to execute our tool with the different available parameters. Note that in the following examples, you have to replace the image transformation names by the ones you want to use from the list of implemented transformations. Also, you need to create the output directory of each example (specified after the -save_dets argument).

How to execute with a single component:

./darknet detect cfg/yolov4.cfg yolov4.weights -single_execution -img_transformations none -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -save_dets out_dir/

How to execute with two redundant components (DMR):

./darknet detect cfg/yolov4.cfg yolov4.weights -dmr_execution -img_transformations none hflip -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -save_dets out_dir_DMR/

How to execute with three redundant components (TMR):

./darknet detect cfg/yolov4.cfg yolov4.weights -tmr_execution -img_transformations none hflip right_shift -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -save_dets out_dir_TMR/

How to perform fault injection with a single component:

./darknet detect cfg/yolov4.cfg yolov4.weights -single_execution -img_transformations none -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -fault_injection -save_dets out_dir_FI/

How to perform fault injection with two redundant components (independent faults):

./darknet detect cfg/yolov4.cfg yolov4.weights -dmr_execution -img_transformations none hflip -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -fault_injection -save_dets out_dir_DMR_FI_Indep/

How to perform fault injection with two redundant components (simultaneous faults):

./darknet detect cfg/yolov4.cfg yolov4.weights -dmr_execution -img_transformations none hflip -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -fault_injection -simultaneous_faults -save_dets out_dir_DMR_FI_Sim/

How to perform fault injection with three redundant components (independent faults):

./darknet detect cfg/yolov4.cfg yolov4.weights -tmr_execution -img_transformations none hflip right_shift -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -fault_injection -save_dets out_dir_TMR_FI_Indep/

How to perform fault injection with three redundant components (simultaneous faults):

./darknet detect cfg/yolov4.cfg yolov4.weights -tmr_execution -img_transformations none hflip right_shift -max_merge -img_transformations_cfg img_cfg.txt -in_dir images_COCO/ -fault_injection -simultaneous_faults -save_dets out_dir_TMR_FI_Sim/

How to execute with a single component using the KITTI dataset (change the input directory and add -kitti_dataset, use other parameters as in the previous examples):

./darknet detect cfg/yolov4.cfg yolov4.weights -single_execution -img_transformations none -img_transformations_cfg img_cfg.txt -kitti_dataset -in_dir images_KITTI/ -save_dets out_dir/

How to evaluate a video rather than a directory of images (call detector demo and add cfg/coco.data, use the other parameters as shown in the previous examples):

./darknet detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights -single_execution -img_transformations none -img_transformations_cfg img_cfg.txt -in_dir video.avi -save_dets out_dir_Video/

Adding other image transformations

How to add more image transformations:

1. Increment the number of image transformations and add the name that you want to use in the following array (darknet.c): char *transformations_names[18] = {"none", "hflip", "vflip", "equalize", "gamma", "gaussian_blur", "gaussian_noise", "median_blur", "raise_value", "salt_pepper", "sharpen", "right_shift", "left_shift", "top_shift", "bottom_shift", "anticlockwise_rotate", "clockwise_rotate", "dropout"};
2. If the new image transformation requires a new parameter, modify void init_img_transformation_params(char *filename) in darknet.c to read the parameter from the cfg file. Also, add the parameter to include/DMR_TMR_Library.h and initialize it in src/DMR_TMR_Library.cu
3. Create a function that implements the new image transformation in image_opencv.cpp
4. Call the new transformation accordingly in extern "C" image image_data_augmentation_extended(image img, int image_transformation) in image_opencv.cpp
5. If the new transformation modifies the coordinates of the bounding boxes (e.g., hflip, vflip) or the pixel position (e.g., shifting), you may need to add a code to undo the transformation to be able to use the merging algorithms properly. See void undo_transformation(network *net, int num_output, int img_transformation, image im) in detector.c

How to change the implementation of the custom convolution (in the fault-free or fault injection case)

1. Add the new convolution implementation in DMR_TMR_Library.cu
2. Call the new function accordingly in void custom_gemm_ongpu(...) (in DMR_TMR_Library.cu)

Fault Injection Methodology

Fault injection mechanism implemented:

1. A random layer is selected (code in network_kernels.cu)
2. If -simultaneous_faults is enabled, all components will have a fault injection, otherwise, only one component will have a fault. The component is selected cyclically. You can modify this behavior in detector.c for images, and demo.c for videos (search for variable faulty_comp)
3. A random operation is selected by calculating a random index of the output (code in DMR_TMR_Library.cu, functions: launch_gemm_FI(), launch_gemm_FI_DMR(), launch_gemm_FI_TMR())
4. The fault is injected appropriately in functions: gemm_FI(), gemm_FI_DMR(), and gemm_FI_TMR()

NOTE: The convolutional layer where the fault needs to be injected is replaced by a different custom convolutional layer. This convolution is not optimized, but the performance impact is very small since we only inject faults in 1 layer.

Modify the subset of classes detected

How to modify the classes of the detector / add support for other datasets that detect a different subset of classes:

1. Modify / Comment / Create the function fuse_vehicle_classes() in detector.c (it is also called in demo.c for videos). In this function you can fuse the classes that you choose to. You need to change the best class, best probability, and set the other class probabilities to 0. This increases accuracy in some cases, such as when the specific type of vehicle is not required
2. Modify / Comment / Create the function ignore_not_vehicle_person() in detector.c (it is also called in demo.c for videos). In this function you can define which classes you want to ignore. You need to set the probabilities of those classes to 0

Calculating the Mean Average Precision (mAP)

How to run the mAP script:

python3 mAP_script.py groundtruths/ detections/ json_aux/ results/ dataset

results/ is the output directory, datataset can be COCO or KITTI to evaluate only the relevant classes, as we have done in our evaluation. Otherwise, modify the mAP_script.py and the darknet code as explained in the previous point. json_aux/ is a temporary directory that will be created and deleted by the script.

IMPORTANT: If you use the KITTI Dataset you need to run the KITTI_filter_script.c script before running the mAP script. This will filter out the detections that are in Do Not Care Zones, which are zones that have not been labeled in the KITTI dataset, and detections within these zones need to be ignored for proper evaluation.

Example of calling the KITTI_filter_script script:

./KITTI_filter_script.o input_directory/ output_directory/ groundtruths_KITTI_FULL/

input_directory/ is the directory that contains the YOLO detections (the one specified in the -save_dets when calling darknet), output_directory/ is the output where the filtered detections will be stored, and groundtruths_KITTI/ is the directory where the groundtruths of the KITTI dataset are stored.

IMPORTANT: You need to call this script with the groundtruths_KITTI_FULL which has the labels that contain the Do Not Care Zones.

Example of the mAP calculation with the COCO dataset:

python3 mAP_script.py groundtruth_COCO/ detections/ json_aux/ results/ COCO

Example of the mAP calculation with the KITTI dataset:

./KITTI_filter_script.o dets_KITTI/ dets_filtered_KITTI/ groundtruths_KITTI_FULL/
python3 mAP_script.py groundtruths_KITTI/ dets_filtered_KITTI/ json_aux/ results/ KITTI

IMPORTANT: Note that the KITTI_filter_script needs the groundtruths_KITTI_FULL/ directory, but the mAP needs the groundtruths_KITTI/ directory.