Multimodal Sensory Learning for Object Manipulation

Multimodal Manipulation Learning Database

The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints. 

Dataset

Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data:

• Kuka IIWA 7 Joint data: /iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states
• Allegro hand joint data: /allegro_hand_right/joint_states
• Tekscan sensor recording (tactile force/pressure sensor data on hand): /tekscan/frame
• Audio data (for microphone attached to hand): /audio/audio /audio/audio_info
• Experiment information: /trialInfo
  • which contains:
    • trial information (motion type, speed, etc.)
    • start/stop of different phases of the trials

Motion Types

The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object.

Materials

The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition.

The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13.

Database Setup

The database consists of the data in two formats: annotated ('annotated_bags_mml.zip') and unannotated/numbered filenames ('numbered_bags_mml.zip') datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below).

The annotated filenames dataset ('annotated_bags_mml.zip') consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (.bag) is saved in the directory, with filename specified ('Date Recorded YYYYMMDD' + '_motion' + '_material' + '_trialID' + '.bag'). Motion Types are: {'vertical', 'rotation'}.  Materials are: {'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29.

All data recordings for the vertical motion have filenames: '20210825_vertical_+ 'material' + 'trialID' +'.bag). For the rotation motion, the empty, gummy and rice classes have filenames: '20210826_rotation_+ 'material' + 'trialID' +'.bag). For cornflakes and vitamins classes, the filenames are: '20210913_rotation_+ 'material' + 'trialID' +'.bag).

The numbered/unannotated file dataset ('numbered_bags_mml.zip') consists of the same 300 data files as in the annotated dataset except here the filenames are numbered '{000-299}.bag'. The directory contains a spreadsheet ('annotations.csv') listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}, where {Year, Month, Day} refer to the date that trial data was collected (either 2021/08/25, 2021/08/26, or 2021/09/13).