Multimodal Sensory Learning for Object Manipulation
Description
Multimodal Manipulation Learning Database
The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints.
Dataset
Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data:
- Kuka IIWA 7 Joint data: /iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states
- Allegro hand joint data: /allegro_hand_right/joint_states
- Tekscan sensor recording (tactile force/pressure sensor data on hand): /tekscan/frame
- Audio data (for microphone attached to hand): /audio/audio /audio/audio_info
- Experiment information: /trialInfo
- which contains:
- trial information (motion type, speed, etc.)
- start/stop of different phases of the trials
- which contains:
Motion Types
The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object.
Materials
The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition.
The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13.
Database Setup
The database consists of the data in two formats: annotated ('annotated_bags_mml.zip') and unannotated/numbered filenames ('numbered_bags_mml.zip') datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below).
The annotated filenames dataset ('annotated_bags_mml.zip') consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (.bag
) is saved in the directory, with filename specified ('Date Recorded YYYYMMDD' + '_motion' + '_material' + '_trialID' + '.bag'). Motion Types are: {'vertical', 'rotation'}. Materials are: {'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29.
All data recordings for the vertical motion have filenames: '20210825_vertical_+ 'material' + 'trialID' +'.bag). For the rotation motion, the empty, gummy and rice classes have filenames: '20210826_rotation_+ 'material' + 'trialID' +'.bag). For cornflakes and vitamins classes, the filenames are: '20210913_rotation_+ 'material' + 'trialID' +'.bag).
The numbered/unannotated file dataset ('numbered_bags_mml.zip') consists of the same 300 data files as in the annotated dataset except here the filenames are numbered '{000-299}.bag'. The directory contains a spreadsheet ('annotations.csv') listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}, where {Year, Month, Day} refer to the date that trial data was collected (either 2021/08/25, 2021/08/26, or 2021/09/13).
Files
annotated_bags_mml.zip
Files
(1.6 GB)
Name | Size | Download all |
---|---|---|
md5:60ac6f12924764dc813977c053934c98
|
795.9 MB | Preview Download |
md5:2d590702cae52e7dd344c0800a944f7a
|
795.9 MB | Preview Download |