PromptFusionNet_dataset

Self-made Tri-Modal Dataset. This dataset provides visual, tactile, and textual recordings collected from a robotic grasping platform to support research on multi-modal perception, robot grasping, and task-guided manipulation. It integrates RGB-D visual data, high-resolution tactile force data, and natural language instructions, enabling studies in perception-driven manipulation, cross-modal learning, and robust object recognition under manufacturing uncertainties.

The dataset was collected to validate the proposed tri-modal fusion method described in our paper. It is publicly available to promote reproducibility and facilitate further research in robotic perception and control.

1. Data Acquisition Platform
The data collection platform consists of:
• Robot: UR10 robotic arm (position control mode, 125 Hz)
• Hand: Barrett BH8-282 three-fingered hand with capacitive tactile sensors embedded in fingertips and palm
• Camera: Intel RealSense RGB-D camera mounted on the end effector

The UR10 performs smooth, high-precision grasping trajectories covering approach, contact, and lifting phases. The Barrett hand provides real-time tactile force distribution, while the RealSense camera captures RGB-D images from multiple viewpoints.

2. Object Set
A total of 15 object categories were selected, covering a wide range of materials, surface characteristics, and shapes:
• Materials: plastic, metal, sponge, wood, fabric, ceramic
• Textures: smooth, rough, soft, rigid
• Shapes: regular geometric objects (e.g., cubes, cylinders) and irregular items (e.g., deformable fabric, curved ceramic pieces)

Each object is grasped under multiple angles and varying illumination conditions to simulate real-world sensing degradation.

3. Visual Data
• Sensor: Intel RealSense RGB-D camera
• Resolution: 1920 × 1080 (RGB)
• Operational Range: 0.1 – 10 m
• Preprocessing: RGB images are resized to 224×224×3 for network input
• Calibration:Camera calibration via chessboard pattern. Hand–eye calibration aligns the camera coordinate frame with the UR10 base frame. Ensures spatial consistency between visual and tactile data

4. Tactile Data
• Sensor Type: Capacitive tactile array embedded in fingertips and palm
• Resolution: Each finger has 8×3 sensor cells, signals from all fingers are combined into a global 8×9 tactile map
• Accuracy: ±0.01 N per cell
• Calibration: All tactile sensors were zero-calibrated under no-load conditions to eliminate baseline drift and improve measurement repeatability
• Data format: Raw readings are normalized to the range 0–255 and rearranged according to sensor layout

5. Textual Data
• Source: Natural language instructions provided by operators
• Content: Task-relevant attributes such as material, shape, or functional cues (e.g., “pick up the soft sponge”, “grasp the metal cylinder”)
• Quantity: 100 unique instructions per object category, totaling 1500 distinct text prompts
• Purpose: Serve as semantic guidance to modulate visual and tactile feature extraction, enabling task-aware perception

For details about the dataset or if you wish to use it in your research, please contact: 230248515@seu.edu.cn (English/Chinese)