TransProteus: Predicting 3D shape and properties of materials, liquids, and objects inside transparent containers from images, as XYZ map, code and trained model weight

Code and trained model for predicting 3D shape of transparent vessel and their content from an image.

Should run out of the box

Given an image containing transparent containers, predict the 3d model and 2D segmentation of the vessel content, the vessel itself, and the vessel opening plane.
The prediction is made in the form of an XYZ map that predicts X,Y,Z coordinates per pixel.
The 3D model prediction is independent of the image source and camera used.
The 3D prediction is scale and translation invariant which means that the 3D shape of the object is predicted, but the scale of the model and its position (translation) is arbitrary.
See paper [Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI datase]
(https://arxiv.org/pdf/2109.07577.pdf) for more details

The same code with train model included (run out of the box) could be download from: [1](https://icedrive.net/0/42jdtddE6S), [2](https://e.pcloud.link/publink/show?code=XZEI55ZNk2cL139W78o1FMk35VeG5j9Jzck)

These code was trained using the TransProteus dataset that can be downloaded from: [Full Dataset 1](https://e.pcloud.link/publink/show?code=kZfx55Zx1GOrl4aUwXDrifAHUPSt7QUAIfV),  [Full DataSet Link2](https://icedrive.net/1/6cZbP5dkNG), [Subset](https://zenodo.org/record/5508261#.YUGsd3tE1H4)
Figure 1. Structure of the net for predicting 3D and 2D segments from the image. XYZ maps are represented as and BGR image with blue, red, and green channels corresponds to the X,Y,Z coordinates of the pixel.

Videos can of results:
https://www.youtube.com/watch?v=EbVvEYespII
https://www.youtube.com/watch?v=zWJJyjmsBko
https://www.youtube.com/watch?v=JC55VmYHB_s

Requirements

Hardware

For using the trained net for prediction, no specific hardware is needed, but the net will run much faster on Nvidia GPU.

For training the net, an Nvidia GPU is needed (the net was trained on RTX 3090)

Setup

Create a environment with the required dependencies (Pytorch, torchvision, scipy and OpenCV, Open3D): conda env create -f environment.yml

Software:

This network was run with Python 3.88 Anaconda with Pytorch 1.8 and OpenCV* package.

• Installing opencv for conda can usually be done using: pip install opencv-python or conda install opencv

Prediction: running the trained net on a single image

1. Open RunOnImage.py
2. Set image path to InputImage parameter
3. Run script to get prediction display on the screen. Additional optional parameters: UseGPU: decide whether to use GPU hardware (True/False). Other optional parameters are in the input parameters section of the script.