Models for "Lightweight Multitask Learning for Robust JND Prediction using Latent Space and Reconstructed Frames"
Authors/Creators
-
1.
Tampere University
-
2.
University of Tehran
-
3.
University of Ottawa
Description
This repository contains the trained neural network (TensorFlow) models, associated with the paper "Lightweight Multitask Learning for Robust JND Prediction using Latent Space and Reconstructed Frames", published in IEEE T-CSVT 2024. The code associated with these models can be accessed through the paper's GitHub page.
Paper
The paper is available on IEEEXplore and a preprint is available on TechArxiv.
Requirements
- Tensorflow
- FFmpeg
Dataset
Our evaluation is conducted on VideoSet and MCL-JCI datasets.
Usage
Our pretrained models are capable of predicting JND values, and they can also be employed for training on a custom dataset.
Note: The dataset used for training and testing should have such a structure.
- rootdir/
- train/
- img#1
- ...
- JND-Levels.txt (a file containing the 3 JND levels per image: first column for the first JND, second column for the second JND, and third column for the third JND level)
- valid/
- img#1
- ...
- JND-Levels.txt (a file containing the 3 JND levels per image: first column for the first JND, second column for the second JND, and third column for the third JND level)
- test/
- img#1
- ...
- jnd1train/
- img#1
- ...
- jnd1valid/
- img#1
- ...
- jnd2train/
- img#1
- ...
- jnd2valid/
- img#1
- ...
- jnd3train/
- img#1
- ...
- jnd3valid/
- img#1
- ...Testing
For prediction with LAT or REC model, the following commands can be used.
python3 [LAT.py or REC.py] test --jnd_value [JND1 or JND2 or JND3] --data_dir "Path-to-the-rootdir/" --model_weights_path "Path-to-the-pretrained-model/" --result_path "Path-to-save-test-results/" --JND_Recon_Models_Path "Path-to-the-pretrained-JND-Reconstruction-models/"For prediction with E2E-LAT or E2E-REC model, the following commands can be used.
python3 [E2ELAT.py or E2EREC.py] test --jnd_value [JND1 or JND2 or JND3] --data_dir "Path-to-the-rootdir/" --model_weights_path "Path-to-the-pretrained-model/" --result_path "Path-to-save-test-results/" --ImgReconstrution_Model_Path "Path-to-the-pretrained-Img-Reconstruction-models/"For prediction with MJ-LAT or MJ-REC model, the following commands can be used.
python3 [MJLAT.py or MJREC.py] test --data_dir "Path-to-the-rootdir/" --model_weights_path "Path-to-the-pretrained-model/" --result_path "Path-to-save-test-results/" --JND_Recon_Models_Path "Path-to-the-pretrained-JND-Reconstruction-models/"
More details about the associated codes can be found on the github page: https://github.com/sanaznami/MTL_JND
Files
FALCON_IEEETCSVT2024_WP3_v1.0.zip
Files
(603.3 MB)
| Name | Size | Download all |
|---|---|---|
|
md5:d673ca77be46752be2e72357fbbad733
|
603.3 MB | Preview Download |
Additional details
Related works
- Is described by
- Journal: 10.1109/TCSVT.2024.3389988 (DOI)
Funding
Software
- Repository URL
- https://github.com/sanaznami/MTL_JND
- Programming language
- Python