Pretrained models for text-guided universal MR image synthesis with TUMSyn
Authors/Creators
- 1. School of Biomedical Engineering, Hainan University, Haikou, 570228, China
- 2. School of Biomedical Engineering, ShanghaiTech University, Shanghai, 201210, China
- 3. Shanghai Clinical Research and Trial Center, Shanghai 201210, China
- 4. Department of Radiology, Affiliated Hangzhou First People's Hospital, Xihu University School of Medicine, Hangzhou, 310030, China
- 5. School of Biomedical Engineering and State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, 201210, China
- 6. State key laboratory of digital medical engineering, School of Biomedical Engineering, Hainan University, Haikou, 570228, China
- 7. Key Laboratory of Biomedical Engineering of Hainan Province, One Health Institute, Hainan University, Haikou, 570228, China
- 8. Shanghai United Imaging Intelligence Co. Ltd., Shanghai, 200230, China
Description
TUMSyn is a Text-guided Universal MR image Synthesis framework, which can flexibly generate brain MR images with demanded image contrast and spatial resolution from routinely-acquired scans guided by imaging metadata as text prompt. The model is trained and evaluated on a brain MR database comprising 31,407 3D images with 7 structural MRI modalities from 13 centers.
To effectively align and fuse image-text pairs, TUMSyn is built upon a two-stage training strategy. In the first stage, we pre-trained a text encoder using contrastive learning to extract textual semantic features that are aligned with the corresponding image features from metadata. Built on the pre-trained text encoder, in the second stage, the text encoder is frozen and used to extract prompt features to steer the cross-sequence synthesis.
This repository contains the trained network parameters for both stages. Among them, "checkpoint_CLIP.pt" is the network parameter of the text encoder in stage one, and "checkpoint.pth" is the network parameter of the image synthesis model in stage two.
Specific instructions for how to use them are provided at: https://github.com/Wangyulin-user/TUMSyn.