Conditioned Image Retrieval for Fashion using Contrastive Learning and CLIP-based Features

Building on the recent advances in multimodal zero-shot represen-
tation learning, in this paper we explore the use of features obtained
from the recent CLIP model to perform conditioned image retrieval.
Starting from a reference image and an additive textual description
of what the user wants with respect to the reference image, we
learn a Combiner network that is able to understand the image
content, integrate the textual description and provide combined
feature used to perform the conditioned image retrieval. Starting
from the bare CLIP features and a simple baseline, we show that
a carefully crafted Combiner network, based on such multimodal
features, is extremely effective and outperforms more complex state
of the art approaches on the popular FashionIQ dataset.