Dataset for the paper "DialogueAV: a Dialogue-attended Audiovisual Dataset"

Introduction

This is the official release of the code for DialogueAV: a Dialogue-attended Audiovisual Dataset. Dialogue-AV is a benchmarking dataset with ~258k video clips. Each clip has two dialogue-based descriptions: a Question-Answering Dialogue (QDA) with ten question-answer pairs and a simulated conversation between two "humans" discussing the video.

The dialogues come from human-created captions in SOTA benchmarking datasets and machine-generated captions. We use verified annotations from these top datasets, focusing solely on describing the audiovisual content.

https://github.com/lvilaca16/dialogue-av/blob/main/docs/figures/example_dialogue.png

Annotations in (4) and (5) undergo automatic validation (3) before they are accepted into Dialogue-AV. In the automatic validation step (3), accepted samples must:

1. Include between 5 and 20 dialogue turns.
2. Each dialogue turn must have at least one complete sentence. A complete sentence requires at least 1 subject, predicate, object or noun, and 1 verb; it should end with appropriate punctuation and begin with a named character. Additionally, each complete sentence must contain a minimum of 3 words after removing punctuation (avoid simple sentences as "It rains.").
3. Avoid using the terms "caption(s)" or "dialogue(s)", thereby eliminating references to the original prompt.

For more details about the data generation process, we refer the reader to the (to be published) manuscript.