Published October 24, 2026 | Version v2

CausalHyperGraph: Typed Hypergraph Neural Networks for Higher-Order Causal Reasoning over Narrative Event Structures

Description

School of Information Science and Engineering, Shandong Institute of Petroleum and Chemical Technology

Abstract

Long-form video question answering requires models to track events, character relationships, and motive evolution across narrative videos spanning tens of minutes to hours, and to reason over cross-event dependencies. Existing hierarchical tree-based or ordinary graph-based approaches typically represent event relations as binary edges. While this facilitates retrieval and propagation, it tends to weaken the joint conditioning of event groups in higher-order narrative causal relationships such as many-to-one and one-to-many causality. To address this limitation, this paper proposes CausalHyperGraph, a typed hypergraph neural network framework for script-subtitle-enhanced long-form video question answering. The framework first segments script-subtitle aligned text into event nodes and constructs many-cause hyperedges, many-effect hyperedges, and character co-occurrence hyperedges. It then propagates higher-order relational information through type-aware hypergraph convolution and employs a hyperedge-aware reasoner to retrieve question-relevant sub-hypergraph evidence for scoring candidate answers. Experiments on the StoryVideoQA dataset demonstrate that CausalHyperGraph achieves an average accuracy of 93.8% across fourteen fine-grained topics, representing a 7.3 percentage point improvement over PlotTree. On the manually annotated MTO/OMT multi-causal specialized test sets, improvements of 18.6 and 17.8 percentage points are achieved, respectively. Ablation studies further demonstrate that the hypergraph structure, type-aware propagation, and hyperedge-aware retrieval provide complementary contributions to performance gains.

Keywords:

script-subtitle-enhanced video understanding, hypergraph neural networks, typed hyperedges, narrative causal reasoning, higher-order relation modeling

Files

Zenodo_CausalHyperGraph.pdf

Files (2.0 MB)

Name Size
md5:c12559626a304b0a02057b2098d6d3c0
2.0 MB Preview Download

Additional details

References

  • [1] Cheng, C., Guan, J., Wu, W., & Yan, R. (2025). Scaling video-language models to 10k frames via hierarchical differential distillation. CoRR, abs/2504.02438.
  • [2] Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., … & Bing, L. (2024). Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. CoRR, abs/2406.07476.
  • [3] Chien, E., Pan, C., Peng, J., & Milenkovic, O. (2022). You are allset: A multiset function framework for hypergraph neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net.
  • [4] Choi, J., Lee, S., Chu, J., Choi, M., & Kim, H. J. (2024). vid-tldr: Training free token merging for light-weight video transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 (pp. 18771–18781). IEEE.
  • [5] Choi, S., On, K.-W., Heo, Y.-J., Seo, A., Jang, Y., Lee, M., & Zhang, B.-T. (2021). Dramaqa: Character-centered video story understanding with hierarchical QA. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021 (pp. 1166–1174). AAAI Press.
  • [6] Curtis, K., Awad, G., Rajput, S., & Soboroff, I. (2020). HLVU: A new challenge to test deep understanding of movies the way humans do. In C. Gurrin et al. (Eds.), Proceedings of the 2020 International Conference on Multimedia Retrieval, ICMR 2020 (pp. 355–361). ACM.
  • [7] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of NAACL-HLT 2019 (pp. 4171–4186). Association for Computational Linguistics.
  • [8] Feng, Y., You, H., Zhang, Z., Ji, R., & Gao, Y. (2019). Hypergraph neural networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019 (pp. 3558–3565). AAAI Press.
  • [9] Fey, M., & Lenssen, J. E. (2019). Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
  • [10] Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021). VIOLET: End-to-end video-language transformers with masked visual-token modeling. CoRR, abs/2111.12681.
  • [11] He, B., Li, H., Jang, Y. K., Jia, M., Cao, X., Shah, A., … & Lim, S.-N. (2024). MA-LMM: Memory-augmented large multimodal model for long-term video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 (pp. 13504–13514). IEEE.
  • [12] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., & Wang, L. (2022). Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net.
  • [13] Huang, J., & Yang, J. (2021). Unignn: A unified framework for graph and hypergraph neural networks. In Z.-H. Zhou (Ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021 (pp. 2563–2569). ijcai.org.
  • [14] Jin, P., Takanobu, R., Zhang, W., Cao, X., & Yuan, L. (2024). Chat-univi: Unified visual representation empowers large language models with image and video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 (pp. 13700–13710). IEEE.
  • [15] Lei, J., Yu, L., Bansal, M., & Berg, T. L. (2018). TVQA: Localized, compositional video question answering. In E. Riloff et al. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 1369–1379). Association for Computational Linguistics.
  • [16] Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2019). TVQA+: Spatio-temporal grounding for video question answering. CoRR, abs/1904.11574.
  • [17] Lei, J., Berg, T. L., & Bansal, M. (2023). Revealing single frame bias for video-and-language learning. In A. Rogers, J. L. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 (pp. 487–507). Association for Computational Linguistics.
  • [18] Li, D., Li, J., Li, H., Niebles, J. C., & Hoi, S. C. H. (2022). Align and prompt: Video-and-language pre-training with entity prompts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 (pp. 4943–4953). IEEE.
  • [19] Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., … & Qiao, Y. (2024). Mvbench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 (pp. 22195–22206). IEEE.
  • [20] Li, X., Wang, Y., Yu, J., Zeng, X., Zhu, Y., Huang, H., … & Wang, L. (2025). Videochat-flash: Hierarchical compression for long-context video modeling. CoRR, abs/2501.00574.
  • [21] Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., & Yuan, L. (2023a). Video-llava: Learning united visual representation by alignment before projection. CoRR, abs/2311.10122.
  • [22] Lin, K., Ahmed, F., Li, L., Lin, C.-C., Azarnasab, E., Yang, Z., … & Wang, L. (2023b). MM-VID: Advancing video understanding with gpt-4v(ision). CoRR, abs/2310.19773.
  • [23] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., … & Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  • [24] Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. In The Seventh International Conference on Learning Representations, ICLR 2019. OpenReview.net.
  • [25] Maaz, M., Rasheed, H. A., Khan, S. H., & Khan, F. S. (2023). Video-chatgpt: Towards detailed video understanding via large vision and language models. CoRR, abs/2306.05424.
  • [26] OpenAI. (2023). Chatgpt: A language model for conversational ai (Technical Report No. 40). OpenAI.
  • [27] OpenAI. (2025). GPT-5.2 technical report. OpenAI.
  • [28] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., … & Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, NeurIPS 2019 (pp. 8024–8035).
  • [29] Ren, S., Yao, L., Li, S., Sun, X., & Hou, L. (2024). Timechat: A time-sensitive multimodal large language model for long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 (pp. 14313–14323). IEEE.
  • [30] Shen, Y., Fu, C., Dong, S., Wang, X., Zhang, Y., Chen, P., … & Sun, X. (2025). Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuracy. CoRR, abs/2502.05177.
  • [31] Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., … & Zhao, B. (2025). Video-xl: Extra-long vision language model for hour-scale video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025 (pp. 26160–26169). Computer Vision Foundation / IEEE.
  • [32] Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., & Fidler, S. (2016). Movieqa: Understanding stories in movies through question-answering. In *2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016* (pp. 4631–4640). IEEE Computer Society.
  • [33] Team, G. (2023). Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805.
  • [34] Team, Q. (2025). Qwen3-vl technical report. CoRR, abs/2511.21631.
  • [35] Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., & Bansal, M. (2025). Videotree: Adaptive tree-based video representation for LLM reasoning on long videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025 (pp. 3272–3283). Computer Vision Foundation / IEEE.
  • [36] Wu, Z., Liu, Z., Chen, A., Zhang, J., Li, R., Ge, H., … & Liang, C. (2026). Storyvideoqa: Scaling deep video understanding with a large-scale, multi-genre and auto-generated dataset. International Journal of Computer Vision, 134(6), 308.
  • [37] Xu, H., Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., … & Zhou, J. (2023). mplug-2: A modularized multi-modal foundation model across text, image and video. CoRR, abs/2302.00402.
  • [38] Yadati, N., Nimishakavi, M., Yadav, P., Nitin, V., Louis, A., & Talukdar, P. P. (2019). Hypergcn: A new method for training graph convolutional networks on hypergraphs. In Advances in Neural Information Processing Systems 32, NeurIPS 2019 (pp. 1509–1520).
  • [39] Yu, S., Cho, J., Yadav, P., & Bansal, M. (2023). Self-chained image-language model for video localization and question answering. In Advances in Neural Information Processing Systems 36, NeurIPS 2023.
  • [40] Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., … & Zhao, D. (2025). Videollama 3: Frontier multimodal foundation models for image and video understanding. CoRR, abs/2501.13106.
  • [41] Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., & Bertasius, G. (2024a). A simple LLM framework for long-range video question-answering. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Proceedings of EMNLP 2024 (pp. 21715–21737). Association for Computational Linguistics.
  • [42] Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., & Bertasius, G. (2024b). A simple LLM framework for long-range video question-answering. In Proceedings of EMNLP 2024 (pp. 21715–21737). Association for Computational Linguistics.