Towards Neural Scaling Laws for Foundation Models on Temporal Graphs

Description

Abstract

The field of temporal graph learning aims to learn from evolving network data to forecast future interactions. Given a collection of observed temporal graphs, is it possible to predict the evolution of an unseen network from the same domain?
To answer this question, we first present the Temporal Graph Scaling(TGS) dataset, a large collection of temporal graphs, consisting of eighty-four ERC20 token transaction networks collected from 2017 to 2023. Next, we evaluate the transferability of Temporal Graph Neural Networks(TGNNs) for the temporal graph property prediction task by pre-training on a collection of up to sixty-four token transaction networks and then evaluate their downstream performance on twenty unseen token types. We observe that the neural scaling law observed in NLP and Computer Vision also applies in temporal graph learning where pre-training on more networks and more parameters leads to better downstream performance. To the best of our knowledge, this is the first time that the transferability of temporal graphs has been shown empirically. Notably, on the downstream token networks, the largest pre-trained model outperforms fine-tuned TGNNs on thirteen unseen test networks. Therefore, we believe this is a promising first step towards building foundation models for temporal graphs. The code and datasets are publicly available at https://github.com/benjaminnNgo/ScalingTGNs under MIT licence.