Attention maps for atmorep: studying large-scale representation learning of atmospheric dynamics

With the escalating threat of climate change, innovative technological solutions have become imperative. In this light, we introduce "AtmoRep," a cutting-edge, large-scale transformer designed specifically for the representation learning of atmospheric dynamics. While transformers have revolutionized numerous domains, their introduction to the field of weather modeling is relatively nascent. Beyond merely achieving enhanced precision, our transformer model demonstrates robustness, especially in zero-shot tasks. However, with the power of these models comes the pressing need for interpretability. This paper delves into the generation, organization, and interpretation of attention maps, a cornerstone of transformer architectures. We tackle the technical challenges associated with extracting these attention maps and the more abstract difficulty of their organization. In doing so, we hope to foster a more transparent and scientifically rigorous methodology in large-scale weather modeling.