We show that explainable neural networks can identify regions of oceanic
variability that contribute predictability on decadal timescales in a
fully coupled Earth system model. The neural networks learn to use
sea-surface temperature anomalies to predict future continental surface
temperature anomalies. We then use a neural network explainability
method called layerwise relevance propagation to infer which oceanic
patterns lead to accurate predictions made by the neural networks. In
particular, regions within the North Atlantic Ocean and North Pacific
Ocean lend the most predictability for surface temperature across
continental North America. We apply the proposed methodology to decadal
variability, although the concept is generalizable to other timescales
of predictability. Furthermore, while our approach focuses on
predictable patterns of internal variability within climate models, it
should be generalizable to observational data as well. Our study
contributes to the growing evidence that interpretable neural networks
are important tools for advancing geoscientific knowledge.