KamLAND-Zen is a liquid scintillator detector searching for neutrinoless double beta decay of Xenon-136. Recently, KamLAND-Zen set world-leading limits on this elusive process. One of the primary challenges of this search is the rejection of backgrounds from radioactive isotopes introduced by cosmic-ray spallation. We developed a state-of-the-art neural network classifier, called KamNet, to reject background events and improve detection sensitivity. However, as we rely more heavily on deep neural networks to play key roles in data analysis, it becomes increasingly important to understand exactly how they work. Here, we take a look at KamNet through the lens of network interpretability. Using Monte Carlo (MC) simulations and experimental data, we present the results of recent studies of the origin of KamNet's rejection power. We find that KamNet has the ability to discern multi-vertex events (one or more gammas in addition to a beta) from single-vertex beta events (only betas). This beta vs beta+gamma discrimination is used to help us ascertain spallation background levels. KamNet's rejection performance for key spallation backgrounds will be presented and we discuss how KamNet can inform us about the types of backgrounds it's rejecting.