Allosteric communication is essential to protein function, facilitating the dynamic regulation of biological responses through the propagation of structural and dynamic changes between regulatory and effector sites in response to stimuli. Traditional approaches to studying protein allostery often rely on static protein structures or abstract representations involving fully connected interaction graphs, which do not capture the temporal and state-dependent nature of these dynamic systems. Here, we introduce AlloPool, a graph neural network (GNN)-based model that iteratively prunes residue interactions to identify minimal, time-dependent interaction networks that govern long-range structural and dynamic responses to chemical or mechanical stimuli. Using temporal attention and graph aggregation, AlloPool accounts for evolving protein conformations in both molecular dynamics (MD) and steered MD (SMD) simulations to predict MD and SMD trajectories. We validate AlloPool on the Pin-1 protein and the ADGRG1 and B1AR receptors, showcasing its ability to accurately recapitulate protein motions, infer allosteric communication pathways, and identify critical allosteric sites. Additionally, AlloPool identifies force-dependent changes in GAIN domain structure and reconstructs directed information flow under mechanical load. Comparative analyses indicate that AlloPool outperforms existing models in MD and SMD trajectory reconstruction, presenting a new framework for analyzing force- and ligand-induced allosteric motions. This work advances the modeling of allosteric systems and offers broad potential for applications in drug discovery, synthetic biology, and protein engineering.