Humans and animals can remember how long ago specific events happened. Little is known about the neural mechanisms that enable remembering the “when” of memories stored for long durations in the episodic memory system – in contrast to interval-timing on the order of seconds and minutes. Based on a systematic exploration of neural coding, association and retrieval schemes, we develop model classes that span the space of possible mechanisms for the reconstruction of the time of past events. In concrete examples we show how network architecture, Hebbian plasticity, synaptic pruning or systems consolidation allow the retrieval of the time of past events. In a simulation, we demonstrate how these mechanisms would enable food-caching animals such as corvids to remember what they cached, where, and how long ago. To dissociate different hypotheses, we propose three kinds of novel, non-verbal experiments that can be run with humans and animals. Our simulations predict the experimental results for different classes of models. Our study shows that remembering the “when” can be implemented by many biologically plausible mechanisms and that carefully designed experiments are needed to pin down the actual neural implementation of the memory for the time of past events in different species.