The quality and quantity of social experience is fundamental to an individual's health and well-being. Early life stress is known to be an important factor in the programming of the social brain that exerts detrimental effects on social behaviors. The peri-adolescent period, comprising late childhood and adolescence, represents a critical developmental window with regard to the programming effects of stress on the social brain. Here, we discuss social behavior and the physiological and neurobiological consequences of stress during peri-adolescence in the context of rodent paradigms that model human adversity, including social neglect and isolation, social abuse, and exposure to fearful experiences. Furthermore, we discuss peri-adolescent stress as a potent component that influences the social behaviors of individuals in close contact with stressed individuals and that can also influence future generations. We also discuss the temporal dynamics programmed by stress on the social brain and debate whether social behavior alterations are adaptive or maladaptive. By revising the existing literature and defining open questions, we aim to expand the framework in which interactions among peri-adolescent stress, the social brain, and behavior can be better conceptualized.