Human gaze plays a crucial role in communication and social interaction. Many recent studies have focused on predicting the 2D pixel location of a person's gaze target in an image. However, this approach has limitations when it comes to studying gaze for downstream applications that require analysis of higher-level social gaze behaviors. Previous works have post-processed the predicted 2D gaze target for social gaze prediction, however, we show that this approach is insufficient. Our proposed method jointly predicts the gaze target and social gaze behaviour, explicitly incorporating people interaction for state of the art results on three social gaze tasks - looking at heads, mutual gaze and shared attention. Additionally, we introduce evaluation protocols for these tasks, presenting a promising avenue for future research in gaze behavior analysis.