With the wide availability of mobile devices and the growing interest in social media, numerous applications have emerged to support student engagement in the classroom. There is conflicting evidence, however, on whether the engagement benefits of such applications outweigh their potential cost as a source of disaffection. To investigate these issues, this paper presents a case study on the usage of a social media app (SpeakUp) during a semester-long face-to-face university course, and its relations with the context and the learning outcomes. In this mixed-methods study, we gathered data from multiple sources (video recordings of the lessons, SpeakUp logs and messages, student questionnaires and course assessments) in order to extract self-reported and observable behavioral and emotional indicators. Our findings reveal that simple measures of behavioral engagement were insufficient to predict academic performance. Nevertheless, our models significantly improved using relatively simple and unobtrusive indicators of both behavioral and emotional engagement and disaffection. This study also points out that the mere introduction of social media in educational settings does not guarantee a positive impact on learning. To promote an effective use of social media in the classroom (raising engagement while avoiding disaffection), teachers need to design the learning activities aligning the use of social media with the learning goals.