The broad availability of mobile computing devices has prompted the apparition of social media applications that support teaching and learning. However, so far, there is conflicting evidence as to whether the benefits such applications provide in terms of engagement and interaction, outweigh their potential cost as a source of distraction. To help in clarifying these issues, the present paper presents a case study on the impact of using SpeakUp (an app aimed at promoting student participation through anonymous chatrooms) in an authentic face-to-face learning scenario. Concretely, we focus on the connection between SpeakUp and the student engagement, distraction, social interaction, and the influence of the teachers' style. Our findings highlight that SpeakUp favored students' engagement and social interaction, but they also point towards its limitations in keeping students communicating about content relevant to the course.