The technological environment that supports the learning process tends to be the main data source for Learning Analytics. However, this trend leaves out those parts of the learning process that are not computer-mediated. To overcome this problem, involving additional data gathering techniques such as ambient sensors, audio and video recordings, or even observations could enrich datasets. This paper focuses on how the data extracted from the observations can be integrated with data coming from activity tracking, resulting in a multimodal dataset. The paper identifies the need for theoretical and pedagogical semantics in multimodal learning analytics, and examines the xAPI potential for the multimodal data gathering and aggregation. Finally, we propose an approach for pedagogy-driven observational data identification. As a proof of concept, we have applied the approach in two research works where observations had been used to enrich or triangulate the results obtained for traditional data sources. Through these examples, we illustrate some of the challenges that multimodal dataset may present when including observational data