Action understanding is a fundamental problem in artificial intelligence, as intelligent systems must infer purposeful behavior from observations that are often temporally structured and incomplete. Traditionally, this problem has been addressed within the domain of computer vision; however, multi-modal observations provide richer and more informative signals. In real-world scenarios, actions cannot be reliably recognized from visual appearance alone. Instead, effective action understanding requires integrating multiple modalities, including body motion, language priors, semantic information, object interactions, and scene context, all of which provide complementary cues. This work studies multi-modal action understanding through the combined lens of datasets, benchmarks, and models, with a focus on interaction-rich human activities. We show that progress depends on capturing motion and interaction with sufficient fidelity. To support this, we introduce HOISDF, a geometry-aware method for monocular 3D hand-object pose estimation under severe occlusion, demonstrating the value of implicit structural priors for recovering fine-grained manipulation dynamics. We then present EPFL-Smart-Kitchen, an ego-exo multi-modal dataset that brings together multi-view RGB-D video, egocentric video, gaze, inertial signals, and 3D kinematics for realistic action analysis in complex kitchen environments. Building on this foundation, we develop benchmarks that move beyond coarse action classification. EPFL-Smart-Kitchen proposes multi-modal action recognition, action segmentation, semantic text-to-motion generation, and video question answering, while LLaVAction challenges multi-modal large language models on fine-grained action understanding. These benchmarks highlight remaining difficulties in action disambiguation, long-horizon temporal reasoning, and spatial grounding. To address these challenges, we propose an action-aware multi-modal large language modeling pipeline and AdaptToken, an entropy-based adaptive token selection method that improves long-video reasoning by allocating computation more effectively over time. Overall, this thesis shows that progress in multi-modal action understanding depends not only on better-suited models, but also on the interplay between data collection, benchmark design and multi-modal integration. By improving how actions are captured, tested, and modeled, we aim to make multi-modal action understanding more robust and informative for future intelligent systems.