The channel impulse response (CIR) of the wireless propagation channel is influenced by the surrounding environment and thus can be used to retrieve environmental information such as location, the presence of objects, and the speed of objects. In this work, we detect hand gestures based on the complex-valued CIR from an ultra-wideband (UWB) transmission link between one transmitter and two receivers. Thanks to the high path delay resolution due to the wide (500 MHz) bandwidth, we can focus on the channel that is influenced by hand gestures in the sensing area. Using different machine learning and deep learning methods, we learn the features from sequences of CIR snapshots that are correlated to hand gestures and recognize them.