Long-term monitoring of real-life physical activity (PA) using wearable devices is increasingly used in clinical and epidemiological studies. The quality of the recorded data is an important issue, as unreliable data may negatively affect the outcome measures. A potential source of bias in PA assessment is the non-wearing of a device during the expected monitoring period. Identification of non-wear time is usually performed as a pre-processing step using data recorded by the accelerometer, which is the most common sensor used for PA analysis algorithms. The main issue is the correct differentiation between non-wear time, sleep time, and sedentary wake time, especially in frail older adults or patient groups. Based on the current state of the art, the objectives of this study were to (1) develop robust non-wearing detection algorithms based on data recorded with a wearable device that integrates acceleration and temperature sensors; (2) validate the algorithms using real-world data recorded according to an appropriate measurement protocol. A comparative evaluation of the implemented algorithms indicated better performances (99%, 97%, 99%, and 98% for sensitivity, specificity, accuracy, and negative predictive value, respectively) for an event-based detection algorithm, where the temperature sensor signal was appropriately processed to identify the timing of device removal/non-wear.