We propose an ultra-low power (ULP) Image Signal Processor (ISP) that performs on-the-fly in-processing frame (de)compression and hierarchical event recognition to exploit the temporal and spatial sparsity in an image sequence to achieve a 16× imaging system energy gain. The ISP is fabricated in 40 nm CMOS and consumes only 170 μW at 5 fps for neural network-based intruder detection and 192× compressed image recording.