We propose an information theoretic model that unifies a wide range of existing information theoretic signal processing algorithms in a compact mathematical framework. It is mainly based on stochastic processes, Markov chains and error probabilities. The proposed framework will allow us to discuss revealing analogies and differences between several well known algorithms and to propose interesting extensions resulting directly from our formalism. We will then describe how the theory can be applied to the rapidly emerging field of multi-modal signal processing: we will show how our framework can be efficiently used for multi-modal medical image processing and for joint analysis of multi-media sequences (audio and video).