We present a unifying framework for the development of state-of-the-art reconstruction algorithms in computational optics with a clear separation between the physical (forward model) and signal-related (regularization, incorporation of prior constraints) aspects of the problem. The pillars of our formulation are: (i) an operator algebra with its set of fast linear solvers, (ii) a variational derivation of reconstruction methods, and (iii) a suite of efficient numerical tools for the resolution of large-scale optimization problems. These core technologies are incorporated into a modular software library featuring the key components for the implementation and testing of iterative reconstruction algorithms. The concept is illustrated with concrete examples in 3D deconvolution microscopy and lenseless imaging.