We present the Personalized News (PEN) recommender systems framework, currently in use by a newspaper website to evaluate various algorithms for news recommendations. We briefly describe its system architecture and related components. We show how a researcher can easily evaluate different algorithms thanks to a web-based interface. Finally, we discuss important factors to take into account when conducting online evaluation, and report on our experience when deploying recommendations on a live-traffic website.