Nowadays, the ability to create panoramic photographs is included with most of the commercial digital cameras. The principle is to shoot several pictures and stitch them together to build a panorama. To ensure the quality of the final image, the different pictures have to be perfectly aligned and the colors of the images should match. While the alignment of images has received a lot of attention from the computer vision community, the mismatch in colors was often ignored and handled using smooth transitions from one picture to the next to mask the mismatch. This paper presents a method to simultaneously estimate the alignment of the pictures and the color transformation between them. By estimating the color transformation from the scene to the pixels, the method is able to remove the mismatch in colors of the different images, and thus leads to better image quality.