In this paper, we present a smart video surveillance system based on standard technologies and wired or wireless IP networking. The key novelty of the system is that it protects the privacy of people under surveillance. More specifically, a video analysis module detects regions of interest in the scene by change detection or face detection techniques. It is assumed that these regions contain private and sensitive information. Scrambling is then applied only to the corresponding regions. Furthermore, the amount of distortion can be controlled by restricting scrambling to some resolution levels. As a result, the scene remains visible, but privacy-sensitive information is unidentifiable. The scrambling is controlled by a private encryption key which is kept under control of legal authorities. The latter are therefore the only ones who can grant authorization to unlock the protection and view the whole scene. Therefore, this system successfully addresses the loss of privacy issue associated with video surveillance.