Website fingerprinting (WF) attacks can compromise a user’s online privacy, by learning network traffic patterns generated by websites through machine learning (ML) techniques. Such attacks remain unaffected by encryption and even defeat anonymity services such as Tor. They thereby make it possible to determine the browsing behaviour of a user, which can lead to tracking and surveillance. In this project, we study the security of ALPaCA, a server-side WF defence which protects webpages at the application layer. Traditionally, evaluating the security of a WF defence is done at the network level, against state-of-the-art WF attacks. We on the other hand chose to measure the security of ALPaCA at the application level. While doing so is possible for ALPaCA, it is in general not the case for other WF defences that operate at the network layer. To quantify security, we estimate the Bayes error, which is the smallest probability of error achievable by a WF adversary for any classification algorithm they may use. We then show that evaluation at the application level i) gives consistent security results with the ones obtained at the network level, ii) gives stronger security guarantees, iii) is more efficient from a data collection perspective and therefore allows us to iv) compare the performance of the three variants of ALPaCA for several choices of parameters.