As machine learning systems move from statistical tools to core societal infrastructure, their trustworthiness has become a primary scientific challenge. This requires a foundational shift from maximizing accuracy to providing formal guarantees on three critical properties: privacy, to protect the confidentiality of user data; robustness, to ensure integrity against malicious participants or data; and unlearning, to provide meaningful user control. This dissertation analyzes these pillars through a unified lens of quantifiable costs---in model performance (utility), system requirements (assumptions), and computation (resources). Our analysis uncovers deep interactions between these trust guarantees, revealing them to be both antagonistic, as when privacy mechanisms hinder robustness, and synergistic, as when robust training provides a foundation for efficient unlearning. It also highlights fundamental separations, showing that unlearning, while related to privacy, is a distinct goal that can be achieved at a significantly lower utility cost. This work therefore provides a foundational map of these interactions, contributing towards a principled science of trustworthy machine learning.