In the last decade, deep neural networks have achieved tremendous success in many fields of machine learning. However, they are shown vulnerable against adversarial attacks: well-designed, yet imperceptible, perturbations can make the state-of-the-art deep neural networks output incorrect results. Understanding adversarial attacks and designing algorithms to make deep neural networks robust against these attacks are key steps to building reliable artificial intelligence in real-life applications.

In this thesis, we will first formulate the robust learning problem. Based on the notations of empirical robustness and verified robustness, we design new algorithms to achieve both of these types of robustness. Specifically, we investigate the robust learning problem from the optimization perspectives. Compared with classic empirical risk minimization, we show the slow convergence and large generalization gap in robust learning. Our theoretical and numerical analysis indicates that these challenges arise, respectively, from non-smooth loss landscapes and model's fitting hard adversarial instances. Our insights shed some light on designing algorithms for mitigating these challenges.

Robust learning has other challenges, such as large model capacity requirements and high computational complexity. To solve the model capacity issue, we combine robust learning with model compression. We design an algorithm to obtain sparse and binary neural networks and make it robust. To decrease the computational complexity, we accelerate the existing adversarial training algorithm and preserve its performance stability.

In addition to making models robust, our research provides other benefits. Our methods demonstrate that robust models, compared with non-robust ones, usually utilize input features in a way more similar to the way human beings use them, hence the robust models are more interpretable. To obtain verified robustness, our methods indicate the geometric similarity of the decision boundaries near data points. Our approaches towards reliable artificial intelligence can not only render deep neural networks more robust in safety-critical applications but also make us better aware of how they work.