Artificial intelligence and machine learning algorithms have become ubiquitous. Although they offer a wide range of benefits, their adoption in decision-critical fields is limited by their lack of interpretability, particularly with textual data. Moreover, with more data available than ever before, it has become increasingly important to explain automated predictions. Generally, users find it difficult to understand the underlying computational processes and interact with the models, especially when the models fail to generate the outcomes or explanations, or both, correctly. This problem highlights the growing need for users to better understand the models' inner workings and gain control over their actions. This dissertation focuses on two fundamental challenges of addressing this need. The first involves explanation generation: inferring high-quality explanations from text documents in a scalable and data-driven manner. The second challenge consists in making explanations actionable, and we refer to it as critiquing. This dissertation examines two important applications in natural language processing and recommendation tasks. First, we present two models for extracting high-quality and personalized justifications from text documents under the paradigm of selective rationalization - a binary selection of input features from which the model computes the outcome. We propose the concept of multi-dimensional rationales and emphasize that a single overall selection does not accurately capture the multi-faceted nature of useful rationales. The first method relies on multi-task learning and learns the rationales in an unsupervised manner, with no prior in the data. Inspired by the role of concept-based thinking in human reasoning, our second model is a generalization: it assumes that only one label is observed. We empirically demonstrate the efficiency of the models in terms of explanation and predictive performance as well as their scalability. It opens new doors for applications, such as recommendation or summarization. Second, we use the high-level idea of multi-dimensional rationalization to mine massive corpora and construct large personalized justification datasets. We show that human users significantly prefer our explanations over those produced by state-of-the-art methods. Last, we propose two conversational explainable recommendation systems. The first explains a user rating by inferring a set of keyphrases. Conditioned on them, it generates an abstractive personal justification. We allow users to interact iteratively with the explanations and refine the recommendation through an unsupervised critiquing method. Our second model is based on multimodal modeling and self-supervision. It enables fast and efficient multi-step critiquing. Using real-world datasets, we show that our models exhibit better performance than state-of-the-art models in terms of recommendation, explanation, and critiquing. Overall, we demonstrate that interpretability does not come at the cost of reduced performance in two consequential applications. Our framework is applicable to other fields as well. This dissertation presents an effective means of closing the gap between promise and practice in artificial intelligence.