The development of immune checkpoint inhibitors (ICIs) has revolutionized cancer therapy but only a fraction of patients benefits from this therapy. Model-informed drug development can be used to assess prognostic and predictive clinical factors or biomarkers associated with treatment response. Most pharmacometric models have thus far been developed using data from randomized clinical trials, and further studies are needed to translate their findings into the real-world setting. We developed a tumor growth inhibition model based on real-world clinical and imaging data in a population of 91 advanced melanoma patients receiving ICIs (i.e., ipilimumab, nivolumab, and pembrolizumab). Drug effect was modeled as an ON/OFF treatment effect, with a tumor killing rate constant identical for the three drugs. Significant and clinically relevant covariate effects of albumin, neutrophil to lymphocyte ratio, and Eastern Cooperative Oncology Group (ECOG) performance status were identified on the baseline tumor volume parameter, as well as NRAS mutation on tumor growth rate constant using standard pharmacometric approaches. In a population subgroup (n = 38), we had the opportunity to conduct an exploratory analysis of image-based covariates (i.e., radiomics features), by combining machine learning and conventional pharmacometric covariate selection approaches. Overall, we demonstrated an innovative pipeline for longitudinal analyses of clinical and imaging RWD with a high-dimensional covariate selection method that enabled the identification of factors associated with tumor dynamics. This study also provides a proof of concept for using radiomics features as model covariates.