The contribution of host genetic and nongenetic factors to immunological differences in humans remains largely undefined. Here, we generated bacterial-, fungal-, and viral-induced immune transcriptional profiles in an age- and sex-balanced cohort of 1,000 healthy individuals and searched for the determinants of immune response variation. We found that age and sex affected the transcriptional response of most immune-related genes, with age effects being more stimulus-specific relative to sex effects, which were largely shared across conditions. Although specific cell populations mediated the effects of age and sex on gene expression, including CD8+ T cells for age and CD4+ T cells and monocytes for sex, we detected a direct effect of these intrinsic factors for the majority of immune genes. The mapping of expression quantitative trait loci (eQTLs) revealed that genetic factors had a stronger effect on immune gene regulation than age and sex, yet they affected a smaller number of genes. Importantly, we identified numerous genetic variants that manifested their regulatory effects exclusively on immune stimulation, including a Candida albicans-specific master regulator at the CR1 locus. These response eQTLs were enriched in disease-associated variants, particularly for autoimmune and inflammatory disorders, indicating that differences in disease risk may result from regulatory variants exerting their effects only in the presence of immune stress. Together, this study quantifies the respective effects of age, sex, genetics, and cellular heterogeneity on the interindividual variability of immune responses and constitutes a valuable resource for further exploration in the context of different infection risks or disease outcomes.