The virulence of Human Immunodeficiency Virus-1 (HIV-1) is partly determined by viral genetic variation. Finding individual genetic variants affecting virulence is important for our understanding of HIV pathogenesis and evolution of virulence; however, very few have been identified. To this end, within the “Bridging the Evolution and Epidemiology of HIV in Europe” (BEEHIVE) collaboration, we produced whole-genome HIV sequence data for 2294 seroconverters from European countries for a genome-wide association study (GWAS). We considered two phenotypes: (i) set-point viral load (SPVL), the approximately stable viral load from 6 to 24 months after infection, and (ii) the rate of CD4 cell count decline. We developed a GWAS method that corrects for population structure with random effects, accounts for two or more alleles at each locus, and tests for the effect of multiple genetic variants including single-nucleotide polymorphisms (SNPs), k-mers, insertions and deletions, within-host variant frequency, the number of rare point mutations, and drug resistance. We confirmed with this new approach that viral genomes explained 26% [95% CI 17%–35%] of the variance in SPVL, while they explained only 0.9% [0.0%–2.1%] of the variance in the rate of CD4 cell count decline. After correction for multiple testing, among all tested variants, only two significantly explained SPVL: an epitope mutation allowing escape from the host HLA-B57 allele and lowering SPVL by −0.26 log10 copies/ml and an epitope mutation allowing escape from the host HLA-B35 allele and increasing SPVL by +0.22 log10 copies/ml. We attempted to replicate these two large effects in two additional independent datasets together encompassing 2445 seroconverters, with mixed results. Overall, the inferred effects of all SNPs and amino-acid variants weakly correlated (R 2 ranging from 0.08 to 0.87%, P-values from 0.001 to 0.32) between our main dataset and these two additional datasets. Lastly, a lasso regression of phenotypes on genetic variants confirmed the heritability of SPVL and explained up to 6% of variance in SPVL in cross-validation datasets. These findings suggest that HIV SPVL is determined by viral genomes through HLA escape variants with potentially large, host-dependent effects that may not always be detected at the population level and many other variants with effects too weak to reach genome-wide significance in our GWAS.