Supercooled liquids are characterized by their fragility: The slowing down of the dynamics under cooling is more sudden and the jump of specific heat at the glass transition is generally larger in fragile liquids than in strong ones. Despite the importance of this quantity in classifying liquids, explaining what aspects of the microscopic structure controls fragility remains a challenge. Surprisingly, experiments indicate that the linear elasticity of the glass-a purely local property of the free energy landscape-is a good predictor of fragility. In particular, materials presenting a large excess of soft elastic modes, the so-called boson peak, are strong. This is also the case for network liquids near the rigidity percolation, known to affect elasticity. Here we introduce a model of the glass transition based on the assumption that particles can organize locally into distinct configurations that are coupled spatially via elasticity. The model captures the mentioned observations connecting elasticity and fragility. We find that materials presenting an abundance of soft elastic modes have little elastic frustration: Energy is insensitive to most directions in phase space, leading to a small jump of specific heat. In this framework strong liquids turn out to lie the closest to a critical point associated with a rigidity or jamming transition, and their thermodynamic properties are related to the problem of number partitioning and to Hopfield nets in the limit of small memory.