A meso-scale numerical model for simulating damage in concrete due to ASR is presented. It accounts for the internal structure of concrete including aggregates and mortar. ASR sites are explicitly represented. This model is extended to account for viscoelastic effects in the mortar. An ASR product expansion law is based on the phenomenological reaction extent equation. The model is calibrated on free ASR expansion experiments. Mortar is modelled both as elastic- and viscoelastic-brittle material. In case of the viscoelastic-brittle mortar, simulations result in a lower level of damage and lower stiffness reduction than in the elastic-brittle case. However, the macroscopic expansion becomes lower than the experimental values. This suggests that to reach sufficient macroscopic strain, a larger damage extent within aggregates is necessary.