High-speed train operation may cause vibration near track facilities and propagate far through the ground, affecting people's lives, work, and normal use of precision instruments in an urban environment. An efficient numerical method is proposed to calculate the non-uniform ground vibration under a moving high-speed railway load. The theory of stochastic variables is used to describe the soil spatial variability of the non-uniform layered elastic ground, and the coupled 2.5D random finite element method (FEM) is proposed to reduce the computational cost without losing accuracy. Vibration propagation and attenuation of the non-uniform layered ground are investigated and the effect of train speed and soil non-homogeneity are analyzed. Results show that (1) at cross speed and high speed, the homogeneity coefficient of the layered ground has the most important influence on the ground vibration amplitude; (2) the upward acceleration is much larger than the downward acceleration at most speeds, and at cross speed and high speed, the acceleration amplitude decreases with the increase in the homogeneity coefficient; (3) as train speed increases from 60 m/s to 130 m/s, the influencing range of the homogeneity coefficient increases to 10 m from 2 m; and (4) the phenomenon of an in increase in local rebound can be observed in the velocity and acceleration attenuation curve at cross speed when the ground soil has a weaker homogeneity.