We present optical absorption spectra from the ultraviolet to the visible for size selected neutral Ag-n clusters (n = 5-120) embedded in solid Ne. We compare the spectra to time-dependent density functional calculations (TDDFT) that address the influence of the Ne matrix. With increasing size, several highly correlated electron excitations gradually develop into a single surface plasmon. Its energy is situated between 3.9 and 4.1 eV and varies with size according to the spherical electronic shell model. The plasmon energy is highest for clusters with atom numbers fully filling states with the lowest radial quantum number (e.g. 1s, 1p, 1d,...). TDDFT calculations for clusters with several candidate geometrical structures embedded in Ne show excellent agreement with the experimental data, demonstrating that the absorption bands depend only weakly on the exact structure of the cluster.