The room temperature stability of nanometer-sized silver clusters in silica matrices has been investigated by following the temporal evolution of their surface plasmon absorption. Ag clusters in SiO2 were prepared by either annealing silica samples doped with atomic silver or by co-deposition of preformed clusters of defined size. Clusters were found to be unstable at ambient conditions on a time scale of days to weeks, independent of preparation conditions. The disappearance of the plasmonic resonance is explained by successive oxidation of the clusters even inside the matrix.