Mitigating the energy requirements of artificial intelligence requires novel physical substrates for computation. Phononic metamaterials have vanishingly low power dissipation and hence are a prime candidate for green, always-on computers. However, their use in machine learning applications has not been explored due to the complexity of their design process. Current phononic metamaterials are restricted to simple geometries (e.g., periodic and tapered) and hence do not possess sufficient expressivity to encode machine learning tasks. A non-periodic phononic metamaterial, directly from data samples, that can distinguish between pairs of spoken words in the presence of a simple readout nonlinearity is designed and fabricated, hence demonstrating that phononic metamaterials are a viable avenue towards zero-power smart devices.|Elastic neural networks composed of phononic metamaterials respond differently to different spoken commands, passively solving a speech classification problem. Their design harnesses the vanishingly low power dissipation of elastic waves, combined with the high expressivity and efficient simulation of metamaterials. This capability can be leveraged to build smart sensors that detect events without standby power consumption.image