The set of finite binary matrices of a given size is known to carry a finite type AA bicrystal structure. We first review this classical construction, explain how it yields a short proof of the equality between Kostka polynomials and one-dimensional sums together with a natural generalisation of the 2M−X2M−X Pitman transform. Next, we show that, once the relevant formalism on families of infinite binary matrices is introduced, this is a particular case of a much more general phenomenon. Each such family of matrices is proved to be endowed with Kac–Moody bicrystal and tricrystal structures defined from the classical root systems. Moreover, we give an explicit decomposition of these multicrystals, reminiscent of the decomposition of characters yielding the Cauchy identities.