The crystal structure of a compound plays an important role in determining its properties. Here we analyze over 4000 intermetallic compounds, and we identify a hierarchical relationship between their crystal structures. By considering each intermetallic compound as a particular chemical ordering over the sites of a higher symmetry crystal structure, we determine that most intermetallic compounds can be derived from a small number of parent crystal structures. While many compounds are chemical orderings over the sites of simple crystal structures such as body-centered-cubic (bcc) and face-centered-cubic (fcc) structures, the majority map onto more complex parent crystal structures. Surprisingly, many intermetallic compounds map onto parent crystal structures that differ from those of their elemental constituents. We find that the occurrence of several of the more complex parent crystal structures, such as the Laves phases, can be understood in terms of simple descriptors such as the ratio of atomic radii and electronegativity differences.