Hexagonal lattice systems (e.g., triangular, honeycomb, kagome) possess a multidimensional irreducible representation corresponding to d(x2-y2) and d(xy) symmetry. Consequently, various unconventional phases that combine these d-wave representations can occur, and in so doing may break time-reversal and spin-rotation symmetries. We show that hexagonal lattice systems with extended repulsive interactions can exhibit instabilities in the particle-hole channel to phases with either d(x2-y2) + d(xy) or d + id symmetry. When lattice translational symmetry is preserved, the phase corresponds to nematic order in the spin channel with broken time-reversal symmetry, known as the beta phase. On the other hand, lattice translation symmetry can be broken, resulting in various d(x2-y2) + d(xy) density wave orders. In the weak-coupling limit, when the Fermi surface lies close to a van Hove singularity, instabilities of both types are obtained in a controlled fashion.