In a power grid, the electricity supply and demand must be balanced at all times to maintain the system's frequency. In practice, the grid operator achieves this balance by procuring frequency reserves in an ahead-of-time market setting. During runtime, these reserves are then dispatched whenever there is an imbalance in the grid. Recently, there has been an increasing interest in engaging electricity consumers, such as plug-in electric vehicles or buildings, to offer such frequency reserves by exploiting their flexibility in power consumption. In this paper, we focus on an aggregation of buildings that places a joint bid on a reserve market. The resulting shared decision is modeled as a large-scale optimization problem. Our main contribution is to show that the aggregation can make its decision in a computationally efficient and conceptually meaningful way, using the alternating direction method of multipliers. The proposed approach exhibits several attractive features that include: (i) the computational burden is distributed between the buildings; (ii) the setup naturally provides privacy and flexibility; (iii) the iterative algorithm can be stopped at any time, providing a feasible (though suboptimal) solution; and (iv) the algorithm provides the foundation for a reward distribution scheme that strengthens the group.