In this work, a distributed multi-agent optimization problem is studied where different subsets of agents are coupled with each other through affine constraints. Moreover, each agent is only aware of its own contribution to the constraints and only knows which neighboring agents share constraints with it. An effective distributed first-order algorithm is developed, which requires sharing dual variables only and takes advantage of the constraint sparsity. The algorithm is shown to converge to the exact minimizer under sufficiently small constant step sizes. A simulation is given to illustrate the effect of the constraint structure and advantages of the proposed algorithm.