In this thesis, we consider commercial buildings with available heating, ventilation and air conditioning (HVAC) systems, and develop methods to assess and exploit their energy storage and production potential to collectively offer ancillary services to the power grid. This demand response problem can be put in the generic framework of multi-agent optimization and control. In this setting, various agents interact through their objectives, constraints or dynamics over a network. In the example of demand response, individual buildings are connected to the power network and coupled via their common objective of providing service and the constraints of the power network. Within this generic multi-agent framework, we develop layers of abstractions that enables efficient coordination of the agents while making sure that the network constraints are satisfied, and the common goal of the agents is achieved. The first approach is based on quantifying the tracking capability of a local system using robust optimization. Different from a standard robust optimization problem, we modify and optimize over the uncertainty set that represents the set of reference trajectories the system is required to track while rejecting external disturbances. The method facilitates hierarchical control by using reference sets for coordinating many agents. In the second approach, we consider coordination of multiple agents by using local cost and constraint approximations. Specifically we consider decomposition of interior point methods in a multi-agent setting and analyze the computation and modeling task for the agents and the coordinator. We further consider decomposition of state of the art predictor-corrector type interior point methods and show that a naive implementation may result in excessive communication in a multi-agent setting. In order to remedy this issue, we propose a modification of the standard algorithm that uses decentralized predictions. We analyze convergence of the method and test the performance with numerical experiments. Finally, we look into applying decomposition based interior point methods in a distributed model predictive control problem that includes dynamic coupling between the agents. Instead of solving the problem to optimality, adding barrier functions to the objective enhances numerical performance significantly, an approach that is well-known in model predictive control (MPC) literature. We consider applying this method in economic MPC problems with terminal equilibrium constraints, which is suitable for decomposition due to the simplicity of terminal constraints. However in this case standard results for MPC with barrier functions do not apply. We propose iterative re-centering of the barriers, which allows interpreting them as a regularizing cost in the problem that penalizes deviation from open-loop predictions. We show that regularizing barrier functions not only improve the numerical performance and facilitate decomposition, but also enhance system theoretical properties.