In the context of increasing concern for anthropogenic CO2 emissions, the residential building sector still represents a major contributor to energy demand. The integration of renewable energy sources, and particularly of photovoltaic (PV) panels, is becoming an increasingly widespread solution for reducing the carbon footprint of building energy systems (BES). However, the volatility of the energy generation and its mismatch with the typical demand patterns are cause for concern, particularly from the viewpoint of the management of the power grid. This paper aims to show the influence of the orientation of photovoltaic panels in designing new BES and to provide support to the decision making process of optimal PV placing. The subject is addressed with a mixed integer linear optimization problem, with costs as objectives and the installation, tilt, and azimuth of PV panels as the main decision variables. Compared with existing BES optimization approaches reported in literature, the contribution of PV panels is modeled in more detail, including a more accurate solar irradiation model and the shading effect among panels. Compared with existing studies in PV modeling, the interaction between the PV panels and the remaining units of the BES, including the effects of optimal, scheduling is considered. The study is based on data from a residential district with 40 buildings in western Switzerland. The results confirm the relevant influence of PV panels’ azimuth and tilt on the performance of BES. Whereas south-orientation remains the most preferred choice, west-orientationed panels better match the demand when compared with east-orientationed panels. Apart from the benefits for individual buildings, an appropriate choice of orientation was shown to benefit the grid: rotating the panels 20° westwards can, together with an appropriate scheduling of the BES, reduce the peak power of the exchange with the power grid by 50% while increasing total cost by only 8.3%. Including the more detailed modeling of the PV energy generation demonstrated that assuming horizontal surfaces can lead to inaccuracies of up to 20% when calculating operating expenses and electricity generated, particularly for high levels of PV penetration.