The Energy Management System (EMS) is an efficient technique to monitor, control and enhance the building performance. In the state-of-the-art, building performance analysis is separated into building simulation and control management: this may cause inaccuracies and extra operating time. Thus, a coherent framework to integrate building physics with various energy technologies and energy control management methods is highly required. This framework should be formed by simplified but accurate models of building physics and building energy technologies, and should allow for the selection of proper control strategies according to the control objectives and scenarios. Therefore, this paper reviews the fundamental mathematical modeling and control strategies to create such a framework. The mathematical models of (i) building physics and (ii) popular building energy technologies (renewable energy systems, common heating and cooling energy systems and energy distribution systems) are first presented. Then, it is shown how the collected mathematical models can be linked. Merging with two frequently used EMS strategies, namely rule-based and model predictive controls, is discussed. This work provides an extendable map to model and control buildings and intends to be a foundation for building researchers, designers and engineers.