By 2050, the global demand for food and energy is expected to grow by 70% and 50%, respectively, as a result of the increase of the world’s population. To keep up with the growing world population, significant changes have to take place in the agricultural sector regarding land and energy use. In this work, two developments in the agricultural sector are combined: the integration of photovoltaic (PV) modules in a horticultural system and the use of light emitting diodes (LEDs) as supplemental lighting source for crops. Four greenhouse systems having different configurations of these two technologies and a plant factory are designed. The performance of these LED-based agrivoltaic systems are analysed for the crops lettuce and tomato and for three different climates: Sweden, the Netherlands and the United Arab Emirates. This study shows that the integration of a PV system and LED technology in a greenhouse system improves the crop production per unit of energy compared to conventional greenhouse systems. Furthermore, it has been found that the production of crop per piece of land can be significantly increased by the use of a plant factory. In extreme climates, the plant factory is the optimal system for the cultivation of crops.