Evaluation of educational innovation at the course level is dominated by frameworks that are linear, built on simple models, lack grounding in theory, and are generally not flexible enough to provide actionable recommendations for improvements. Moreover, the guidelines provided by existing literature are difficult to transfer to other contexts as most of these are aimed at specific innovations only. Therefore, there is a need for a new approach that is flexible enough to enable adaptation to different contexts, and that draws on the strengths of existing evaluation traditions. This paper forms a starting point for a larger research initiative for developing a comprehensive innovation evaluation framework. The context of the paper is a project based within the four universities of technology in The Netherlands (4TU) dedicated to strengthening collaboration in engineering education and research needed to take on global societal problems. The potential reach of our framework has implications for the wider field. This paper describes the results of an exploratory literature review to draw on the strengths of various evaluation theories to develop a new approach to evaluating educational innovation in courses. Value, Methods, and Use are the main components identified for this approach. Secondly, we developed a workshop to pilot this approach. Lastly, the workshop was presented to educational researchers, engineering educators, and educational advisors. We share lessons learned from the workshop and conclude with descriptions of future research to refine the framework to prepare its application in real innovation initiatives.

Engineering problems are not naturally restricted to artificial discipline-oriented boundaries (Ertas et al. 2003). To solve such complex problems, future engineers need to collaborate with both (academic) experts and non-academic stakeholders from different fields and backgrounds and take various perspectives into account. Societal stakeholders can contribute valuable input to support the creation of engineering solutions. Addressing big challenges (as the 14 grand engineering challenges formulated by the National Academy of Engineering) demands a joint effort of diverse teams, different disciplines, different companies, people viewing and tackling the problems from different perspectives and angles. The students we are educating now are likely to be part of such teams, which are not separated from the economic, societal and political aspects of our society. One of the main questions that we thus need to ask, whether we are educating students now to be part of such inter- and transdisciplinary teams and whether they can navigate in the societal trends.

As a result, in any engineering development, future engineers must consistently be aware of the size and extent of the impact. The fact that this comes with major uncertainties implies that future engineers should not only be educated in the “hard” technique and management of stakeholders but also in how to deal with uncertainty. Technical and social systems in society have become complex or wicked; consequently, a planned and control-focused approach will invariably fail. Even when not designing them themselves, engineers need skills to cope with unanticipated events, values and stakeholder positions.

This requires students to learn how to anticipate the social, technical, societal and environmental impact of their actions. For this, they need skills that transcend the ‘hard’ scientific and technical skills related to disciplinary education and focus also on e.g. transdisciplinary skills. Tan et al. (2019) listed systems thinking, metacognition, empathy, and open-mindedness as essential for reaching transdisciplinarity.

Much has been written about the necessity of such skills, but less about how these could be translated to effective learning and teaching strategies for specific, dedicated and desired learning outcomes fitting to the development level (1st to 5th year students) of the students within their respective programs (BSc, Minor, Master) that are also assessable in an educational context.

In this session, we will briefly discuss the necessity of an approach to dissecting transdisciplinary tools into their basic concepts, collecting already existing pedagogical methods, and designing new ways to practice these skills. Then, we will ask the audience to participate in a quick brainstorm session to generate ideas for how systems thinking, metacognition, empathy, or open-mindedness could be incorporated in educational programmes. After sharing the results of the brainstorming, our panel will discuss some important aspects of transdisciplinary education we came across during or university-wide research on teaching practices, led by statements and dilemmas.

While courses are periodically updated and improved with the integration of new teaching methods and technologies, we still lack a systematic, research- informed method for evaluating educational innovations. Existing evaluations often lack contextual transparency to enable transferability to other courses and are mostly limited to student surveys. To take on this problem, we developed a framework for evaluating innovation in courses. The accompanying poster will present elements from this framework. The results presented in the accompanying poster are based on two literature reviews and a workshop. The framework prescribes an iterative process: (1) analysis of the innovation and its context of implementation, and (2) development of the evaluation plan. To apply the framework, both a formal (consultancy format) and an informal method (workshop) have been developed. Educators can use the framework to take a more scholarly approach to evaluating educational innovations for better decision-making, and to make teaching achievements more visible.

For universities, educational change at institutional level is a slow process [1], [2]. To keep up with societal and technological advancement, education innovation project leaders at universities need practical guidelines and procedures in place that will enable sustainable and scalable innovation that can meet the needs of industry as we transition from Industry 4.0 to Industry 5.0 [3]. To develop such guidelines and procedures, we need to conduct socially responsible, evidence-based educational research [4]. This paper is part of a larger study during which we will conceptualize the planning and evaluation of innovation in engineering education at the Delft University of Technology (TU Delft). From this conceptualization, a framework for planning and evaluation of education innovation will emerge. The data collection process will take place in six phases: (1) Exploration of the problem (2) feasibility studies; (3) conceptualization and development of the framework; (4) piloting of the framework and its associated processes; (5) field study; and lastly, (6) evaluation of the framework. This paper provides an initial overview of the literature, as well as an explanation of the proposed research methodology.