As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available. In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice. The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.@en

Analyzing large quantities of real-world textual data has the potential to provide new insights for researchers. However, such data present challenges for both human and computational methods, requiring a diverse range of specialist skills, often shared across a number of individuals. In this paper we use the analysis of a real-world data set as our case study, and use this exploration as a demonstration of our “insight workflow,” which we present for use and adaptation by other researchers. The data we use are impact case study documents collected as part of the UK Research Excellence Framework (REF), consisting of 6,679 documents and 6.25 million words; the analysis was commissioned by the Higher Education Funding Council for England (published as report HEFCE 2015). In our exploration and analysis we used a variety of techniques, ranging from keyword in context and frequency information to more sophisticated methods (topic modeling), with these automated techniques providing an empirical point of entry for in-depth and intensive human analysis. We present the 60 topics to demonstrate the output of our methods, and illustrate how the variety of analysis techniques can be combined to provide insights. We note potential limitations and propose future work.@en

While robust evidence is one ingredient in the policymaking process, it is by no means the only one. Engaging with policymakers and the policymaking process requires collaborative working models, navigating through the experiences, values and perspectives of policymakers and other stakeholders, as well as communicating evidence in an accessible manner. As a response to these requirements, over recent years there has been proliferation of activities that engage producers of evidence (specifically, academics), policymakers, practitioners, and the public in policy formulation, implementation and evaluation. In this article, we describe one engagement approach for facilitating research evidence uptake into policy and practice—an activity called a ‘Policy Lab’—as conducted by the team at The Policy Institute at King’s College London on numerous policy challenges over the past four years. Drawing on our experience in running 15 Policy Labs between January 2015 and September 2019, we (a) provide a guide to how we have run Policy Labs, while sharing our learning on what has worked best in conducting them and (b) demonstrate how these labs can contribute to bringing evidence closer to policymaking, by comparing their characteristics to enablers for doing so identified in the literature. While this approach to Policy Labs is not the only one of its kind, we suggest that these types of Labs manifest characteristics identified in previous studies for influencing the policymaking process; namely: providing a forum for open, honest conversations around a policy topic; creating new networks, collaborations and partnerships between academics and policymakers; synthesising available evidence on a policy topic in a robust and accessible format; and providing timely access to evidence relevant to a policy issue. We recognise the limitations of measuring and evaluating how these Labs change policy in the long-term and recommend viewing the Policy Lab as part of a process for engaging evidence and policymaking and not an isolated activity. This process serves to build a coalition through participation of diverse communities (thereby establishing ‘trust’), work on the language and presentation of evidence (thereby enabling effective ‘translation’ of evidence) and engage policymakers early to respond when policy windows emerge (thereby taking into account ‘timing’ for creating policy action).@en

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we - a group of scholars and practitioners from Africa, America, Asia and Europe - argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.@en

The impact case studies submitted by UK Higher Education Institutions to the Research Excellence Framework (REF) in 2014 provide a rich resource of text describing impact beyond academia and across all disciplines. Using text mining techniques and qualitative assessment, the 6,679 non-redacted case studies submitted were analysed and the impact described was found to be multidisciplinary, multi-impactful, and multinational. By digging deeper into the data, the health gains from health research in terms of Quality Adjusted Life Years was also estimated. Similar analyses are possible using these case studies, but will require the data to be 're-purposed' from the original intention (i.e., for assessment purposes) for robust analysis.@en

The allocation of research funding can benefit greatly from robust analysis of what has worked in research. In turn, these analyses can help advocacy initiatives and demonstrate accountability to taxpayers and donors. Capturing and mapping data on the inputs, processes, outputs, outcome and impact of research is crucial for these analyses. In this article we argue that the research community as a whole—including funders, researchers and administrators—is potentially in a position where it can assess or evaluate research not just according to academic outputs (production of knowledge), but also its outcomes and/or impact (effects on society). Using an exploratory framework that assesses effectiveness, efficiency and equity (3e’s) of research and research assessment both in terms of academic outputs and non-academic impact, we also argue that most assessments are primarily examining the effectiveness of research, as tools are not yet available to systematically assess research for its efficiency and equity. This article is published as part of a special issue on the future of research assessment.@en