From Text to Knowledge : A New Approach to REF2014 Impact Case Study Analytics

Abstract

The Research Excellence Framework (REF), initiated in 2014, evaluates the quality of academic research conducted by universities in the United Kingdom (UK). In the assessment, 20% of the total score was allocated based on peer-reviewed evaluations of research impact, underscoring the growing emphasis on impact in UK government policy. In academia, the impact is characterised as the influence, change, or benefit contributed to various sectors outside the academic domain, such as the economy, society, culture, public policy, health, environment, and overall quality of life. Universities submitted four-page impact case studies to substantiate their research impact claims, primarily composed of free-form text detailing and evidencing the significance and relevance of the research. Existing analyses of these case studies have predominantly relied on qualitative methods or rudimentary text searching and analysis techniques. However, these approaches face limitations, including the time-consuming nature of manual data analysis, inconsistent research definitions, and the suboptimal quality of answers generated by applying computational analysis to unstructured, context-deficient free-text data. This thesis introduces a novel approach that addresses these challenges by leveraging a structured, queryable semantic representation of text extracted from impact case studies. We designed the ontology to structure the information and illustrate how Semantic Web technologies are utilised for data storage and querying. Evaluations are conducted to inform the development of a domain ontology that is not only pertinent to the current use case but also adaptable for future applications. Experimental results demonstrate that our approach offers three key advantages over existing methods: enhanced precision in question answering, the capability to address a broader spectrum of questions by integrating data from external sources, and creating a reusable and extensible ontology design. Furthermore, we explore the potential of this structured representation to enable the application of supervised machine learning algorithms for predicting the assigned grade of each case study and explain the underlying rationale behind the results. The findings conclude that predictions for impact case studies in the Computer Science and Informatics Unit of Assessment can be made with high reliability and accuracy. This research underscores the efficacy of employing structured semantic representations and machine learning techniques to analyse and predict the impact of academic research, offering valuable insights for policymakers, funding bodies, and researchers.