Cognitive effort in post-editing of machine translation : evidence from eye movements, subjective ratings, and think-aloud protocols

Abstract

This thesis investigates the expenditure of cognitive effort in post-editing of machine translation. A mixed-method approach involving the use of eye movements, subjective ratings and think-aloud protocols was adopted for the investigation. The project aims at revealing connections between cognitive effort and variables including linguistic characteristics of the source text and the machine-translation output, post-editors’ individual traits, different linguistic aspects of the activity attended to during the task, and the overall quality of the post-edited text, assessed by human translators in terms of fluency (linguistic quality) and adequacy (faithfulness to the source text). Two tasks were conducted to pursue these aims: one involving eye tracking and a self-report scale of cognitive effort, and another carried out by a different, but comparable, sample of participants, under a think-aloud condition. Results indicate that variables such as an automatic machine-translation quality score and source-text type-token ratio are good predictors of cognitive effort in post-editing. The relationship between cognitive effort and post-editors’ traits was found to be a complex one, with significant links in this respect only appearing in the context of interactions between variables. A complex relationship was also found between editing behaviour and the quality of the post-edited text: the amount of changes implemented was found to have a generally positive association with post-edited fluency, though cognitive effort was found to be negatively correlated with both the fluency and adequacy of the post-edited text. Mental processes involving grammar and lexis were significantly related to the levels of cognitive effort expended by participants, being also the aspects most frequently attended to in the activity. From a methodological perspective, despite the criticisms received by the think-aloud method in previous research, empirical data obtained in this project indicate that think-aloud protocols correlate with eye movements and subjective ratings as measures of cognitive effort.