Automated Writing Feedback
P.J. van der Laan (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Marcus M. Specht – Mentor (TU Delft - Web Information Systems)
Manuel Valle Valle Torre – Mentor (TU Delft - Web Information Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The rising number of students challenges the teacher’s time-consuming task to provide consistent and high-quality feedback for all students. To address the traditional education’s challenges, researchers refer to online educational tools to assist teachers. Although large writing tools (for instance, Grammarly and Microsoft Editor) assist students to write effectively, their primary objective is not to educate students. Therefore, we propose RevisionCoach – an automated writing feedback system that iteratively constructs educational, localized feedback to assist students to learn how to write. Clear and effective writing is important for students to succeed in academic endeavors and allows teachers to focus on feedback for the assignment’s primary task. RevisionCoach’s objective is to educate, and for that reason, the design considers learning by deliberate practice, differentiated learning, and self-regulated learning. In addition, RevisionCoach’s feedback has four layers: a sentence-level mistake highlight, an assessment category (coherence, cohesion, readability, and formality), a correction category (rewrite, reword, and rephrase), and a correction suggestion.
RevisionCoach’s categorized feedback allows for convenient evaluation that addresses RevisionCoach’s capability to predict the writing mistake’s importance. In the evaluation, we ask writing experts, students, Grammarly, and Microsoft Editor to find writing mistakes in text and to rate the mistake’s importance. Compared to the experts’ importance predictions, RevisionCoach predicts the mistake importances more accurately (1.89 mean square error (MSE)) than a random baseline (2.70 MSE), student baseline (2.50 MSE), Grammarly (2.52 MSE), and Microsoft Editor (2.02 MSE). Furthermore, the study illustrates the experts’ challenge to provide localized feedback for writing skills because the experts have merely 71.2% agreement about the top 2 most severe mistakes. At the same time, RevisionCoach achieves an average agreement of 72.9% with the experts’ mistake predictions.