Validation methodology for expert-annotated datasets

Validation methodology for expert-annotated datasets: Event annotation case study

Inel, O. (TU Delft Web Information Systems; Vrije Universiteit Amsterdam)
Aroyo, Lora (Google LLC)

de Melo, Gerard (editor)
Klimek, Bettina (editor)
Fath, Christian (editor)
Buitelaar, Paul (editor)
Dojchinovski, Milan (editor)
Eskevich, Maria (editor)
McCrae, John P. (editor)
Chiarcos, Christian (editor)

2019-05-01

Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%.

Crowdsourcing
Event extraction
Human-in-the-loop
Time extraction

http://resolver.tudelft.nl/uuid:4ede1172-ac7a-415d-bc66-d1ec8fe3bd19

https://doi.org/10.4230/OASIcs.LDK.2019.12

Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing

9783959771054

2nd Conference on Language, Data and Knowledge, LDK 2019, 70

2nd Conference on Language, Data and Knowledge, LDK 2019, 2019-05-20 → 2019-05-23, Leipzig, Germany

Institutional Repository

conference paper

© 2019 O. Inel, Lora Aroyo