Print Email Facebook Twitter Evaluating and comparing ontology alignment systems Title Evaluating and comparing ontology alignment systems: An MCDM approach Author Mohammadi, Majid (TU Delft Information and Communication Technology; Jheronimus Academy of Data Science) Rezaei, J. (TU Delft Transport and Logistics) Date 2020 Abstract Ontology alignment is vital in Semantic Web technologies with numerous applications in diverse disciplines. Due to diversity and abundance of ontology alignment systems, a proper evaluation can portray the evolution of ontology alignment and depicts the efficiency of a system for a particular domain. Evaluation can help system designers recognize the strength and shortcomings of their systems, and aid application developers to select a proper alignment system. This article presents a new evaluation and comparison methodology based on multiple performance metrics that accommodates experts’ preferences via a multi-criteria decision-making (MCDM) method, i.e., Bayesian best–worst method (BWM). First, the importance of a performance metric for a specific task/application is determined according to experts’ preferences. The alignment systems are then evaluated based on proposed expert-based collective performance (ECP) that takes into account multiple metrics as well as their calibrated importance. For comparison, the alignment systems are ranked based on a probabilistic scheme, where it includes the extent to which one alignment system is preferred over another. The proposed methodology is applied to six tracks from ontology alignment evaluation initiative (OAEI), where the importance of performance metrics are calibrated by designing a survey and eliciting the preferences of ontology alignment experts. Accordingly, the participating alignment systems in the OAEI 2018 are evaluated and ranked. While the proposed methodology is applied to six OAEI tracks to demonstrate its applicability, it can also be applied to any benchmark or application of ontology alignment. Subject Bayesian BWMEvaluationMCDMOntology alignmentRanking To reference this document use: http://resolver.tudelft.nl/uuid:88f8b0a9-58fc-415e-8e9f-dee391f78fad DOI https://doi.org/10.1016/j.websem.2020.100592 ISSN 1570-8268 Source Journal of Web Semantics, 64 Part of collection Institutional Repository Document type journal article Rights © 2020 Majid Mohammadi, J. Rezaei Files PDF 1_s2.0_S1570826820300330_main.pdf 753.37 KB Close viewer /islandora/object/uuid:88f8b0a9-58fc-415e-8e9f-dee391f78fad/datastream/OBJ/view