OI
O. Inel
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Characterization and Mitigation of High-Confidence Errors Through the Use of Human-In-The-Loop Methods
Domain Expert Driven Approach to Model Development
Master thesis
(2022)
-
Pavel Hoogland, Jie Yang, Oana Inel, Margje Schuur, Jasper van Vliet, G.J.P.M. Houben, Lydia Chen
In the use of Machine Learning systems, attaining the trust of those that are the end-users can often be difficult. Many of the current state-of-the-art systems operate as Black-Boxes. Errors produced by these Black-Box systems, without further explanation as to why these decisions were made, will deteriorate trust. This effect is especially strong when these erroneous decisions are generated with high confidence. This thesis presents both a data-driven as well as a human-in-the-Loop based methodology to characterize and mitigate high-confidence errors. We propose an Iterative Expert Session based methodology. By engaging domain experts through a series of interaction sessions, we aim to reduce the disconnect and knowledge gap between data scientists and domain experts, and to ultimately increase trust in the model. A practical approach was taken working in close connection with the practice of the data scientists of the ILT, helping them in improving their model and providing a direct contribution. We study the problem in the context of Road and Transportation law violations, by engaging inspectors (i.e., domain experts) in day-in-the-life and in-house interview sessions.
A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors.
We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables. ...
A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors.
We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables. ...
In the use of Machine Learning systems, attaining the trust of those that are the end-users can often be difficult. Many of the current state-of-the-art systems operate as Black-Boxes. Errors produced by these Black-Box systems, without further explanation as to why these decisions were made, will deteriorate trust. This effect is especially strong when these erroneous decisions are generated with high confidence. This thesis presents both a data-driven as well as a human-in-the-Loop based methodology to characterize and mitigate high-confidence errors. We propose an Iterative Expert Session based methodology. By engaging domain experts through a series of interaction sessions, we aim to reduce the disconnect and knowledge gap between data scientists and domain experts, and to ultimately increase trust in the model. A practical approach was taken working in close connection with the practice of the data scientists of the ILT, helping them in improving their model and providing a direct contribution. We study the problem in the context of Road and Transportation law violations, by engaging inspectors (i.e., domain experts) in day-in-the-life and in-house interview sessions.
A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors.
We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables.
A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors.
We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables.
Multiperspectivity in online news
An analysis of how reading behaviour is affected by viewpoint diverse news recommendations and how they are presented
Previous research on diversity in recommender systems define diversity as the opposite of similarity and propose methods that are based on topic diversity. Diversity in news media, however, is understood as multiperspectivity and scholars generally agree that fostering diversity is the key responsibility of the press in a democratic society. Therefore, a novel viewpoint diversification method was developed, based on the reranking of recommendation lists within the topic using framing aspects. Among other results, an offline evaluation indicated that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a metric from literature. However, to truly enable multiperspectivity in automatic online news environments, users should also be willing to consume viewpoint diverse news recommendation. Therefore, an online study was conducted, assessing how viewpoint diverse recommendations and their presentation characteristics affect the reading behaviour of Blendle users. During a two-week experiment, two groups of 1038 users were presented a set of three recommendations below the content of two articles every day. Thereby, one group received recommendations based on relevance to the original article, while the other group received viewpoint diverse recommendations. Three implicit and one explicit measure of the reading behaviour were analysed. Additionally, the influence of the presentation characteristics of the recommendation on the reading behaviour was analysed. Generally, no major differences were found in the reading behaviour of both user groups. Only the results of the click-through rate calculated per recommendation set indicated a significant difference of 6.5% to the advantage of the baseline users. For the other measures of the reading behaviour, no significant differences were found between the baseline and diverse users. However, the results do show that multiple presentation characteristics have a significant influence on the reading behaviour. Therefore, these results suggest that future research on how recommendation can be presented is just as important as novel viewpoint diversification methods to truly achieve multiperspectivity in automated online news environments.
...
Previous research on diversity in recommender systems define diversity as the opposite of similarity and propose methods that are based on topic diversity. Diversity in news media, however, is understood as multiperspectivity and scholars generally agree that fostering diversity is the key responsibility of the press in a democratic society. Therefore, a novel viewpoint diversification method was developed, based on the reranking of recommendation lists within the topic using framing aspects. Among other results, an offline evaluation indicated that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a metric from literature. However, to truly enable multiperspectivity in automatic online news environments, users should also be willing to consume viewpoint diverse news recommendation. Therefore, an online study was conducted, assessing how viewpoint diverse recommendations and their presentation characteristics affect the reading behaviour of Blendle users. During a two-week experiment, two groups of 1038 users were presented a set of three recommendations below the content of two articles every day. Thereby, one group received recommendations based on relevance to the original article, while the other group received viewpoint diverse recommendations. Three implicit and one explicit measure of the reading behaviour were analysed. Additionally, the influence of the presentation characteristics of the recommendation on the reading behaviour was analysed. Generally, no major differences were found in the reading behaviour of both user groups. Only the results of the click-through rate calculated per recommendation set indicated a significant difference of 6.5% to the advantage of the baseline users. For the other measures of the reading behaviour, no significant differences were found between the baseline and diverse users. However, the results do show that multiple presentation characteristics have a significant influence on the reading behaviour. Therefore, these results suggest that future research on how recommendation can be presented is just as important as novel viewpoint diversification methods to truly achieve multiperspectivity in automated online news environments.