JU
J. Urbano Merino
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
31 records found
1
Personalized QA agents increasingly retrieve from both external documents and private conversational memory. This raises a basic design question: does answer correctness depend on where the supporting evidence is stored? Existing evaluations make this difficult to answer because memory and document conditions often differ in their questions, facts, or available sources. We construct a controlled evaluation setting from MultiHop-RAG in which the same questions and evidence are held fixed while golden evidence is placed entirely in documents, entirely in conversational memory, or split across both. We evaluate four retrieval architectures across three persona-specific memory corpora. The results show that evidence placement does not exert a simple, uniform penalty. Instead, its effect is significantly moderated by the topic match of the question to the persona's memory profile. In particular, questions whose domain matched the surrounding memory profile were answered more accurately, a surprising result that challenges the expectation that topically concentrated memory should harm retrieval and reasoning. This performance degradation during off-topic exploration suggests personalized agents may create soft algorithmic `filter bubbles' that penalize user curiosity.
...
Personalized QA agents increasingly retrieve from both external documents and private conversational memory. This raises a basic design question: does answer correctness depend on where the supporting evidence is stored? Existing evaluations make this difficult to answer because memory and document conditions often differ in their questions, facts, or available sources. We construct a controlled evaluation setting from MultiHop-RAG in which the same questions and evidence are held fixed while golden evidence is placed entirely in documents, entirely in conversational memory, or split across both. We evaluate four retrieval architectures across three persona-specific memory corpora. The results show that evidence placement does not exert a simple, uniform penalty. Instead, its effect is significantly moderated by the topic match of the question to the persona's memory profile. In particular, questions whose domain matched the surrounding memory profile were answered more accurately, a surprising result that challenges the expectation that topically concentrated memory should harm retrieval and reasoning. This performance degradation during off-topic exploration suggests personalized agents may create soft algorithmic `filter bubbles' that penalize user curiosity.
This study examines how effectively widely used offline information retrieval (IR) metrics reflect changes in online performance. As offline evaluation plays a central role in model development, understanding its alignment with user‑oriented signals is essential. Using 52 diverse ranking pipelines and approximately 2,000 queries from the MS MARCO DL19 and DL20 benchmarks, we analyze the sensitivity of five offline metrics: Precision@10, Recall@10, MAP, MRR, and NDCG@10, to five simulated online metrics: CTR, SSR, ZRR, ADT, and SAR. Sensitivity is quantified through slope-based analysis, and alignment is assessed using the Pearson correlation coefficient. Our results show that NDCG@10 and Recall@10 are the most sensitive offline metrics across multiple online behaviors, while Precision@10 consistently exhibits low sensitivity. Furthermore, we demonstrate that sensitivity and alignment capture complementary aspects of offline–online relationships: some metric pairs show strong responsiveness but weak linear consistency. Overall, this study provides a detailed and reproducible evaluation of how offline metrics behave in relation to simulated online performance, offering practical guidance for selecting offline metrics that better reflect user-centric outcomes.
https://github.com/AinzOoalGown123/Metric-Sensitivity-Analysis ...
https://github.com/AinzOoalGown123/Metric-Sensitivity-Analysis ...
This study examines how effectively widely used offline information retrieval (IR) metrics reflect changes in online performance. As offline evaluation plays a central role in model development, understanding its alignment with user‑oriented signals is essential. Using 52 diverse ranking pipelines and approximately 2,000 queries from the MS MARCO DL19 and DL20 benchmarks, we analyze the sensitivity of five offline metrics: Precision@10, Recall@10, MAP, MRR, and NDCG@10, to five simulated online metrics: CTR, SSR, ZRR, ADT, and SAR. Sensitivity is quantified through slope-based analysis, and alignment is assessed using the Pearson correlation coefficient. Our results show that NDCG@10 and Recall@10 are the most sensitive offline metrics across multiple online behaviors, while Precision@10 consistently exhibits low sensitivity. Furthermore, we demonstrate that sensitivity and alignment capture complementary aspects of offline–online relationships: some metric pairs show strong responsiveness but weak linear consistency. Overall, this study provides a detailed and reproducible evaluation of how offline metrics behave in relation to simulated online performance, offering practical guidance for selecting offline metrics that better reflect user-centric outcomes.
https://github.com/AinzOoalGown123/Metric-Sensitivity-Analysis
https://github.com/AinzOoalGown123/Metric-Sensitivity-Analysis
As line datasets grow larger, the demand for effective visual data analysis becomes increasingly important. Understanding large‑scale datasets remains a fundamental challenge. A critical trade‑off is presented by existing line selection methods: they either produce efficiency, accuracy, or human interpretability, rarely achieving all three simultaneously. This gap is addressed by the development of human‑guided and context‑aware brushing techniques, which are supported by manual, semi‑automatic and automatic refinement methods. Through empirical evaluation via two user studies, it was found that, whilst context‑aware brushes offer theoretical promise, statistical superiority over conventional brushing approaches is not demonstrated. However, selection accuracy is consistently improved by refinement techniques, with manual refinement yielding the highest accuracy gains (12.6\%) followed by semi‑automatic refinement (9.8\%). Notably, efficiency gains from refinement remain dataset‑dependent, with no single technique universally dominating across varied data characteristics. Manual and semi‑automatic refinements are preferred by users seeking high‑accuracy improvements. Although similar efficiency scores are exhibited by manual and semi‑automatic refinements, the lowest variance is observed for the semi‑automatic method; consequently, it is recommended for users prioritising efficiency. The findings emphasise a fundamental design principle: Interpretability and user agency should be prioritised over full automation.
...
As line datasets grow larger, the demand for effective visual data analysis becomes increasingly important. Understanding large‑scale datasets remains a fundamental challenge. A critical trade‑off is presented by existing line selection methods: they either produce efficiency, accuracy, or human interpretability, rarely achieving all three simultaneously. This gap is addressed by the development of human‑guided and context‑aware brushing techniques, which are supported by manual, semi‑automatic and automatic refinement methods. Through empirical evaluation via two user studies, it was found that, whilst context‑aware brushes offer theoretical promise, statistical superiority over conventional brushing approaches is not demonstrated. However, selection accuracy is consistently improved by refinement techniques, with manual refinement yielding the highest accuracy gains (12.6\%) followed by semi‑automatic refinement (9.8\%). Notably, efficiency gains from refinement remain dataset‑dependent, with no single technique universally dominating across varied data characteristics. Manual and semi‑automatic refinements are preferred by users seeking high‑accuracy improvements. Although similar efficiency scores are exhibited by manual and semi‑automatic refinements, the lowest variance is observed for the semi‑automatic method; consequently, it is recommended for users prioritising efficiency. The findings emphasise a fundamental design principle: Interpretability and user agency should be prioritised over full automation.
A Probabilistic Account of the Uncertainty Due to Ties in Rank-Biased Overlap
Efficient Estimation of the Uncertainty Distribution for Tied Data
Rank similarity quantifies the difference between two ordered sets of items. Rank-Biased Overlap (RBO) is a top-weighted measure of rank similarity that can be used for a pair of indefinite rankings, such that only a prefix is known and that items need not be present in both rankings. This method is frequently used in Information Retrieval (IR), such as to compare search engine results. RBO defines tight lower and upper bounds, RBO_min and RBO_max, which give the uncertainty due to items in the unseen suffix. Another source of uncertainty are ties: two items are tied in a ranking if their true order is not known. Recent work on the treatment of ties in RBO has made it a tie-aware measure. However, unlike the uncertainty due to unseen items, uncertainty due to ties does not disappear for longer prefixes. Determining the distribution of possible scores is O((n!)^2) if all arrangements of ties are considered, and existing methods only find the lower and upper bound for RBO with respect to ties. We investigate whether a probabilistic estimator for the uncertainty distribution can be constructed. We use an iterative convolution method to compose the marginal PMFs of each item. By evaluating against synthetic data, we show that this estimate distribution can be used to reliably compute confidence intervals, mean, and variance. We conclude that a probabilistic method is a viable solution when seeking deterministic results with fast computation.
...
Rank similarity quantifies the difference between two ordered sets of items. Rank-Biased Overlap (RBO) is a top-weighted measure of rank similarity that can be used for a pair of indefinite rankings, such that only a prefix is known and that items need not be present in both rankings. This method is frequently used in Information Retrieval (IR), such as to compare search engine results. RBO defines tight lower and upper bounds, RBO_min and RBO_max, which give the uncertainty due to items in the unseen suffix. Another source of uncertainty are ties: two items are tied in a ranking if their true order is not known. Recent work on the treatment of ties in RBO has made it a tie-aware measure. However, unlike the uncertainty due to unseen items, uncertainty due to ties does not disappear for longer prefixes. Determining the distribution of possible scores is O((n!)^2) if all arrangements of ties are considered, and existing methods only find the lower and upper bound for RBO with respect to ties. We investigate whether a probabilistic estimator for the uncertainty distribution can be constructed. We use an iterative convolution method to compose the marginal PMFs of each item. By evaluating against synthetic data, we show that this estimate distribution can be used to reliably compute confidence intervals, mean, and variance. We conclude that a probabilistic method is a viable solution when seeking deterministic results with fast computation.
Quantifying Uncertainty due to Ties in Rank Correlation Coefficients
An algorithmic approach to computing the bounds of uncertainty
Rank correlation coefficients are a common tool for describing similarity between ordered data. This study examines the use of the popular coefficient Kendall's τ, specifically in the case where the rankings contain tied items that should not be tied. Ties in this case represent uncertainty in the ranking, induced by the system that produced it, usually due to effects such as missing information or loss of precision (rounding). We propose two variants, τmin and τmax, which represent the lowest and highest possible correlation over all ways of arbitrating tied items. Our contribution is a novel quadratic-time algorithm for computing an arbitration of ties which yields the extremal correlation values τmin, τmax. We formally prove the correctness of the algorithm for the original Kendall's τ, and we suggest an adaptation for weighted variants of τ, such as τAP by Yilmaz et al. and τh by Vigna. Empirical evaluation on both synthetic ranking pairs and TREC ad-hoc system outputs demonstrates that ties often induce wide intervals [τmin, τmax], indicating that no single value can fully encapsulate the uncertainty in correlation. These wide intervals also appear in rankings where current methods of computing τ correlation in presence of ties, namely τa and τb, have values large enough (≥0.9) for researchers to use as evidence of strong correlation. This indicates that currently used methods may yield false positive results. By reporting τ alongside its uncertainty bounds τmin and τmax, researchers are able to make more informed decisions,
by demonstrating the reliability of correlation in presence of uncertainty-induced ties.
...
Rank correlation coefficients are a common tool for describing similarity between ordered data. This study examines the use of the popular coefficient Kendall's τ, specifically in the case where the rankings contain tied items that should not be tied. Ties in this case represent uncertainty in the ranking, induced by the system that produced it, usually due to effects such as missing information or loss of precision (rounding). We propose two variants, τmin and τmax, which represent the lowest and highest possible correlation over all ways of arbitrating tied items. Our contribution is a novel quadratic-time algorithm for computing an arbitration of ties which yields the extremal correlation values τmin, τmax. We formally prove the correctness of the algorithm for the original Kendall's τ, and we suggest an adaptation for weighted variants of τ, such as τAP by Yilmaz et al. and τh by Vigna. Empirical evaluation on both synthetic ranking pairs and TREC ad-hoc system outputs demonstrates that ties often induce wide intervals [τmin, τmax], indicating that no single value can fully encapsulate the uncertainty in correlation. These wide intervals also appear in rankings where current methods of computing τ correlation in presence of ties, namely τa and τb, have values large enough (≥0.9) for researchers to use as evidence of strong correlation. This indicates that currently used methods may yield false positive results. By reporting τ alongside its uncertainty bounds τmin and τmax, researchers are able to make more informed decisions,
by demonstrating the reliability of correlation in presence of uncertainty-induced ties.
Frequently used in modern applications, rankings provide users with a list of the most relevant items. In information retrieval research, the τ, τap, and τh correlation coefficients are commonly applied to assess the similarity of the underlying systems by comparing the rankings they produce. Traditionally, these comparisons focus solely on item ranking, but introducing relevance values has enabled systems to be analysed based on how element utility relates to retrieval order. In this work, τ, τap, and τh are extended to incorporate relevance values, presenting several coefficients rooted in an axiomatic approach. These measures compare the utility of items or indices, enabling a granular relevance-based ranking comparison. Overall, the results demonstrate that including relevance judgments leads to significant deviations from traditional rank correlation metrics, highlighting the impact of relevance-aware measures in evaluating system performance and similarity.
...
Frequently used in modern applications, rankings provide users with a list of the most relevant items. In information retrieval research, the τ, τap, and τh correlation coefficients are commonly applied to assess the similarity of the underlying systems by comparing the rankings they produce. Traditionally, these comparisons focus solely on item ranking, but introducing relevance values has enabled systems to be analysed based on how element utility relates to retrieval order. In this work, τ, τap, and τh are extended to incorporate relevance values, presenting several coefficients rooted in an axiomatic approach. These measures compare the utility of items or indices, enabling a granular relevance-based ranking comparison. Overall, the results demonstrate that including relevance judgments leads to significant deviations from traditional rank correlation metrics, highlighting the impact of relevance-aware measures in evaluating system performance and similarity.
Rank-Biased Overlap (RBO) is a widely used metric for comparing ranked lists, due to its ability to handle incomplete and non-conjoint rankings while emphasizing top-ranked items. However, traditional RBO only considers the identity of ranked items, ignoring any associated relevance values. In many real-world applications, different systems may retrieve non-overlapping documents with similar informational value. This paper proposes an extension of RBO that incorporates graded relevance scores, enabling the comparison of rankings based on the information they convey rather than shared items alone.
Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns. ...
Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns. ...
Rank-Biased Overlap (RBO) is a widely used metric for comparing ranked lists, due to its ability to handle incomplete and non-conjoint rankings while emphasizing top-ranked items. However, traditional RBO only considers the identity of ranked items, ignoring any associated relevance values. In many real-world applications, different systems may retrieve non-overlapping documents with similar informational value. This paper proposes an extension of RBO that incorporates graded relevance scores, enabling the comparison of rankings based on the information they convey rather than shared items alone.
Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns.
Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns.
Exploring Neural IR Approaches in Europeana
Unlocking Multilingual Insights for Cultural Heritage Search
Europeana is a digital library of Europe's cultural heritage, housing a large corpus of data representing artworks, literature, historical locations and many culturally significant items. Europeana currently relies of traditional text-matching retrieval, such as BM25, to facilitate their search and discovery across millions of multilingual metadata-based records. However, these models are not capable of semantic understanding and require additional treatments to facilitate multilingual retrieval which costs Europeana resources, these treatments entail translating queries and data from other language into English and enriching content by adding entities from linked open data. Europeana's current methodology is ultimately limited in its ability to provide semantically relevant multilingual search results.
This thesis investigates the application of Neural Information Retrieval (NIR) to enhance Europeana's search capabilities. This investigation aims to assess the impact of NIR on multilingual retrieval and retrieval performance while also determining the value of existing translation and enrichment processes. To support this investigation, we contribute by developing a structured and preprocessed dataset specifically for NIR, as no such dataset previously existed for NIR. We conduct an extensive evaluation of NIR models, analyzing the impact of fine-tuning, query treatments, and document treatments on retrieval quality. Additionally, we assess the computational requirements, scalability, and practicality of deploying NIR, identifying trade-offs in retrieval efficiency and resource consumption, to provide an idea of an infrastructure Europeana would need to implement NIR.
This research required meticulous planning across all stages—from data collection and formatting to model training and evaluation—since applying NIR at this scale for metadata search is new for Europeana. Therefore, research not only provides insights into the viability of NIR as a replacement or enhancement to Europeana's existing search system but also lays the foundation for future advancements in multilingual retrieval for Europeana.
Through this thesis, we found that NIR models can offer promising improvements in multilingual retrieval and semantic search, reducing reliance on exact term matching. Our analysis suggests that not all of Europeana’s current preprocessing treatments are necessary for NIR models, as they inherently capture cross-lingual relationships more effectively than BM25, though the benefits vary depending on the model and configuration used. Overall, we recommend that a hybrid retrieval system that leverages both lexical and neural approaches may be the most practical solution for Europeana and warrants further exploration.
The integration of NIR presents several challenges, particularly in terms of infrastructure and evaluation. NIR models are sensitive to changes in document structure and content, requiring careful consideration of indexing and fine-training. Furthermore, while these models improve semantic search, they may struggle with entity-based queries, where BM25’s exact matching approach remains valuable.
A major limitation of this study was the absence of explicit relevance judgements in our dataset, which constrained our ability to make definitive conclusions about retrieval effectiveness. Future work should prioritize the development of a comprehensive evaluation framework, incorporating expert and user-based relevance assessments, to enable a more robust analysis of NIR’s impact. ...
This thesis investigates the application of Neural Information Retrieval (NIR) to enhance Europeana's search capabilities. This investigation aims to assess the impact of NIR on multilingual retrieval and retrieval performance while also determining the value of existing translation and enrichment processes. To support this investigation, we contribute by developing a structured and preprocessed dataset specifically for NIR, as no such dataset previously existed for NIR. We conduct an extensive evaluation of NIR models, analyzing the impact of fine-tuning, query treatments, and document treatments on retrieval quality. Additionally, we assess the computational requirements, scalability, and practicality of deploying NIR, identifying trade-offs in retrieval efficiency and resource consumption, to provide an idea of an infrastructure Europeana would need to implement NIR.
This research required meticulous planning across all stages—from data collection and formatting to model training and evaluation—since applying NIR at this scale for metadata search is new for Europeana. Therefore, research not only provides insights into the viability of NIR as a replacement or enhancement to Europeana's existing search system but also lays the foundation for future advancements in multilingual retrieval for Europeana.
Through this thesis, we found that NIR models can offer promising improvements in multilingual retrieval and semantic search, reducing reliance on exact term matching. Our analysis suggests that not all of Europeana’s current preprocessing treatments are necessary for NIR models, as they inherently capture cross-lingual relationships more effectively than BM25, though the benefits vary depending on the model and configuration used. Overall, we recommend that a hybrid retrieval system that leverages both lexical and neural approaches may be the most practical solution for Europeana and warrants further exploration.
The integration of NIR presents several challenges, particularly in terms of infrastructure and evaluation. NIR models are sensitive to changes in document structure and content, requiring careful consideration of indexing and fine-training. Furthermore, while these models improve semantic search, they may struggle with entity-based queries, where BM25’s exact matching approach remains valuable.
A major limitation of this study was the absence of explicit relevance judgements in our dataset, which constrained our ability to make definitive conclusions about retrieval effectiveness. Future work should prioritize the development of a comprehensive evaluation framework, incorporating expert and user-based relevance assessments, to enable a more robust analysis of NIR’s impact. ...
Europeana is a digital library of Europe's cultural heritage, housing a large corpus of data representing artworks, literature, historical locations and many culturally significant items. Europeana currently relies of traditional text-matching retrieval, such as BM25, to facilitate their search and discovery across millions of multilingual metadata-based records. However, these models are not capable of semantic understanding and require additional treatments to facilitate multilingual retrieval which costs Europeana resources, these treatments entail translating queries and data from other language into English and enriching content by adding entities from linked open data. Europeana's current methodology is ultimately limited in its ability to provide semantically relevant multilingual search results.
This thesis investigates the application of Neural Information Retrieval (NIR) to enhance Europeana's search capabilities. This investigation aims to assess the impact of NIR on multilingual retrieval and retrieval performance while also determining the value of existing translation and enrichment processes. To support this investigation, we contribute by developing a structured and preprocessed dataset specifically for NIR, as no such dataset previously existed for NIR. We conduct an extensive evaluation of NIR models, analyzing the impact of fine-tuning, query treatments, and document treatments on retrieval quality. Additionally, we assess the computational requirements, scalability, and practicality of deploying NIR, identifying trade-offs in retrieval efficiency and resource consumption, to provide an idea of an infrastructure Europeana would need to implement NIR.
This research required meticulous planning across all stages—from data collection and formatting to model training and evaluation—since applying NIR at this scale for metadata search is new for Europeana. Therefore, research not only provides insights into the viability of NIR as a replacement or enhancement to Europeana's existing search system but also lays the foundation for future advancements in multilingual retrieval for Europeana.
Through this thesis, we found that NIR models can offer promising improvements in multilingual retrieval and semantic search, reducing reliance on exact term matching. Our analysis suggests that not all of Europeana’s current preprocessing treatments are necessary for NIR models, as they inherently capture cross-lingual relationships more effectively than BM25, though the benefits vary depending on the model and configuration used. Overall, we recommend that a hybrid retrieval system that leverages both lexical and neural approaches may be the most practical solution for Europeana and warrants further exploration.
The integration of NIR presents several challenges, particularly in terms of infrastructure and evaluation. NIR models are sensitive to changes in document structure and content, requiring careful consideration of indexing and fine-training. Furthermore, while these models improve semantic search, they may struggle with entity-based queries, where BM25’s exact matching approach remains valuable.
A major limitation of this study was the absence of explicit relevance judgements in our dataset, which constrained our ability to make definitive conclusions about retrieval effectiveness. Future work should prioritize the development of a comprehensive evaluation framework, incorporating expert and user-based relevance assessments, to enable a more robust analysis of NIR’s impact.
This thesis investigates the application of Neural Information Retrieval (NIR) to enhance Europeana's search capabilities. This investigation aims to assess the impact of NIR on multilingual retrieval and retrieval performance while also determining the value of existing translation and enrichment processes. To support this investigation, we contribute by developing a structured and preprocessed dataset specifically for NIR, as no such dataset previously existed for NIR. We conduct an extensive evaluation of NIR models, analyzing the impact of fine-tuning, query treatments, and document treatments on retrieval quality. Additionally, we assess the computational requirements, scalability, and practicality of deploying NIR, identifying trade-offs in retrieval efficiency and resource consumption, to provide an idea of an infrastructure Europeana would need to implement NIR.
This research required meticulous planning across all stages—from data collection and formatting to model training and evaluation—since applying NIR at this scale for metadata search is new for Europeana. Therefore, research not only provides insights into the viability of NIR as a replacement or enhancement to Europeana's existing search system but also lays the foundation for future advancements in multilingual retrieval for Europeana.
Through this thesis, we found that NIR models can offer promising improvements in multilingual retrieval and semantic search, reducing reliance on exact term matching. Our analysis suggests that not all of Europeana’s current preprocessing treatments are necessary for NIR models, as they inherently capture cross-lingual relationships more effectively than BM25, though the benefits vary depending on the model and configuration used. Overall, we recommend that a hybrid retrieval system that leverages both lexical and neural approaches may be the most practical solution for Europeana and warrants further exploration.
The integration of NIR presents several challenges, particularly in terms of infrastructure and evaluation. NIR models are sensitive to changes in document structure and content, requiring careful consideration of indexing and fine-training. Furthermore, while these models improve semantic search, they may struggle with entity-based queries, where BM25’s exact matching approach remains valuable.
A major limitation of this study was the absence of explicit relevance judgements in our dataset, which constrained our ability to make definitive conclusions about retrieval effectiveness. Future work should prioritize the development of a comprehensive evaluation framework, incorporating expert and user-based relevance assessments, to enable a more robust analysis of NIR’s impact.
Master thesis
(2024)
-
F. Xu, J. Urbano Merino, O.E. Scharenborg, J.C. van Gemert, Benjamin Timmermans, Z. Li
Visual impairment affects over 2.2 billion individuals globally, emphasizing the critical need for effective assistive technologies. This work focuses on developing a video captioning model explicitly tailored for visually impaired users, leveraging advancements in deep learning techniques. Video captioning involves converting video frames into textual descriptions, effectively bridging the domains of computer vision (CV) and natural language processing (NLP). We surveyed young visually impaired individuals from the Visio organization, who provided key insights into the design of our model.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions. ...
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions. ...
Visual impairment affects over 2.2 billion individuals globally, emphasizing the critical need for effective assistive technologies. This work focuses on developing a video captioning model explicitly tailored for visually impaired users, leveraging advancements in deep learning techniques. Video captioning involves converting video frames into textual descriptions, effectively bridging the domains of computer vision (CV) and natural language processing (NLP). We surveyed young visually impaired individuals from the Visio organization, who provided key insights into the design of our model.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions.
Rank-biased Overlap (RBO) is a measure that is used to compare two rankings against each other mathematically using a hyperparameter for persistence, p, to define the importance of items higher up in the rankings. This is able to follow the properties of incompleteness, indefiniteness, and top-heaviness for its results, making it a flexible option for rank similarity. In traditional RBO, the intersection of the items in each of the rankings is weighted by the persistence to reach the final value for RBO as it tends towards infinity. RBO has several assumptions, such as on what happens when rankings are tied, having an infinitely long ranking, and a degree of conjointness between the rankings. In this paper, two new variations are derived on the aspects of having the rankings be fully conjoint, as well as the aspect of having a known finite domain for the rankings. These are described through the equations of RBOc, for fully conjoint rankings, and RBOf , for rankings within a known finite domain. While RBOc tends to be slightly larger and RBOf tends to be smaller when compared to traditional RBO, both can be more fitting depending on the greater context of their use cases.
...
Rank-biased Overlap (RBO) is a measure that is used to compare two rankings against each other mathematically using a hyperparameter for persistence, p, to define the importance of items higher up in the rankings. This is able to follow the properties of incompleteness, indefiniteness, and top-heaviness for its results, making it a flexible option for rank similarity. In traditional RBO, the intersection of the items in each of the rankings is weighted by the persistence to reach the final value for RBO as it tends towards infinity. RBO has several assumptions, such as on what happens when rankings are tied, having an infinitely long ranking, and a degree of conjointness between the rankings. In this paper, two new variations are derived on the aspects of having the rankings be fully conjoint, as well as the aspect of having a known finite domain for the rankings. These are described through the equations of RBOc, for fully conjoint rankings, and RBOf , for rankings within a known finite domain. While RBOc tends to be slightly larger and RBOf tends to be smaller when compared to traditional RBO, both can be more fitting depending on the greater context of their use cases.
As a point estimate of the similarity score between two possibly indefinite rankings, extrapolated rank-biased overlap (RBOEXT) uses the assumption that the agreement observed at the last evaluation depth continues indefinitely across the unseen tails of the two lists. This assumption does not account for any patterns that occur in the visible prefixes, imposing a strict restriction on the extrapolation. In an effort to improve the accuracy of RBOEXT, three reformulations with a relaxed theoretical basis are proposed in this paper: one continually re-uses the agreement from the previous depth while the other two rely on regression to fit a function on the seen agreements. Using synthetic data, the performance of these new extrapolation methods is compared to the original's in terms of closeness to the true RBO score as well as the average distance between assumed and actual agreement in the rankings' unseen tails. Overall, an impactful difference is observed in the estimates of agreement generated by the four approaches: as the trends from the visible prefixes are barely captured by the simpler techniques or closely-reproduced by the more flexible ones, the trade-off between under- and overfitting becomes increasingly relevant. The results thus indicate a need for some middle-ground to be established such that it factors in the observed patterns while also generalizing well for the tails.
...
As a point estimate of the similarity score between two possibly indefinite rankings, extrapolated rank-biased overlap (RBOEXT) uses the assumption that the agreement observed at the last evaluation depth continues indefinitely across the unseen tails of the two lists. This assumption does not account for any patterns that occur in the visible prefixes, imposing a strict restriction on the extrapolation. In an effort to improve the accuracy of RBOEXT, three reformulations with a relaxed theoretical basis are proposed in this paper: one continually re-uses the agreement from the previous depth while the other two rely on regression to fit a function on the seen agreements. Using synthetic data, the performance of these new extrapolation methods is compared to the original's in terms of closeness to the true RBO score as well as the average distance between assumed and actual agreement in the rankings' unseen tails. Overall, an impactful difference is observed in the estimates of agreement generated by the four approaches: as the trends from the visible prefixes are barely captured by the simpler techniques or closely-reproduced by the more flexible ones, the trade-off between under- and overfitting becomes increasingly relevant. The results thus indicate a need for some middle-ground to be established such that it factors in the observed patterns while also generalizing well for the tails.
Average Rank-Biased Overlap between independent rankings
Revealing average benchmarks: An Empirical Investigation
Rankings play a crucial role in various contexts but often exhibit incompleteness, top-weightedness, and indefiniteness. Comparing rankings can reveal underlying similarities, yet traditional correlation coefficients like Kendall's tau do not adequately address these complexities. Rank-Biased Overlap (RBO) addresses these challenges by accommodating differences in rank length, appropriately weighting ranks, and minimizing data assumptions. This paper investigates the average Rank-Biased Overlap (RBO) between independent rankings, addressing the need for clearly indicated reference values similar to those of correlation coefficients. Our study explores how the expected RBO changes with varying p-parameters, prefix lengths, and degrees of conjointness between domains.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings. ...
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings. ...
Rankings play a crucial role in various contexts but often exhibit incompleteness, top-weightedness, and indefiniteness. Comparing rankings can reveal underlying similarities, yet traditional correlation coefficients like Kendall's tau do not adequately address these complexities. Rank-Biased Overlap (RBO) addresses these challenges by accommodating differences in rank length, appropriately weighting ranks, and minimizing data assumptions. This paper investigates the average Rank-Biased Overlap (RBO) between independent rankings, addressing the need for clearly indicated reference values similar to those of correlation coefficients. Our study explores how the expected RBO changes with varying p-parameters, prefix lengths, and degrees of conjointness between domains.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings.
Effects of the assumption on ties in unseen parts of a ranking
What will happen if we relax the assumption that ties do not occur in unseen parts?
Rankings are more present in our daily lives than most people realize. Whether you are browsing Netflix and getting movies or shows based on your previous likes or dislikes, or you want to compare search engine results. To use rankings in the field of Computer Science a rank similarity is needed. Rank-Biased Overlap is one of those. It is top-weighted, can be used on uneven rankings, and when only a part of the ranking is known. A well-known problem in rank similarity measures is ties. There have been some ways of dealing with ties proposed since RBO was introduced. These ways have been shown to be promising but they only relate to the seen part. The unseen part of rankings is still a new concept with little research done about it. This paper aims to change that a bit. First, a full explanation is given of the three variations of dealing with ties. Then using these variants we show how the assumption that no ties exist in the unseen part affects these variants. Also, the current extrapolation method is researched as there is also a big influence of the above-mentioned assumption. We then use simulated data to give a clear data visualization to show how the theory relates to practice. We have tried to be clear and concise with our explanations and data visualizations so future researchers can use this paper to improve and progress RBO in the world of rank similarity measures.
...
Rankings are more present in our daily lives than most people realize. Whether you are browsing Netflix and getting movies or shows based on your previous likes or dislikes, or you want to compare search engine results. To use rankings in the field of Computer Science a rank similarity is needed. Rank-Biased Overlap is one of those. It is top-weighted, can be used on uneven rankings, and when only a part of the ranking is known. A well-known problem in rank similarity measures is ties. There have been some ways of dealing with ties proposed since RBO was introduced. These ways have been shown to be promising but they only relate to the seen part. The unseen part of rankings is still a new concept with little research done about it. This paper aims to change that a bit. First, a full explanation is given of the three variations of dealing with ties. Then using these variants we show how the assumption that no ties exist in the unseen part affects these variants. Also, the current extrapolation method is researched as there is also a big influence of the above-mentioned assumption. We then use simulated data to give a clear data visualization to show how the theory relates to practice. We have tried to be clear and concise with our explanations and data visualizations so future researchers can use this paper to improve and progress RBO in the world of rank similarity measures.
Adaptive Synthetic Generation of Indefinite Rankings
Enhancing Algorithm Flexibility with Tunable Conjointness, Overlap, and Tie Distribution
Reducing the similarity of two ranked lists to a single value proves to be useful in various fields of research and industry, such as Information Retrieval and Recommender Systems, leading to the introduction of several similarity measures. One such measure is Rank-Biased Overlap and its variants, possessing qualities such as the ability to handle incomplete rankings, non-conjoint ranking domains and the graceful evaluation of ranking pairs with tied items.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically. ...
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically. ...
Reducing the similarity of two ranked lists to a single value proves to be useful in various fields of research and industry, such as Information Retrieval and Recommender Systems, leading to the introduction of several similarity measures. One such measure is Rank-Biased Overlap and its variants, possessing qualities such as the ability to handle incomplete rankings, non-conjoint ranking domains and the graceful evaluation of ranking pairs with tied items.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically.
Recommender Systems have drawn extensive attention in recent decades, because they are a powerful tool with the potential to help several business stakeholders –including end users, sellers, and platformproviders– through personalized recommendations. The most important factor to make a recommender succeed is user satisfaction, which is largely reflected by the recommendation accuracy. Therefore, one primary question in recommender systems research is how to make all users enjoy good recommendation accuracy. This thesis dives into this question from two different perspectives that, unfortunately, are at tension with each other: achieving the maximum overall recommendation accuracy, and balancing that accuracy among all users.
The first part of this thesis focuses on the first perspective, that is, maximizing the overall recommendation accuracy. This accuracy is usually evaluated with some useroriented metric tailored to the recommendation scenario, but because recommendation is usually treated as a machine learning problem, recommendation models are trained to maximize some other generic criteria that does not necessarily align with the criteria ultimately captured by the user-oriented evaluation metric. Recent research aims at bridging this gap between training and evaluation via direct ranking optimization, but still assumes that the metric used for evaluation should also be the metric used for training. We challenge this assumption, mainly because some metrics are more informative than others. Indeed, we show that models trained via the optimization of a loss inspired by Rank-Biased Precision (RBP) tend to yield higher accuracy, even when accuracy is measured with metrics other than RBP. However, the superiority of this RBP-inspired loss stems from further benefiting users who are already well-served, rather than helping those who are not.
This observation inspires the second part of this thesis, where our focus turns to helping non-mainstream users. These are users who are difficult to recommend to either because there is not enough data to model them, or because they have niche taste and thus few similar users to look at when recommending in a collaborative way. These differences in mainstreamness introduce a bias reflected in an accuracy gap between users or user groups, which we try to narrow.
Our first effort consists in using side data, beyond the user-item interaction matrix, so that users and items are better represented in the recommendation model. This will be of benefit specially for the non-mainstream users, for which the user-item matrix alone is ineffective. We propose Neural AutoEncoder Collaborative Filtering (NAECF), an adversarial learning architecture that, in addition to maximizing the recommendation accuracy, leverages side data to preserve the intrinsic properties of users and items. We experiment with review texts as side data, and show that NAECF leads to better recommendations specially for non-mainstream users, while at the same time there is a marginal loss for the mainstream ones.
Our second effort consists in explicitly signaling to the training process what users it should focus on, that is, the non-mainstream ones. In particular, we propose a mechanism based on cost-sensitive learning that weighs users according to their mainstreamness, so that they get more attention during training. Here we argue for not quantifying mainstreamness directly, but rather its effect, and therefore weigh users depending on how well they are served by a vanilla recommendation model. The result is a recommendation model tailored to non-mainstream users, that narrows the accuracy gap, and again at negligible cost to the mainstream users.
...
The first part of this thesis focuses on the first perspective, that is, maximizing the overall recommendation accuracy. This accuracy is usually evaluated with some useroriented metric tailored to the recommendation scenario, but because recommendation is usually treated as a machine learning problem, recommendation models are trained to maximize some other generic criteria that does not necessarily align with the criteria ultimately captured by the user-oriented evaluation metric. Recent research aims at bridging this gap between training and evaluation via direct ranking optimization, but still assumes that the metric used for evaluation should also be the metric used for training. We challenge this assumption, mainly because some metrics are more informative than others. Indeed, we show that models trained via the optimization of a loss inspired by Rank-Biased Precision (RBP) tend to yield higher accuracy, even when accuracy is measured with metrics other than RBP. However, the superiority of this RBP-inspired loss stems from further benefiting users who are already well-served, rather than helping those who are not.
This observation inspires the second part of this thesis, where our focus turns to helping non-mainstream users. These are users who are difficult to recommend to either because there is not enough data to model them, or because they have niche taste and thus few similar users to look at when recommending in a collaborative way. These differences in mainstreamness introduce a bias reflected in an accuracy gap between users or user groups, which we try to narrow.
Our first effort consists in using side data, beyond the user-item interaction matrix, so that users and items are better represented in the recommendation model. This will be of benefit specially for the non-mainstream users, for which the user-item matrix alone is ineffective. We propose Neural AutoEncoder Collaborative Filtering (NAECF), an adversarial learning architecture that, in addition to maximizing the recommendation accuracy, leverages side data to preserve the intrinsic properties of users and items. We experiment with review texts as side data, and show that NAECF leads to better recommendations specially for non-mainstream users, while at the same time there is a marginal loss for the mainstream ones.
Our second effort consists in explicitly signaling to the training process what users it should focus on, that is, the non-mainstream ones. In particular, we propose a mechanism based on cost-sensitive learning that weighs users according to their mainstreamness, so that they get more attention during training. Here we argue for not quantifying mainstreamness directly, but rather its effect, and therefore weigh users depending on how well they are served by a vanilla recommendation model. The result is a recommendation model tailored to non-mainstream users, that narrows the accuracy gap, and again at negligible cost to the mainstream users.
...
Recommender Systems have drawn extensive attention in recent decades, because they are a powerful tool with the potential to help several business stakeholders –including end users, sellers, and platformproviders– through personalized recommendations. The most important factor to make a recommender succeed is user satisfaction, which is largely reflected by the recommendation accuracy. Therefore, one primary question in recommender systems research is how to make all users enjoy good recommendation accuracy. This thesis dives into this question from two different perspectives that, unfortunately, are at tension with each other: achieving the maximum overall recommendation accuracy, and balancing that accuracy among all users.
The first part of this thesis focuses on the first perspective, that is, maximizing the overall recommendation accuracy. This accuracy is usually evaluated with some useroriented metric tailored to the recommendation scenario, but because recommendation is usually treated as a machine learning problem, recommendation models are trained to maximize some other generic criteria that does not necessarily align with the criteria ultimately captured by the user-oriented evaluation metric. Recent research aims at bridging this gap between training and evaluation via direct ranking optimization, but still assumes that the metric used for evaluation should also be the metric used for training. We challenge this assumption, mainly because some metrics are more informative than others. Indeed, we show that models trained via the optimization of a loss inspired by Rank-Biased Precision (RBP) tend to yield higher accuracy, even when accuracy is measured with metrics other than RBP. However, the superiority of this RBP-inspired loss stems from further benefiting users who are already well-served, rather than helping those who are not.
This observation inspires the second part of this thesis, where our focus turns to helping non-mainstream users. These are users who are difficult to recommend to either because there is not enough data to model them, or because they have niche taste and thus few similar users to look at when recommending in a collaborative way. These differences in mainstreamness introduce a bias reflected in an accuracy gap between users or user groups, which we try to narrow.
Our first effort consists in using side data, beyond the user-item interaction matrix, so that users and items are better represented in the recommendation model. This will be of benefit specially for the non-mainstream users, for which the user-item matrix alone is ineffective. We propose Neural AutoEncoder Collaborative Filtering (NAECF), an adversarial learning architecture that, in addition to maximizing the recommendation accuracy, leverages side data to preserve the intrinsic properties of users and items. We experiment with review texts as side data, and show that NAECF leads to better recommendations specially for non-mainstream users, while at the same time there is a marginal loss for the mainstream ones.
Our second effort consists in explicitly signaling to the training process what users it should focus on, that is, the non-mainstream ones. In particular, we propose a mechanism based on cost-sensitive learning that weighs users according to their mainstreamness, so that they get more attention during training. Here we argue for not quantifying mainstreamness directly, but rather its effect, and therefore weigh users depending on how well they are served by a vanilla recommendation model. The result is a recommendation model tailored to non-mainstream users, that narrows the accuracy gap, and again at negligible cost to the mainstream users.
The first part of this thesis focuses on the first perspective, that is, maximizing the overall recommendation accuracy. This accuracy is usually evaluated with some useroriented metric tailored to the recommendation scenario, but because recommendation is usually treated as a machine learning problem, recommendation models are trained to maximize some other generic criteria that does not necessarily align with the criteria ultimately captured by the user-oriented evaluation metric. Recent research aims at bridging this gap between training and evaluation via direct ranking optimization, but still assumes that the metric used for evaluation should also be the metric used for training. We challenge this assumption, mainly because some metrics are more informative than others. Indeed, we show that models trained via the optimization of a loss inspired by Rank-Biased Precision (RBP) tend to yield higher accuracy, even when accuracy is measured with metrics other than RBP. However, the superiority of this RBP-inspired loss stems from further benefiting users who are already well-served, rather than helping those who are not.
This observation inspires the second part of this thesis, where our focus turns to helping non-mainstream users. These are users who are difficult to recommend to either because there is not enough data to model them, or because they have niche taste and thus few similar users to look at when recommending in a collaborative way. These differences in mainstreamness introduce a bias reflected in an accuracy gap between users or user groups, which we try to narrow.
Our first effort consists in using side data, beyond the user-item interaction matrix, so that users and items are better represented in the recommendation model. This will be of benefit specially for the non-mainstream users, for which the user-item matrix alone is ineffective. We propose Neural AutoEncoder Collaborative Filtering (NAECF), an adversarial learning architecture that, in addition to maximizing the recommendation accuracy, leverages side data to preserve the intrinsic properties of users and items. We experiment with review texts as side data, and show that NAECF leads to better recommendations specially for non-mainstream users, while at the same time there is a marginal loss for the mainstream ones.
Our second effort consists in explicitly signaling to the training process what users it should focus on, that is, the non-mainstream ones. In particular, we propose a mechanism based on cost-sensitive learning that weighs users according to their mainstreamness, so that they get more attention during training. Here we argue for not quantifying mainstreamness directly, but rather its effect, and therefore weigh users depending on how well they are served by a vanilla recommendation model. The result is a recommendation model tailored to non-mainstream users, that narrows the accuracy gap, and again at negligible cost to the mainstream users.
Learning to Rank is the application of Machine Learning in order to create and optimize ranking functions. Most Learning to Rank methods follow a listwise approach and optimize a listwise loss function which closely resembles the same metric used in the evaluation. Popular listwise loss functions such as nDCG, AP and nRBP do not have consistent bounds across topics and do not account for instance-difficulty. As a result, the loss score does not solely reflect the performance of the model but also depends on the instance properties. During training, each instance is assumed to be equally informative, while in reality, this informativeness might depend on the difficulty of the instance. In this thesis, we propose four bounding methods which utilize some notion of instance-difficulty to produce difficulty-aware losses. Experimental results showed that, in most cases, optimizing a bounded variant of nDCG, AP or nRBP results in a consistent but marginal increase in the overall performance. More interestingly, we found that optimizing a bounded variant of nRBP and AP may increase the nDCG@k score, increasing the recommendation utility. Overall, our results show promising results for user/query-wise metric bounding in Learning to Rank, especially when applied to nRBP.
...
Learning to Rank is the application of Machine Learning in order to create and optimize ranking functions. Most Learning to Rank methods follow a listwise approach and optimize a listwise loss function which closely resembles the same metric used in the evaluation. Popular listwise loss functions such as nDCG, AP and nRBP do not have consistent bounds across topics and do not account for instance-difficulty. As a result, the loss score does not solely reflect the performance of the model but also depends on the instance properties. During training, each instance is assumed to be equally informative, while in reality, this informativeness might depend on the difficulty of the instance. In this thesis, we propose four bounding methods which utilize some notion of instance-difficulty to produce difficulty-aware losses. Experimental results showed that, in most cases, optimizing a bounded variant of nDCG, AP or nRBP results in a consistent but marginal increase in the overall performance. More interestingly, we found that optimizing a bounded variant of nRBP and AP may increase the nDCG@k score, increasing the recommendation utility. Overall, our results show promising results for user/query-wise metric bounding in Learning to Rank, especially when applied to nRBP.
In the field of Information Retrieval (IR), the reliable evaluation of systems is a key component in order to progress the state-of-the-art. Much of IR research focuses on optimizing the various aspects of evaluation. Stochastic simulation is one technique that can be used to assist this kind of research. It allows researchers to overcome certain limitations associated with IR data, such as limited size, and lack of control. Recently, there have been two parallel lines of work that use stochastic simulation to study the question of "which statistical significance test is optimal for IR evaluation data?". Surprisingly, the authors reach different conclusions, despite the fact that both use stochastic simulation. One line of work, lead by Urbano et al., simulates scores for a fixed set of systems on new random topics, and concluded that the t-test is optimal. Another line of work, lead by Parapar et al., simulates new random retrieval runs for a fixed set of topics, and concluded that the Wilcoxon test is optimal. Interestingly these two tests are the most popular in IR literature. In an attempt to shed some light on this disagreement between the two conclusions, we made a first attempt at providing some empirical evidence regarding the quality of the simulation approach that was used by Urbano et al. Our main findings is that the quality of the simulation is moderately good, and also discovered some opportunities to refine it. In addition, we proposed a new model selection criterion, that showed some promising results, and in many cases managed to select models more optimally than other, more established criteria, such as AIC.
...
In the field of Information Retrieval (IR), the reliable evaluation of systems is a key component in order to progress the state-of-the-art. Much of IR research focuses on optimizing the various aspects of evaluation. Stochastic simulation is one technique that can be used to assist this kind of research. It allows researchers to overcome certain limitations associated with IR data, such as limited size, and lack of control. Recently, there have been two parallel lines of work that use stochastic simulation to study the question of "which statistical significance test is optimal for IR evaluation data?". Surprisingly, the authors reach different conclusions, despite the fact that both use stochastic simulation. One line of work, lead by Urbano et al., simulates scores for a fixed set of systems on new random topics, and concluded that the t-test is optimal. Another line of work, lead by Parapar et al., simulates new random retrieval runs for a fixed set of topics, and concluded that the Wilcoxon test is optimal. Interestingly these two tests are the most popular in IR literature. In an attempt to shed some light on this disagreement between the two conclusions, we made a first attempt at providing some empirical evidence regarding the quality of the simulation approach that was used by Urbano et al. Our main findings is that the quality of the simulation is moderately good, and also discovered some opportunities to refine it. In addition, we proposed a new model selection criterion, that showed some promising results, and in many cases managed to select models more optimally than other, more established criteria, such as AIC.
With recent advances in performance and complexity, multi-party computation, a privacy-preserving technology which allows for joint processing of hidden input data, has lately been found to be applicable in a number of use cases. Despite existing implementations for secure data aggregation, substantial adoptions of the technology remain limited in the industry, in particular within the domain of smart mobility. This paper addresses the current issue of the mobility data shortage by investigating the potential and feasibility of multi-party computation to share data with policy makers, and proposes a solution based on additive secret sharing. On the basis of a literature study and interviews with infrastructure management authorities, as well as micro-mobility service providers, the drivers of, and barriers to employing a secure data aggregation scheme were identified. The results suggest that the technical solution appears feasible given existing implementations, while trust, acceptance and willingness of participants emerged as obstacles to a realisation.
...
With recent advances in performance and complexity, multi-party computation, a privacy-preserving technology which allows for joint processing of hidden input data, has lately been found to be applicable in a number of use cases. Despite existing implementations for secure data aggregation, substantial adoptions of the technology remain limited in the industry, in particular within the domain of smart mobility. This paper addresses the current issue of the mobility data shortage by investigating the potential and feasibility of multi-party computation to share data with policy makers, and proposes a solution based on additive secret sharing. On the basis of a literature study and interviews with infrastructure management authorities, as well as micro-mobility service providers, the drivers of, and barriers to employing a secure data aggregation scheme were identified. The results suggest that the technical solution appears feasible given existing implementations, while trust, acceptance and willingness of participants emerged as obstacles to a realisation.
Secure MPC-Sortition
Consolidating Innovations in Democracy and Cryptography
Globally, citizens’ assemblies have been gaining momentum as a way to counter dissatisfaction in democracies. Central to the citizens’ assembly is sortition, the process of randomly selecting political representatives given certain demographic criteria. In order to have an assembly representative of the population, personal data is necessary to perform the sortition, making participant privacy a matter of concern when guaranteeing fairness of the process. Secure multi-party computation (MPC)makes it possible to perform calculations on encrypted data from multiple sources without revealing it to the other processing parties or data contributors. This paper outlines two sortition designs that use MPC to guarantee participant privacy during the sortition process. Design 1 allows participants to hide their personal details to the citizens’ assembly organisers. Design 2 makes it possible for involved local government to contribute citizen data to the sortition. Different domain experts were interviewed as a means to survey the needs of the sortition community and to provide feedback on the designs. We found that little research has yet been conducted on the effects of increased privacy on sign-up rates for deliberative events, creating a potential avenue for future research.
...
Globally, citizens’ assemblies have been gaining momentum as a way to counter dissatisfaction in democracies. Central to the citizens’ assembly is sortition, the process of randomly selecting political representatives given certain demographic criteria. In order to have an assembly representative of the population, personal data is necessary to perform the sortition, making participant privacy a matter of concern when guaranteeing fairness of the process. Secure multi-party computation (MPC)makes it possible to perform calculations on encrypted data from multiple sources without revealing it to the other processing parties or data contributors. This paper outlines two sortition designs that use MPC to guarantee participant privacy during the sortition process. Design 1 allows participants to hide their personal details to the citizens’ assembly organisers. Design 2 makes it possible for involved local government to contribute citizen data to the sortition. Different domain experts were interviewed as a means to survey the needs of the sortition community and to provide feedback on the designs. We found that little research has yet been conducted on the effects of increased privacy on sign-up rates for deliberative events, creating a potential avenue for future research.
Around the world millions of people get injured due to traffic accidents. Autonomous vehicles are expected to significantly reduce these numbers. To increase safety, autonomous vehicle communication can be used. Current vehicle communication networks called VANETs have security and privacy protection problems and the vehicle industry is reluctant to use them. Multi-Party Computation (MPC) is a cryptographic technique which allows a set of parties to compute the output of a function while not revealing the input. This paper investigates the potential and feasibility of multi-party computation to solve the present problems of autonomous vehicle communication. Two architectures are proposed and discussed on the basis of a literature study and interviews with experts in MPC and autonomous vehicle communication. The solutions seem to be feasible, but further experimentation is required to confirm this. The main obstacle for the implementation of MPC in autonomous vehicle communication is that at the moment, the autonomous vehicle industry is not concerned with improving autonomous vehicle communication.
...
Around the world millions of people get injured due to traffic accidents. Autonomous vehicles are expected to significantly reduce these numbers. To increase safety, autonomous vehicle communication can be used. Current vehicle communication networks called VANETs have security and privacy protection problems and the vehicle industry is reluctant to use them. Multi-Party Computation (MPC) is a cryptographic technique which allows a set of parties to compute the output of a function while not revealing the input. This paper investigates the potential and feasibility of multi-party computation to solve the present problems of autonomous vehicle communication. Two architectures are proposed and discussed on the basis of a literature study and interviews with experts in MPC and autonomous vehicle communication. The solutions seem to be feasible, but further experimentation is required to confirm this. The main obstacle for the implementation of MPC in autonomous vehicle communication is that at the moment, the autonomous vehicle industry is not concerned with improving autonomous vehicle communication.