Z. Yue
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
14 records found
1
Backdoor Attacks in Active Learning
An Extensive Analysis of Backdoor Injection in Active Learning-Trained Computer Vision Models
Deep learning sustained great success in several domains, particularly in computer vision, where it facilitates tasks such as image classification and object recognition. However, one significant challenge in deep learning is data labeling, due to the high cost and effort required for human annotators to go over this process manually. Active learning addresses this problem by selecting a smaller amount of the most relevant data for this labeling process, maximizing efficiency. Despite its advantages, active learning presents new security threats. In particular, backdoor attacks, where adversaries poison part of the training data to modify the behavior of the model in the presence of a hidden trigger. Although backdoor attacks have been extensively studied in traditional deep learning contexts, their impact on active learning remains largely uncertain.
Here, the vulnerabilities of active learning against backdoor attacks in computer vision models were analyzed. Various configurations, datasets and deep learning models were used to evaluate their effectiveness. Backdoor attacks managed to hit ASR values over 95% with just 1% of the data being poisoned on simple datasets like MNIST, particularly when using certainty-based sampling and CNNs. More complex datasets like CIFAR-10 and models like ResNet proved to be more resilient. Furthermore, different attack techniques were explored, such as progressive parameter adjustment, sub-trigger division and clean label attacks on advanced backdoor triggers like LIRA and WaNet. The analysis revealed that although global LIRA triggers were the most effective, sub-trigger and progressive poisoning methods offered promising alternatives, especially because they allow poisoning smaller parts of images across training cycles. Additionally, it is revealed that attack success in clean-label scenarios was highly dependent on the number of poisoned samples per cycle, due to the post-query constraint that only allows poisoning already-selected samples, often limiting impact when the target label appears infrequently. Finally, different poisoning timings were compared. From this, post-query poisoning consistently outperformed pre-query methods in terms of ASR, even at low poison rates. However, it also pointed out that this approach has its limitations in real-world scenarios, where attackers usually do not have control over the samples being queried. Clean accuracy remained unaltered to a large extent, demonstrating the stealth and hidden threat of backdoor attacks in active learning settings.
...
Here, the vulnerabilities of active learning against backdoor attacks in computer vision models were analyzed. Various configurations, datasets and deep learning models were used to evaluate their effectiveness. Backdoor attacks managed to hit ASR values over 95% with just 1% of the data being poisoned on simple datasets like MNIST, particularly when using certainty-based sampling and CNNs. More complex datasets like CIFAR-10 and models like ResNet proved to be more resilient. Furthermore, different attack techniques were explored, such as progressive parameter adjustment, sub-trigger division and clean label attacks on advanced backdoor triggers like LIRA and WaNet. The analysis revealed that although global LIRA triggers were the most effective, sub-trigger and progressive poisoning methods offered promising alternatives, especially because they allow poisoning smaller parts of images across training cycles. Additionally, it is revealed that attack success in clean-label scenarios was highly dependent on the number of poisoned samples per cycle, due to the post-query constraint that only allows poisoning already-selected samples, often limiting impact when the target label appears infrequently. Finally, different poisoning timings were compared. From this, post-query poisoning consistently outperformed pre-query methods in terms of ASR, even at low poison rates. However, it also pointed out that this approach has its limitations in real-world scenarios, where attackers usually do not have control over the samples being queried. Clean accuracy remained unaltered to a large extent, demonstrating the stealth and hidden threat of backdoor attacks in active learning settings.
...
Deep learning sustained great success in several domains, particularly in computer vision, where it facilitates tasks such as image classification and object recognition. However, one significant challenge in deep learning is data labeling, due to the high cost and effort required for human annotators to go over this process manually. Active learning addresses this problem by selecting a smaller amount of the most relevant data for this labeling process, maximizing efficiency. Despite its advantages, active learning presents new security threats. In particular, backdoor attacks, where adversaries poison part of the training data to modify the behavior of the model in the presence of a hidden trigger. Although backdoor attacks have been extensively studied in traditional deep learning contexts, their impact on active learning remains largely uncertain.
Here, the vulnerabilities of active learning against backdoor attacks in computer vision models were analyzed. Various configurations, datasets and deep learning models were used to evaluate their effectiveness. Backdoor attacks managed to hit ASR values over 95% with just 1% of the data being poisoned on simple datasets like MNIST, particularly when using certainty-based sampling and CNNs. More complex datasets like CIFAR-10 and models like ResNet proved to be more resilient. Furthermore, different attack techniques were explored, such as progressive parameter adjustment, sub-trigger division and clean label attacks on advanced backdoor triggers like LIRA and WaNet. The analysis revealed that although global LIRA triggers were the most effective, sub-trigger and progressive poisoning methods offered promising alternatives, especially because they allow poisoning smaller parts of images across training cycles. Additionally, it is revealed that attack success in clean-label scenarios was highly dependent on the number of poisoned samples per cycle, due to the post-query constraint that only allows poisoning already-selected samples, often limiting impact when the target label appears infrequently. Finally, different poisoning timings were compared. From this, post-query poisoning consistently outperformed pre-query methods in terms of ASR, even at low poison rates. However, it also pointed out that this approach has its limitations in real-world scenarios, where attackers usually do not have control over the samples being queried. Clean accuracy remained unaltered to a large extent, demonstrating the stealth and hidden threat of backdoor attacks in active learning settings.
Here, the vulnerabilities of active learning against backdoor attacks in computer vision models were analyzed. Various configurations, datasets and deep learning models were used to evaluate their effectiveness. Backdoor attacks managed to hit ASR values over 95% with just 1% of the data being poisoned on simple datasets like MNIST, particularly when using certainty-based sampling and CNNs. More complex datasets like CIFAR-10 and models like ResNet proved to be more resilient. Furthermore, different attack techniques were explored, such as progressive parameter adjustment, sub-trigger division and clean label attacks on advanced backdoor triggers like LIRA and WaNet. The analysis revealed that although global LIRA triggers were the most effective, sub-trigger and progressive poisoning methods offered promising alternatives, especially because they allow poisoning smaller parts of images across training cycles. Additionally, it is revealed that attack success in clean-label scenarios was highly dependent on the number of poisoned samples per cycle, due to the post-query constraint that only allows poisoning already-selected samples, often limiting impact when the target label appears infrequently. Finally, different poisoning timings were compared. From this, post-query poisoning consistently outperformed pre-query methods in terms of ASR, even at low poison rates. However, it also pointed out that this approach has its limitations in real-world scenarios, where attackers usually do not have control over the samples being queried. Clean accuracy remained unaltered to a large extent, demonstrating the stealth and hidden threat of backdoor attacks in active learning settings.
Organisations are becoming more conscious and deploying more and more security tools to ensure they are adequately protected against cyber-attacks. That means two things: (i) those extra tools inherently augment companies’ attack surface, and (ii) the Security Operations Centre (SOC) gets overwhelmed with the number of false positives those tools generate – leading to attack fatigue. In many cases, the SOC team cannot get through all alerts properly, allowing potential attacks to go unnoticed or be caught much later. Moreover, within a typical CISO organisation, the analysis of “attack” and “defence” data is done somewhat in silos. That means vulnerability data, red-team exercises, and the several available defence tooling data are not looked at as one.
Our work proposes an innovative way to bridge the gap between vulnerability data (CVEs) and security alert data originating from multiple security tools that protect servers using MITRE ATT&CK tactics. That would provide more context to the alerts which would be useful in their classification as attacks or false positives. We use DeBERTa (Decoding-enhanced BERT with Disentangled Attention), a deeplearning state-of-the-art model, to map CVE descriptions to MITRE ATT&CK tactics. Then, we map security alerts to MITRE ATT&CK tactics, which will be used as input to a context-enriched machinelearning model (by CVEs and tactics). That machine-learning model is used to classify security alerts as malicious or benign. We tested our approach using over 5.5 million security alert data combined with red-team exercise attacks and incident response labelling from the company, a large international organization with over 60,000 employees. Our CVE+tactic model (without hyperparameter tuning) detects 64% more true positives than the machine-learning model without that information. In addition, the SOC needs to investigate less than 1400 alerts to catch the red-team attacks in our test set, compared to more than 5500 generated by the model without CVE and tactics. Moreover, assuming a standard response time of 8 minutes per alert, this improved model would save the SOC team up to 550 person hours. That yields a model that catches red-team attacks without overwhelming the SOC with too many false positives. ...
Our work proposes an innovative way to bridge the gap between vulnerability data (CVEs) and security alert data originating from multiple security tools that protect servers using MITRE ATT&CK tactics. That would provide more context to the alerts which would be useful in their classification as attacks or false positives. We use DeBERTa (Decoding-enhanced BERT with Disentangled Attention), a deeplearning state-of-the-art model, to map CVE descriptions to MITRE ATT&CK tactics. Then, we map security alerts to MITRE ATT&CK tactics, which will be used as input to a context-enriched machinelearning model (by CVEs and tactics). That machine-learning model is used to classify security alerts as malicious or benign. We tested our approach using over 5.5 million security alert data combined with red-team exercise attacks and incident response labelling from the company, a large international organization with over 60,000 employees. Our CVE+tactic model (without hyperparameter tuning) detects 64% more true positives than the machine-learning model without that information. In addition, the SOC needs to investigate less than 1400 alerts to catch the red-team attacks in our test set, compared to more than 5500 generated by the model without CVE and tactics. Moreover, assuming a standard response time of 8 minutes per alert, this improved model would save the SOC team up to 550 person hours. That yields a model that catches red-team attacks without overwhelming the SOC with too many false positives. ...
Organisations are becoming more conscious and deploying more and more security tools to ensure they are adequately protected against cyber-attacks. That means two things: (i) those extra tools inherently augment companies’ attack surface, and (ii) the Security Operations Centre (SOC) gets overwhelmed with the number of false positives those tools generate – leading to attack fatigue. In many cases, the SOC team cannot get through all alerts properly, allowing potential attacks to go unnoticed or be caught much later. Moreover, within a typical CISO organisation, the analysis of “attack” and “defence” data is done somewhat in silos. That means vulnerability data, red-team exercises, and the several available defence tooling data are not looked at as one.
Our work proposes an innovative way to bridge the gap between vulnerability data (CVEs) and security alert data originating from multiple security tools that protect servers using MITRE ATT&CK tactics. That would provide more context to the alerts which would be useful in their classification as attacks or false positives. We use DeBERTa (Decoding-enhanced BERT with Disentangled Attention), a deeplearning state-of-the-art model, to map CVE descriptions to MITRE ATT&CK tactics. Then, we map security alerts to MITRE ATT&CK tactics, which will be used as input to a context-enriched machinelearning model (by CVEs and tactics). That machine-learning model is used to classify security alerts as malicious or benign. We tested our approach using over 5.5 million security alert data combined with red-team exercise attacks and incident response labelling from the company, a large international organization with over 60,000 employees. Our CVE+tactic model (without hyperparameter tuning) detects 64% more true positives than the machine-learning model without that information. In addition, the SOC needs to investigate less than 1400 alerts to catch the red-team attacks in our test set, compared to more than 5500 generated by the model without CVE and tactics. Moreover, assuming a standard response time of 8 minutes per alert, this improved model would save the SOC team up to 550 person hours. That yields a model that catches red-team attacks without overwhelming the SOC with too many false positives.
Our work proposes an innovative way to bridge the gap between vulnerability data (CVEs) and security alert data originating from multiple security tools that protect servers using MITRE ATT&CK tactics. That would provide more context to the alerts which would be useful in their classification as attacks or false positives. We use DeBERTa (Decoding-enhanced BERT with Disentangled Attention), a deeplearning state-of-the-art model, to map CVE descriptions to MITRE ATT&CK tactics. Then, we map security alerts to MITRE ATT&CK tactics, which will be used as input to a context-enriched machinelearning model (by CVEs and tactics). That machine-learning model is used to classify security alerts as malicious or benign. We tested our approach using over 5.5 million security alert data combined with red-team exercise attacks and incident response labelling from the company, a large international organization with over 60,000 employees. Our CVE+tactic model (without hyperparameter tuning) detects 64% more true positives than the machine-learning model without that information. In addition, the SOC needs to investigate less than 1400 alerts to catch the red-team attacks in our test set, compared to more than 5500 generated by the model without CVE and tactics. Moreover, assuming a standard response time of 8 minutes per alert, this improved model would save the SOC team up to 550 person hours. That yields a model that catches red-team attacks without overwhelming the SOC with too many false positives.
Deep learning models are now widely deployed on edge IoT devices. However, most of these models are trained under supervised conditions and can only recognize seen classes learned from the training stage. Zero-shot learning (ZSL) is a popular method for identifying unseen classes by leveraging the semantic information from both seen and unseen classes. Foundation models (FMs) trained on web-scale data have shown impressive ZSL capability in natural language processing and visual understanding. However, leveraging FMs' generalized knowledge for zero-shot Internet of Things (IoT) sensing using signals such as mmWave, IMU, and Wi-Fi has not been fully investigated. In this work, we align the IoT data embeddings with the semantic embeddings generated by an FM's text encoder for zero-shot IoT sensing. To utilize the physics principles governing the generation of IoT sensor signals to derive more effective prompts for semantic embedding extraction, we propose to use a multi-source information fusion strategy, cross-attention, to combine a hard prompt generated by Large Language Models (LLMs) and a soft prompt consisting of learnable vectors. To address the problem of IoT embeddings biasing to seen classes due to the lack of unseen class data during training, we propose using data augmentation to synthesize unseen class IoT data for fine-tuning the IoT feature extractor and embedding projector. We evaluate our approach on multiple IoT sensing tasks. Experiment results show that our approach achieves an average improvement of 1.0% in open-set detection and 9.5% in generalized zero-shot learning compared with multiple baselines on three datasets.
...
Deep learning models are now widely deployed on edge IoT devices. However, most of these models are trained under supervised conditions and can only recognize seen classes learned from the training stage. Zero-shot learning (ZSL) is a popular method for identifying unseen classes by leveraging the semantic information from both seen and unseen classes. Foundation models (FMs) trained on web-scale data have shown impressive ZSL capability in natural language processing and visual understanding. However, leveraging FMs' generalized knowledge for zero-shot Internet of Things (IoT) sensing using signals such as mmWave, IMU, and Wi-Fi has not been fully investigated. In this work, we align the IoT data embeddings with the semantic embeddings generated by an FM's text encoder for zero-shot IoT sensing. To utilize the physics principles governing the generation of IoT sensor signals to derive more effective prompts for semantic embedding extraction, we propose to use a multi-source information fusion strategy, cross-attention, to combine a hard prompt generated by Large Language Models (LLMs) and a soft prompt consisting of learnable vectors. To address the problem of IoT embeddings biasing to seen classes due to the lack of unseen class data during training, we propose using data augmentation to synthesize unseen class IoT data for fine-tuning the IoT feature extractor and embedding projector. We evaluate our approach on multiple IoT sensing tasks. Experiment results show that our approach achieves an average improvement of 1.0% in open-set detection and 9.5% in generalized zero-shot learning compared with multiple baselines on three datasets.
Smoking remains one of the largest health concerns worldwide, which is why eHealth applications with virtual coaches have been developed to assist smokers with quitting. Providing additional feedback from human coaches during such smoking cessation programs can further improve the effectiveness of the intervention. However, due to budgetary constraints and the limited availability of human coaches, it is important to make informed decisions about when someone gets human support to optimize the effectiveness. This research investigates the use of reinforcement learning (RL) to determine when to provide human feedback in quitting smoking with a virtual coach. Using data from a longitudinal study, we implemented an RL model that decides when to involve a human coach based on users' appreciation for human support and their self-efficacy, optimizing the effort that people spend on preparatory activities and their likelihood of returning to the program. Results show that the model is effective in allocating human support, increasing users' effort and return likelihood while considering the cost of human coaches. These findings support using RL to help with determining when to provide human support in smoking cessation programs.
...
Smoking remains one of the largest health concerns worldwide, which is why eHealth applications with virtual coaches have been developed to assist smokers with quitting. Providing additional feedback from human coaches during such smoking cessation programs can further improve the effectiveness of the intervention. However, due to budgetary constraints and the limited availability of human coaches, it is important to make informed decisions about when someone gets human support to optimize the effectiveness. This research investigates the use of reinforcement learning (RL) to determine when to provide human feedback in quitting smoking with a virtual coach. Using data from a longitudinal study, we implemented an RL model that decides when to involve a human coach based on users' appreciation for human support and their self-efficacy, optimizing the effort that people spend on preparatory activities and their likelihood of returning to the program. Results show that the model is effective in allocating human support, increasing users' effort and return likelihood while considering the cost of human coaches. These findings support using RL to help with determining when to provide human support in smoking cessation programs.
Background. Quitting smoking is a challenge nowadays. Virtual coaches offer autonomous, personalized guidance for smoking cessation. However, such systems cannot replace human coaches completely. In situations, when human coaches cannot provide help to everyone - a virtual coach could follow a set of ethical principles to decide on who should get the feedback from a human.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare. ...
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare. ...
Background. Quitting smoking is a challenge nowadays. Virtual coaches offer autonomous, personalized guidance for smoking cessation. However, such systems cannot replace human coaches completely. In situations, when human coaches cannot provide help to everyone - a virtual coach could follow a set of ethical principles to decide on who should get the feedback from a human.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare.
Smoking and vaping cessation remains a significant public health challenge despite the availability of numerous aids and eHealth applications. This study explores the reasons behind users' preference for human feedback when preparing to quit smoking or vaping, aiming to address a gap in existing literature on the integration of human elements in eHealth platforms. The research involved 479 participants interacting with a virtual coach, with some receiving human feedback. We conducted a thematic analysis of two open-text questions with 265 responses each from post-questionnaires, and key themes such as emotional connection, personalized advice, effectiveness, motivation, and accountability were identified. Through using quantitative data and previously published research, these findings were further explained. The results from the quantitative analysis show that incorporating human elements in eHealth applications can enhance smoking cessation support. This research provides insights into the main reasons how the human support in eHealth applications should be designed. Key recommendations include designing human feedback to offer empathy and validation, tailoring feedback to individual needs, incorporating interactive elements to maintain engagement, providing constant encouragement, and establishing accountability mechanisms.
...
Smoking and vaping cessation remains a significant public health challenge despite the availability of numerous aids and eHealth applications. This study explores the reasons behind users' preference for human feedback when preparing to quit smoking or vaping, aiming to address a gap in existing literature on the integration of human elements in eHealth platforms. The research involved 479 participants interacting with a virtual coach, with some receiving human feedback. We conducted a thematic analysis of two open-text questions with 265 responses each from post-questionnaires, and key themes such as emotional connection, personalized advice, effectiveness, motivation, and accountability were identified. Through using quantitative data and previously published research, these findings were further explained. The results from the quantitative analysis show that incorporating human elements in eHealth applications can enhance smoking cessation support. This research provides insights into the main reasons how the human support in eHealth applications should be designed. Key recommendations include designing human feedback to offer empathy and validation, tailoring feedback to individual needs, incorporating interactive elements to maintain engagement, providing constant encouragement, and establishing accountability mechanisms.
Dysarthric speech, characterized by articulation problems and a slower speech rate, shows lower automatic speech recognition (ASR) performance compared to normal speech. To improve performance, researchers often try to enhance dysarthric speech to be more like normal speech before passing it through an ASR trained on normal speech. In this project, we compare different signal processing and voice conversion techniques for dysarthric-to-normal speech enhancement. The resulting enhanced speech is objectively evaluated using an ASR system trained on normal speech. Also, the naturalness and intelligibility of the enhanced dysarthric speech are evaluated through listening experiments. Finally, the correlation between subjective and objective evaluations was analyzed. We found that among the techniques investigated, time-stretching demonstrated superior performance in objective evaluation experiments, surpassing state-of-the-art voice conversion methods. Across all methods, improvements in naturalness and intelligibility were positively correlated with improvements in automatic speech recognition (ASR) performance. However, this correlation was significant for some methods but not for others.
...
Dysarthric speech, characterized by articulation problems and a slower speech rate, shows lower automatic speech recognition (ASR) performance compared to normal speech. To improve performance, researchers often try to enhance dysarthric speech to be more like normal speech before passing it through an ASR trained on normal speech. In this project, we compare different signal processing and voice conversion techniques for dysarthric-to-normal speech enhancement. The resulting enhanced speech is objectively evaluated using an ASR system trained on normal speech. Also, the naturalness and intelligibility of the enhanced dysarthric speech are evaluated through listening experiments. Finally, the correlation between subjective and objective evaluations was analyzed. We found that among the techniques investigated, time-stretching demonstrated superior performance in objective evaluation experiments, surpassing state-of-the-art voice conversion methods. Across all methods, improvements in naturalness and intelligibility were positively correlated with improvements in automatic speech recognition (ASR) performance. However, this correlation was significant for some methods but not for others.
Podcasts are a rapidly growing medium for information sharing, but their audio and one-way communication format presents unique challenges in addressing misinformation. This thesis explores how to empower podcast listeners to identify and respond to misinformation effectively. Study I investigates listening habits, user trust, confidence, and behavioral responses to misinformation in podcasts through a survey of diverse participants. Key findings highlight gaps in user confidence, the impact of demographic factors, and preferences for incentives to flag misinformation. Study II builds upon these insights to design, implement, and evaluate three interventions—PAUSE, ALERT, and VOLUNTARY—aimed at optimizing user engagement in flagging misinformation. A labeled podcast dataset was created to facilitate this task-based experiment. The findings offer insights into the design of user-centric misinformation detection systems. Interventions have shown potential in empowering users to identify misinformation in podcasts. Although, whether they are able to address misinformation in podcasts effectively remains uncertain and needs further exploration. This work not only addresses a significant gap in the literature but also lays the groundwork for future innovations in combating misinformation in podcasts.
...
Podcasts are a rapidly growing medium for information sharing, but their audio and one-way communication format presents unique challenges in addressing misinformation. This thesis explores how to empower podcast listeners to identify and respond to misinformation effectively. Study I investigates listening habits, user trust, confidence, and behavioral responses to misinformation in podcasts through a survey of diverse participants. Key findings highlight gaps in user confidence, the impact of demographic factors, and preferences for incentives to flag misinformation. Study II builds upon these insights to design, implement, and evaluate three interventions—PAUSE, ALERT, and VOLUNTARY—aimed at optimizing user engagement in flagging misinformation. A labeled podcast dataset was created to facilitate this task-based experiment. The findings offer insights into the design of user-centric misinformation detection systems. Interventions have shown potential in empowering users to identify misinformation in podcasts. Although, whether they are able to address misinformation in podcasts effectively remains uncertain and needs further exploration. This work not only addresses a significant gap in the literature but also lays the groundwork for future innovations in combating misinformation in podcasts.
Targeted and successful cellular therapies for disease treatment require an extensive mapping of the complex structure and dynamics of molecular mechanisms which determine the behaviour and function of cell. CELL-seq is a genome-wide screening procedure measuring specific and targeted protein quantities as phenotypic readouts and is employed by the Netherlands Cancer Institute to analyze which genes regulate the protein state of interest. This research aims to explore the current compendium of CELL-seq screens that investigate a range of phenotypes, to create a mapping of gene-gene associations that share similar phenotypic profiles and elucidate biology that is hard to uncover with more conventional screening techniques.
We perform exploratory research to investigate the ability of the screen compendium to show network structures that reflect known biological processes. We find that with stringent requirements on interactions the screen compendium shows enrichment for a wide range of biological processes and known protein-protein interactions. We further conclude that the experimental design biases network behaviour and needs to be accounted for when constructing networks. We recommended a mutual k-nearest neighbor network construction approach, which yielded networks with the most biological relevance.
We compare the CELL-seq screens using findings from the approaches to the DepMap dataset, a well-known collection of synthetic lethality CRISPR screens, and find that the behaviour of these datasets is in many ways mirrored. We conclude that this is both due to the biology they represent and the differences in the number of screens in each dataset. Finally, we compare the coverage of biological processes between the HAP1 compendium and DepMap, and show large overlap in their coverage. Nonetheless, the differences they do show leads us to bring forward two hypotheses for gene-gene interactions that score strongly uniquely in the CELL-seq networks which are biologically plausible but are not found in DepMap or curated literature, warranting future investigations.
All code pertaining to the methods and figures in this work are hosted on GitLab by the High Performance Computing Facility of the Netherlands Cancer Institute. As such the code can be viewed by supervisors, but further details could be shared upon request. ...
We perform exploratory research to investigate the ability of the screen compendium to show network structures that reflect known biological processes. We find that with stringent requirements on interactions the screen compendium shows enrichment for a wide range of biological processes and known protein-protein interactions. We further conclude that the experimental design biases network behaviour and needs to be accounted for when constructing networks. We recommended a mutual k-nearest neighbor network construction approach, which yielded networks with the most biological relevance.
We compare the CELL-seq screens using findings from the approaches to the DepMap dataset, a well-known collection of synthetic lethality CRISPR screens, and find that the behaviour of these datasets is in many ways mirrored. We conclude that this is both due to the biology they represent and the differences in the number of screens in each dataset. Finally, we compare the coverage of biological processes between the HAP1 compendium and DepMap, and show large overlap in their coverage. Nonetheless, the differences they do show leads us to bring forward two hypotheses for gene-gene interactions that score strongly uniquely in the CELL-seq networks which are biologically plausible but are not found in DepMap or curated literature, warranting future investigations.
All code pertaining to the methods and figures in this work are hosted on GitLab by the High Performance Computing Facility of the Netherlands Cancer Institute. As such the code can be viewed by supervisors, but further details could be shared upon request. ...
Targeted and successful cellular therapies for disease treatment require an extensive mapping of the complex structure and dynamics of molecular mechanisms which determine the behaviour and function of cell. CELL-seq is a genome-wide screening procedure measuring specific and targeted protein quantities as phenotypic readouts and is employed by the Netherlands Cancer Institute to analyze which genes regulate the protein state of interest. This research aims to explore the current compendium of CELL-seq screens that investigate a range of phenotypes, to create a mapping of gene-gene associations that share similar phenotypic profiles and elucidate biology that is hard to uncover with more conventional screening techniques.
We perform exploratory research to investigate the ability of the screen compendium to show network structures that reflect known biological processes. We find that with stringent requirements on interactions the screen compendium shows enrichment for a wide range of biological processes and known protein-protein interactions. We further conclude that the experimental design biases network behaviour and needs to be accounted for when constructing networks. We recommended a mutual k-nearest neighbor network construction approach, which yielded networks with the most biological relevance.
We compare the CELL-seq screens using findings from the approaches to the DepMap dataset, a well-known collection of synthetic lethality CRISPR screens, and find that the behaviour of these datasets is in many ways mirrored. We conclude that this is both due to the biology they represent and the differences in the number of screens in each dataset. Finally, we compare the coverage of biological processes between the HAP1 compendium and DepMap, and show large overlap in their coverage. Nonetheless, the differences they do show leads us to bring forward two hypotheses for gene-gene interactions that score strongly uniquely in the CELL-seq networks which are biologically plausible but are not found in DepMap or curated literature, warranting future investigations.
All code pertaining to the methods and figures in this work are hosted on GitLab by the High Performance Computing Facility of the Netherlands Cancer Institute. As such the code can be viewed by supervisors, but further details could be shared upon request.
We perform exploratory research to investigate the ability of the screen compendium to show network structures that reflect known biological processes. We find that with stringent requirements on interactions the screen compendium shows enrichment for a wide range of biological processes and known protein-protein interactions. We further conclude that the experimental design biases network behaviour and needs to be accounted for when constructing networks. We recommended a mutual k-nearest neighbor network construction approach, which yielded networks with the most biological relevance.
We compare the CELL-seq screens using findings from the approaches to the DepMap dataset, a well-known collection of synthetic lethality CRISPR screens, and find that the behaviour of these datasets is in many ways mirrored. We conclude that this is both due to the biology they represent and the differences in the number of screens in each dataset. Finally, we compare the coverage of biological processes between the HAP1 compendium and DepMap, and show large overlap in their coverage. Nonetheless, the differences they do show leads us to bring forward two hypotheses for gene-gene interactions that score strongly uniquely in the CELL-seq networks which are biologically plausible but are not found in DepMap or curated literature, warranting future investigations.
All code pertaining to the methods and figures in this work are hosted on GitLab by the High Performance Computing Facility of the Netherlands Cancer Institute. As such the code can be viewed by supervisors, but further details could be shared upon request.
A Comparative Analysis of Learning Curve Models and their Applicability in Different Scenarios
Finding datasets patterns which lead to certain parametric curve model
Learning curves display predictions of the chosen model’s performance for different training set sizes. They can help estimate the amount of data required to achieve a minimal error rate, thus aiding in reducing the cost of data collection. However, our understanding and knowledge of the various shapes of learning curves and their applicability are still insufficient. Despite the presence of a curve that demonstrates a high level of accuracy on average, this parametric model can still exhibit inadequate performance in certain scenarios. Therefore, the objective of this research is to identify specific patterns in the datasets that influence the selection of a particular parametric curve model. To accomplish this, I conduct experiments to assess the performance of different parametric learning curves including power, exponential and Morgan-Mercer-Flodin (mmf) based on the number of features, classes, outliers, and machine learning models. I find that mmf and exponential curves outperform power law for all machine learning models. All curves work best with Logistic Regression, Bernoulli Naive Bayers and Multinomial Naive Bayers models. Exponential and mmf curves provide better results than power law for a small number of classes. Mmf also outperforms power law for the majority of numbers of features and outlier percentages.
...
Learning curves display predictions of the chosen model’s performance for different training set sizes. They can help estimate the amount of data required to achieve a minimal error rate, thus aiding in reducing the cost of data collection. However, our understanding and knowledge of the various shapes of learning curves and their applicability are still insufficient. Despite the presence of a curve that demonstrates a high level of accuracy on average, this parametric model can still exhibit inadequate performance in certain scenarios. Therefore, the objective of this research is to identify specific patterns in the datasets that influence the selection of a particular parametric curve model. To accomplish this, I conduct experiments to assess the performance of different parametric learning curves including power, exponential and Morgan-Mercer-Flodin (mmf) based on the number of features, classes, outliers, and machine learning models. I find that mmf and exponential curves outperform power law for all machine learning models. All curves work best with Logistic Regression, Bernoulli Naive Bayers and Multinomial Naive Bayers models. Exponential and mmf curves provide better results than power law for a small number of classes. Mmf also outperforms power law for the majority of numbers of features and outlier percentages.
Non-Monotonicity in Empirical Learning Curves
Identifying non-monotonicity through slope approximations on discrete points
Learning curves are used to shape the performance of a Machine Learning (ML) model with respect to the size of the set used for training it. It was commonly thought that adding more training samples would increase the model's accuracy (i.e., they are monotone), but recent works show that may not always be the case. In other words, some learners on some problems show non-monotonic behaviour. To this extent, we introduce a new method to identify non-monotonicity in empirical learning curves by approximating the curve's slope through regression around the discrete points it is defined on.This paper formalises this metric and then evaluates its accuracy through different experiments. Finally, we run the proposed metric on a subset of the extensive Learning Curve Database (LCDB) by Mohr et al. to gain better insights into the problem of non-monotonicity of learning. We found that the metric can identify non-monotonicity in learning curves well (98% experimental accuracy) and does not consider small increases due to measurement error as non-monotonicity in the curve. Finally, we have identified that non-monotonicity may be a property of some classifiers, such as Linear Discriminant Analysis. Moreover, we identified that non-monotonicity is frequently observed in datasets with faster training times.
...
Learning curves are used to shape the performance of a Machine Learning (ML) model with respect to the size of the set used for training it. It was commonly thought that adding more training samples would increase the model's accuracy (i.e., they are monotone), but recent works show that may not always be the case. In other words, some learners on some problems show non-monotonic behaviour. To this extent, we introduce a new method to identify non-monotonicity in empirical learning curves by approximating the curve's slope through regression around the discrete points it is defined on.This paper formalises this metric and then evaluates its accuracy through different experiments. Finally, we run the proposed metric on a subset of the extensive Learning Curve Database (LCDB) by Mohr et al. to gain better insights into the problem of non-monotonicity of learning. We found that the metric can identify non-monotonicity in learning curves well (98% experimental accuracy) and does not consider small increases due to measurement error as non-monotonicity in the curve. Finally, we have identified that non-monotonicity may be a property of some classifiers, such as Linear Discriminant Analysis. Moreover, we identified that non-monotonicity is frequently observed in datasets with faster training times.
Empirical Investigation of Learning Curves
Assessing Convexity Characteristics
Nonconvexity in learning curves is almost always undesirable. A machine learning model with a non-convex learning curve either requires a larger quantity of data to observe progress in its accuracy or experiences an exponential decrease of accuracy at low sample sizes, with no improvement in accuracy even when more data points are added. This paper proposes a novel approach to determine the convexity of a learning curve, which relies on calculating the second derivative of the learning curve to estimate its convexity. Along the way, we have confirmed the correctness of the proposed method from multiple perspectives, such as testing it with baselines or establishing confidence intervals for the convexity of the learning curve. Lastly, we compare our method to an alternative method and highlight some of its shortcomings.
...
Nonconvexity in learning curves is almost always undesirable. A machine learning model with a non-convex learning curve either requires a larger quantity of data to observe progress in its accuracy or experiences an exponential decrease of accuracy at low sample sizes, with no improvement in accuracy even when more data points are added. This paper proposes a novel approach to determine the convexity of a learning curve, which relies on calculating the second derivative of the learning curve to estimate its convexity. Along the way, we have confirmed the correctness of the proposed method from multiple perspectives, such as testing it with baselines or establishing confidence intervals for the convexity of the learning curve. Lastly, we compare our method to an alternative method and highlight some of its shortcomings.
”How Much Data is Enough?” Learning curves for machine learning
Investigating alternatives to the Levenberg-Marquardt algorithm for learning curve extrapolation
The conducted research explores fitting algorithms for learning curves. Learning curves describe how the performance of a machine learning model changes with the size of the training input. Therefore, fitting these learning curves and extrapolating them can help determine the required data set size for any desired performance.
The paper specifically explores the Learning Curve Database (LCDB) and investigates alternative fitting algorithms to the employed Levenberg-Marquardt (LM). These algorithms are Gradient Descent and BFGS, and the paper aims to determine whether they are more suitable for fitting learning curves than LM.
The algorithms were implemented, both in their default and optimised states, and the results were compared to LM. The results measured mean-squared error (MSE), L1 Loss, individual parametric model performance, and computation time.
The findings showed that Gradient Descent is not a suitable alternative to LM; however, BFGS proved to be competitive, as it is practically identical in performance while being significantly faster than LM. The results answered the proposed aim of the paper and generated new questions that need answering.
Further exploration of the BFGS algorithm and its application on learning curve fitting is recommended. Comparisons between the MSE distribution of LM and BFGS can be further explored, as well as comparisons on new parametric models, learners, and datasets. ...
The paper specifically explores the Learning Curve Database (LCDB) and investigates alternative fitting algorithms to the employed Levenberg-Marquardt (LM). These algorithms are Gradient Descent and BFGS, and the paper aims to determine whether they are more suitable for fitting learning curves than LM.
The algorithms were implemented, both in their default and optimised states, and the results were compared to LM. The results measured mean-squared error (MSE), L1 Loss, individual parametric model performance, and computation time.
The findings showed that Gradient Descent is not a suitable alternative to LM; however, BFGS proved to be competitive, as it is practically identical in performance while being significantly faster than LM. The results answered the proposed aim of the paper and generated new questions that need answering.
Further exploration of the BFGS algorithm and its application on learning curve fitting is recommended. Comparisons between the MSE distribution of LM and BFGS can be further explored, as well as comparisons on new parametric models, learners, and datasets. ...
The conducted research explores fitting algorithms for learning curves. Learning curves describe how the performance of a machine learning model changes with the size of the training input. Therefore, fitting these learning curves and extrapolating them can help determine the required data set size for any desired performance.
The paper specifically explores the Learning Curve Database (LCDB) and investigates alternative fitting algorithms to the employed Levenberg-Marquardt (LM). These algorithms are Gradient Descent and BFGS, and the paper aims to determine whether they are more suitable for fitting learning curves than LM.
The algorithms were implemented, both in their default and optimised states, and the results were compared to LM. The results measured mean-squared error (MSE), L1 Loss, individual parametric model performance, and computation time.
The findings showed that Gradient Descent is not a suitable alternative to LM; however, BFGS proved to be competitive, as it is practically identical in performance while being significantly faster than LM. The results answered the proposed aim of the paper and generated new questions that need answering.
Further exploration of the BFGS algorithm and its application on learning curve fitting is recommended. Comparisons between the MSE distribution of LM and BFGS can be further explored, as well as comparisons on new parametric models, learners, and datasets.
The paper specifically explores the Learning Curve Database (LCDB) and investigates alternative fitting algorithms to the employed Levenberg-Marquardt (LM). These algorithms are Gradient Descent and BFGS, and the paper aims to determine whether they are more suitable for fitting learning curves than LM.
The algorithms were implemented, both in their default and optimised states, and the results were compared to LM. The results measured mean-squared error (MSE), L1 Loss, individual parametric model performance, and computation time.
The findings showed that Gradient Descent is not a suitable alternative to LM; however, BFGS proved to be competitive, as it is practically identical in performance while being significantly faster than LM. The results answered the proposed aim of the paper and generated new questions that need answering.
Further exploration of the BFGS algorithm and its application on learning curve fitting is recommended. Comparisons between the MSE distribution of LM and BFGS can be further explored, as well as comparisons on new parametric models, learners, and datasets.
Learning curves in machine learning are graphical representations that depict the relationship between a model's performance and the amount of training data it has been exposed to. They play a fundamental role in obtaining the knowledge and skills across a range of domains. Although there are already quite some researches studying machine learning curves, explaining the importance and practical application of learning curves, we still know very little about the factors that influence the parameters of the learning curve. The aim of this research is to give a better understanding of different factors affecting the parameters of the learning curve. Specifically, we are interested in how the dimensionality of a dataset can influence the parameters of the learning curve. Since learning curves are useful and have several applications, such as estimation of the time required to complete production runs, we would like to know if the dimensionality has any effect on the shapes of learning curves. To conduct the research I applied principal component analysis (PCA) three times with different amount of information preserved to reduce number of dimensions on several datasets and analysed the changes in the parameters of the obtained learning curves. The research showed that potentially there might be some relation between dimensionality and shape of the curve, but only in cases of specific machine learning model. The amount of experiments conducted is not sufficient to make solid conclusions and it is advised to continue with proposed experimental setup, but train machine learning models on increased number of datasets.
...
Learning curves in machine learning are graphical representations that depict the relationship between a model's performance and the amount of training data it has been exposed to. They play a fundamental role in obtaining the knowledge and skills across a range of domains. Although there are already quite some researches studying machine learning curves, explaining the importance and practical application of learning curves, we still know very little about the factors that influence the parameters of the learning curve. The aim of this research is to give a better understanding of different factors affecting the parameters of the learning curve. Specifically, we are interested in how the dimensionality of a dataset can influence the parameters of the learning curve. Since learning curves are useful and have several applications, such as estimation of the time required to complete production runs, we would like to know if the dimensionality has any effect on the shapes of learning curves. To conduct the research I applied principal component analysis (PCA) three times with different amount of information preserved to reduce number of dimensions on several datasets and analysed the changes in the parameters of the obtained learning curves. The research showed that potentially there might be some relation between dimensionality and shape of the curve, but only in cases of specific machine learning model. The amount of experiments conducted is not sufficient to make solid conclusions and it is advised to continue with proposed experimental setup, but train machine learning models on increased number of datasets.