Online searching for healthcare information has gradually become a widely used internet case. Suppose a patient suffers the symptom but is unsure of the action he needs to take, a self-diagnosis tool can help the patient identify the possible conditions and whether this patient needs to seek immediate medical help. However, the accuracy and quality of the service provided by those self-diagnosis tools are still disappointing and need further improvement. This thesis focuses on an automatic differential diagnosis task with a comprehensive evaluation of reinforcement learning methods. Also, we present a systematic method to simulate medically correct patients records, which integrates a standard symptom modeling approach called NLICE. In this way, we can bridge the gap between limited available patients records and data-driven healthcare methodologies. This project investigates both flat-RL methods and hierarchical RL in an automatic differential diagnosis setting and evaluates the performance of those two kinds of methods on simulated patients records. More specifically, the action space for the differential diagnosis task is inevitably large, so the flat-RL performs relatively poorly in complicated scenarios. The hierarchical RL method can split a complex diagnosis task into smaller tasks: it contains two-level of policy learning, and each low-layer policy imitates one medical specialty. Therefore hierarchical RL method increases the Top 1 success rate from 23.1\% in flat-RL method to 45.4\%.

Besides the advanced policy learning strategy, this thesis explores the ability of NLICE symptom modeling in distinguishing conditions that share similar symptoms. The experimental results experience increases in flat-RL and hierarchical RL models and finally achieve 36.2\% and 71.8\% Top 1 success rates, respectively. To further solve the sparse action space problem in the automatic diagnosis domain, the reward shaping algorithm is implemented in the reward configuration part. The average gained reward of hierarchical RL increases from -3.65 to 0.87. Additionally, we model the general demographic background of patients and utilize contextual information to perform the policy transformation strategy, which eliminates the miss classification problem in highly sex-age related diseases.

Primary health care facilities are usually the first point of call for patients seeking medical help. However, mis-diagnosis at this stage of the clinical encounter is still quite prevalent. Mis-diagnosis can be potentially harmful to the patient and even when not the case, there is an increased financial cost of arriving at the correct diagnosis borne by the patient and an increased pressure on the capacity of the medical system. The focus of this thesis is an evaluation of machine learning models which can make a differential diagnosis of possible patient conditions from presented symptoms. In this project, a systematic approach to the acquisition and generation of data relevant to the task is presented. This approach sidesteps one of the major barriers to the application of artificial intelligence methods in the health care domain i.e. access to data. With a generated dataset of approximately 5 million records, containing 801 conditions and 376 symptoms, three machine learning models - Naive Bayes, Random Forest and Multilayer perceptron (MLP) - are evaluated and compared on the generated data using the accuracy, precision and Top-5 accuracy as evaluation metrics. The Naive Bayes model achieves a 58.8% accuracy score, 63.3% precision score and an 85.3% top-5 accuracy score. The Random Forest achieves 57.1% accuracy with a precision score of 61.2% and a top-5 accuracy of 84.5%. The MLP model achieves similar performance with Naive Bayes with an accuracy of 58.8%, a precision of 63.0% and a top-5 accuracy of 85.5%. The number of symptoms expressed per condition was shown to have a strong effect on the achieved metric scores. When evaluated on a generated dataset with at least 5 symptoms per condition, the accuracy score lay between 80.2% and 83.6%, the precision was within the range of 84.2% and 87.6% and the top-5 accuracy was between 95.7% and 96.6% across all evaluated models. For a better understanding of the potential efficacy of these models in a real world setting, a number of possible real world scenarios are proposed and new datasets are generated based on these scenarios. The trained models are then evaluated on these new datasets. It is shown that model performance is closely related to the relevance and number of observed symptoms for each condition - a higher number of symptoms expressed per condition results in higher performance by the models. It is also shown that model performance degrades considerably when the new datasets are very different from the original generated data. The models perform poorly especially in the case when symptoms not usually associated with a condition are presented even when the presentation probability is still low.