Chatbots are the text based conversational agents with which users interact in natural language. They are becoming more and more popular with the immense growth in messaging apps and tools to develop text based conversational agents. Despite of advances in Artificial Intelligence and Natural Language Processing, chatbots still struggle in accurately understanding user requests, thus providing wrong answers or no response. An effective solution to tackle this problem is involving human's capabilities in chatbot’s operations for understanding user requests. There are many existing systems using humans in chatbots but they are not capable to scale up with the increasing number of users. To address this problem, we provide insights in how to design such chatbot system having humans in the loop and how to involve humans efficiently. We perform an extensive literature survey about chatbots, and human computation applied for a chatbot, to guide the design of our reference chatbot system. Then we address the problem of cold starting chatbot systems. We propose a methodology to generate high quality training data, with which, chatbot’s Natural Language Understanding (NLU) model can be trained, making a chatbot capable of handling user requests efficiently at run time. Finally we provide a methodology to estimate the reliability of black box NLU models based on the confidence threshold of their prediction functionality. We study and discuss the effect of parameters such as training data set size, type of intents on automatic NLU model.