Community question answering (CQA) platforms provide a social environment for users to share knowledge online. Users can submit complex and subjective questions on CQA platforms and then derive the desired answer from other community users. A large number of user-generated data has been produced by various CQA sites (e.g., Quora, StackExchange) and been used in different CQA researches. Question retrieval task is one of the popular CQA tasks aiming at solving the overloading issue of CQA platforms and increasing user satisfaction by reducing their waiting time. A question retrieval system is expected to automatically retrieve similar questions from the CQA archives regarding a new question, and the answers to similar questions are returned to users directly. Different information retrieval (IR) approaches have been proposed for question retrieval task ranging from the conventional retrieval models to the learning to rank models and neural ranking models. However, the IR community is now facing the issue of overusing the weak baselines. Thus, it is hard for researchers to identify the reported improvement of the newly-proposed methods, which greatly impedes the development of the community. Some researchers have already proposed several competitive baselines for ad-hoc retrieval task, but currently, the proposals of strong baselines for question retrieval are still not enough. Thus, this work targets on identifying the suitable baselines for question retrieval task on different datasets. We conduct an empirical comparison among different retrieval models on two representative datasets and analyze the performance of models on different question sets. Analyzing on CQA questions is challenging since the CQA questions are more diverse and complex, compared to the questions on traditional question answering (QA) (e.g., Wikipedia) system as well as the queries on traditional search engine (e.g., Google). Our work investigates the impact of the question from two perspectives. We first display how retrieval performance changes on various question sets (e.g., questions with different lengths and different levels of specificity) and then explain the reasons for the performance changes. Moreover, we conduct an error analysis to reveal the hard types of questions for different retrieval models on two datasets. In order to overcome the existing weakness of the retrieval models, we further select two techniques that have already proven effective in other retrieval tasks. We hypothesize that the two techniques can also be useful on question retrieval task. We then implement the two techniques on our datasets to validate the hypothesis. Our findings show that one of them can not help to enhance the retrieval effectiveness of models due to the different characteristics of task design while another technique successfully demonstrates the additive effectiveness gains. Based on our findings, we find out the suitable baseline models on different datasets as well as emphasize their relative strength and limitation. We believe our work can provide useful guidance on how to select an appropriate baseline for future works on question retrieval.