The wide-spread popularity of online social networks and the resulting availability of data to researchers has enabled the investigation of new research questions, such as the analysis of information diffusion and how individuals are influencing opinion formation in groups. Many of these new questions however require an automatic assessment of the sentiment of user statements, a challenging task further aggravated by the unique communication style used in online social networks. This paper compares the sentiment classification performance of current analyzers against a human-tagged reference corpus, identifies the major challenges for sentiment classification in online social applications and describes a novel hybrid system that achieves higher accuracy in this type of environment.