A problem with machine learning (ML) techniques for detecting intrusions in the Internet of Things (IoT) is that they are ineffective in the detection of low-frequency intrusions. In addition, as ML models are trained using specific attack categories, they cannot recognize unknown attacks. This article integrates strategies of cost-sensitive learning and multitask learning into a hybrid ML model to address these two challenges. The hybrid model consists of an autoencoder for feature extraction and a support vector machine (SVM) for detecting intrusions. In the cost-sensitive learning phase for the class imbalance problem, the hinge loss layer is enhanced to make a classifier strong against low-distributed intrusions. Moreover, to detect unknown attacks, we formulate the SVM as a multitask problem. Experiments on the UNSW-NB15 and BoT-IoT datasets demonstrate the superiority of our model in terms of recall, precision, and F1-score averagely 92.2%, 96.2%, and 94.3%, respectively, over other approaches.