Object Tracking has been a very active area in the field of C omputer Vision. Over the years, a variety of approaches have been put forth to solve this problem and though many of them have demonstrate considerable success none of them have been completely successful. With more methods being written each day, the evaluation of such systems becomes a very important task. If an evaluation system exists that is able to point out specific flaws in the stage of development, it can lead to a very robust and improved algorithm. This work attempts to create such an evaluation framework. Given an algorithm that detects people and simultaneously tracks them, we evaluate its output by considering the complexity of the input scene. Some videos used for the evaluation are recorded using the Kinect sensor and a benchmark dataset from the PETS workshop is also used. To analyze the performance of the tracking system,the reasons due to which the algorithm might fail are investigated and quantified over the entire video sequence. A set of features called Scene C omplexity Measures are obtained for each input frame. The variability in the algorithm performance is modeled by these complexity measures using various regression models. From the regression statistics, we show that we can compare the performance of two different algorithms and also quantify the relative influence of the scene complexity measures on a given algorithm.