Since the stone age the human race seeks for strategies to extend its viewing range. With the rise of technology in the twentieth century, cameras are found to be a very useful tool to survey a large area with limited resources. With an increasing numbers of cameras, it becomes more difficult to watch every monitor and prevent incidents in the surveillance area. For the last decades, research seeks for possibilities to automatize the process of video surveillance. For this thesis, we approach the surveillance task from the human perspective: we try to emulate what human operators do when they watch the monitors. To perform this task, state-of-the-art techniques from Computer Vision and Artificial Intelligence are applied. An object tracking technique called P-N Learning is used that enables the tracker to learn from its mistakes. The Java Agent Development Framework (JADE) is used to enable communication between agents in the FIPA Agent Communication Language standard. A surveillance system model is designed that detects suspicious behavior in a non-public area. Its task is to alert the operators about suspicious events to give them the chance to investigate and take action. Two prototype applications are implemented and experiments are conducted to show the performance. We showed the proof-of-concept of a system which is able to emulate operators and can potentially outperform a human being. Once the system knows what is considered suspicious behavior it can be automatically detected.