Nerdalize B.V. is an infrastructure-as-a-service (IaaS) cloud provider aiming to offer substantially lower prices than its competitors. In order to visualize its cost savings to customers and measure its own systems against competitors in a cloud market reigned by opaque pricing models, it would like to utilize an application benchmarker to give customers insight into the resource utilization and operation costs of their applications among various cloud providers. We have done research into the fields of cloud computing, benchmarking and the intersection thereof, determined requirements for such a benchmarker, and assessed any existing solutions in the field. We then chose Nerdalize’s internal prototype implementation as a suitable base to develop a fully featured, production-ready application benchmarker. We identified five main design goals of correctness, robustness, security, extensibility and maintainability. We then analyzed and prioritized potential improvements and extensions to this prototype, and implemented them in an agile-driven Extreme Programming (XP) development process. The main contributions lie in designing and implementing a fully automated test suite and system, vastly improving the accuracy and stability of the benchmarker, re-designing the deployment model from monolithic to modularized and extensible, implementing support for provisioning to arbitrary Linux-based hosts, and deployment of complex workload architectures. We then experimentally verify the accuracy of the benchmarker, and assess that its deployment overhead is very small to negligible, and run several tests against real-world cloud providers. The end result of this project is a stable, well-tested, featured benchmarker application that is used in production environments at Nerdalize.