Containerization for High Performance Computing

Abstract

Containerization, a lightweight form of virtualization, increasingly became more popular in the last decade. Containers can offer a level of isolation and privacy to the user, which are not always sought after. High performance computing workloads benefit from having a custom container filesystem, but would suffer from any overhead incurred by isolation. This research assesses overhead from Singularity containers for workloads that work on different levels of the memory hierarchy. Different configurations for number of processes and threading are tested for different types of benchmarks. No significant overhead was found which encouraged to test Singularity outside the high performance computing world. Singularity was implemented in Kubernetes, a dynamic container scheduler, by restructuring and rewriting the deprecated Singularity container runtime interface. The re-newed container runtime interface was implemented in Sykube, a local Kubernetes framework for Singularity. A Kafka streaming application was used as benchmark to assess latency in messaging and deployed in a local Docker container network and a local Singularity container network. The results of the local container networks were compared against the performance of the benchmark in Kubernetes. In the local setup Singularity outperformed Docker, while in Kubernetes setup Docker outperformed Singularity.