System call sandboxing represents a pivotal security measure in the contemporary digital landscape, where reducing the attack surface of applications is crucial to mitigate potential cyber threats. This paper investigates the efficacy of static versus dynamic system call filtering techniques across different execution phases of selected applications, namely PWD and NGINX. Employing automated tools such as sysfilter, and chestnut, we collected comprehensive data through strace to delineate essential system calls required for each application phase. Our analysis compares these results with the policies generated by the automated tools, providing insights into the strengths and limitations of static and dynamic sandboxing methodologies. This study ultimately seeks to refine system call policies and balance robust security with necessary application functionality.

All complex programs are bound to contain software bugs, of which some could be exploited. These exploits rely on the application being able to start – or become – a process that it should not normally. To exploit these applications in this way, the attacker needs the operating system’s kernel to escalate the attacked program’s permissions or to start another process. Sandboxing is a way of stopping the attacker by limiting, which calls a program can make to the kernel – if a program never does start new processes, the sandbox application intercepts all requests (system calls) to do so. In this paper, I develop a script to collect a list of a program’s used system calls (shortened to syscalls) obtained dynamically by an external tool, and compare its output to existing – static – solutions, which extract a program’s system calls by examining its machine code from the binary. Three out of four tested programs functioned correctly after applying the filter, and the fourth one’s failure may be caused by the program I used to perform the sandboxing – firejail. All four resulting lists were almost half the size of the next best performing tested application. Further research should be done into why one of the applications failed in the same scenario that was used to record the syscalls, and how each syscall’s arguments can be filtered to further decrease the possible attack surface.

System calls are a primary way in which applications to communicate with the kernel. This is to allow them to perform sensitive tasks, however, an application will typically not require all of the system calls available to function properly. Despite this, the Linux kernel allows a program to perform any system call it wishes. This is bad for security, as it allows an attacker full access to the kernel after gaining code execution in a vulnerable program. By extracting a minimal set of system calls for a given program, we can sandbox it and only allow those system calls to be executed, greatly reducing the attack surface. In this paper, we analyze existing solutions that address system call set extraction. In particular, we will focus on applying these to configurable binaries. That is, binaries which can be compiled with a variety of different settings. For this paper, we have chosen to analyze cat as a minimal example, and busybox as the configurable application. We compile busybox in the following configurations, among variations: the default configuration, a configuration containing a minimal set of features and a configuration containing a maximal set of features. We analyze the performance of the tools Binalyzer, Sysfilter and Confine on these binaries. We see that Confine has significantly worse performance than both Binalyzer and Sysfilter. We also see that Sysfilter has better performance than Binalyzer when the complexity of the busybox binary is increased. We conclude that Sysfilter outperforms Binalyzer on binaries without debug symbols, while the opposite is true when performing analysis on binaries with debug symbols.

Small embedded devices are becoming more prevalent in the world with each passing year to improve our quality of life. However, as more devices are created, an increasing number of older devices are declared obsolete despite still being used. This results in an increasing amount of devices being vulnerable to exploitation due to a lack of security updates. Identifying these vulnerabilities manually without any system knowledge is an arduous task, and current state-of-the-art technologies do not perform generalized analysis in RTOS-based firmware. In this work, we present PinDown, an analysis framework that enables the automated identification of application code in RTOS-based firmware without requiring partial system knowledge. By identifying functions that modify the heap, we can identify RTOS components that can be leveraged to locate memory regions that host application code.