OPTISAN: Using Multiple Spatial Error Defenses to Optimize Stack Memory Protection within a Budget

Spatial memory errors continue to be the cause of many vulnerabilities. While researchers have proposed several defenses to prevent exploitation of spatial memory errors, systems currently rely on defenses that only protect a small fraction of stack data (e.g., return addresses) and leave a window of vulnerability (e.g., by only enforcing on function returns). One proposal to address this problem is to place defenses at the lowest cost locations until a cost budget was met, but this approach only considers a single defense and does not account for the security implications of possible placements. In this paper, we propose the OptiSan system, which is the first system to apply multiple spatial memory defenses to maximize the number of objects protected from spatial memory errors within a cost budget. OptiSan analyzes each program to identify the stack objects that may be exploited by spatial memory errors, called usable targets, and estimates the overhead for individual defense operations, for both metadata management and spatial checks, to enable flexibility in placement choices. OptiSan applies this information in a novel Mixed-Integer Non-Linear Programming formulation to generate an optimal placement. We apply OptiSan to generate placements using a combination of identity-based (i.e., influential BaggyBounds) and location-based (i.e., widely used AddressSanitizer (ASan)) spatial memory defenses, finding that OptiSan utilizes the more effective Baggy Bounds defense broadly, augmenting it with ASan to increase the number of memory operations with usable targets protected by 18.4% on average across a set of benchmark and server programs. OptiSan shows that using multiple spatial memory defenses provides valuable flexibility to prevent the exploitation of many spatial memory errors within a cost budget.