A Reference Architecture for Securing Big Data Infrastructures

Enterprise information repositories contain sensitive, and in some cases personally identifiable information that allows organizations to understand lifetime value of their customers, enable journey mapping to provide better and more targeted products and services, and improve their own internal operations by reducing cost of operations and driving profitability. Analysis of such large quantities of disparate data is possible today leveraging Hadoop and other Big data technologies. These large data sets create a very attractive target for hackers and are worth a lot of money in the cyber black market. Recent data breaches such as the ones at Anthem, Sony and others, drive the need to secure these infrastructures consistently and effectively.

In the form of a case study, we introduce the security requirements for such ecosystems, elaborate on the inherent security gaps and challenges in such environments and go on to propose a reference architecture for securing big data (Hadoop) infrastructures in financial institutions. The high-level requirements that drive the implementation of secure Hadoop implementations for the enterprise, including privacy issues, authentication, authorization, marking, key management, multi-tenancy, monitoring and current technologies available to the end user will be explored in detail. Furthermore, the case study will elaborate on how these requirements, enabled by the reference architecture are applied to actual systems implementations.