Integrating SIEM with Big Data – Part I


Security Information and Event Management (SIEM) solutions are used by organizations to collect and analyze data to support use cases such as network security event monitoring, advanced/unknown threat detection, incident investigations/forensics, user activity monitoring and compliance reporting. SIEMS collect event and flow data generated by security devices, network infrastructures, systems and applications. This data is normalized and combined with contextual information about users, assets, threats and vulnerabilities. The normalized and aggregated data is then processed against a set of vendor supplied or custom developed correlation algorithms and detection rules for analysis to support the specific use cases. To support this, most SIEMs provide the following capabilities, as shown in the figure:

  • Collection and storage of security and management data from multiple sources
  • Data aggregation and correlation
  • Data enrichment and contextualization
  • Event detection and alerting
  • Audit data management
  • Compliance reporting
  • Security reporting and visualization

With the growing number and sophistication of cyber threats, more data has to be collected from a larger variety of sources, processed at higher speeds and analyzed over longer periods of time to be able to detect, prevent, and/or respond to these threats. This presents significant performance and scalability challenges for the existing SIEM solutions that organizations already have in place. Organizations have invested significant time and money to deploy and configure their SIEM solutions, especially in configuring the rules-based algorithms that detect anomalous and malicious events on their networks. Thus, it does not make sense to simply rip and replace them with new solutions. Big Data technologies and Big Data analytics can be integrated with SIEM solutions to offload the data and processing intensive capabilities such as collection and storage, data aggregation and correlation, and enrichment and contextualization. By integrating their SIEM solutions with Big Data Technologies and Analytics, organizations can preserve their existing investments while enhancing and scaling those solutions to combat the challenges presented by the evolving cyber threats. Stay tuned for Part II where we’ll describe the core components of such a solution.