Security of Real-time Big Data Analytics Pipeline

Abstract

In today’s world, real-time data or streaming data can be conceived as a continuous and changing sequence of data that continuously arrive at a system to store or process. Big Data is also one of the hottest research topics in big data computing and it requires different approaches: techniques, tools and architecture. Big data security also faces the need to effectively enforce security policies to protect sensitive data. Trying to satisfy this need, we proposed the secure big data pipeline architecture for the scalability and security. Throughout our work, we emphasize about the security of message. We use Apache Kafka and Apache Storm for real time streaming pipeline and also use sticky policies and encryption/decryption algorithm for security.