BIG Data Hadoop Spark Application Simple Architecture
Pictorial Presentation diagram of BIG Data Hadoop Spark Application Architecture
Source Data: It contains Unstructured, Semi-structured and Structured data. Example e-commerce, trading, click stream, Social Media and Realtime stream data. ETL / ELT (Extract Transform Load (or) Extract Load Transform): Depending on the nature of the data such as volume, latency and complexity – the ETL / ELT technology stack can vary. Spark / Scala / Java / Python: Spark is a frame. We can implement Scala, Java and Python code on Spark frame work to implement most complex transformations. Sqoop: It is like import / export utility between RDBMS and HDFS. Kafka: It is for real time streaming of the data. MapReduce: It is slowly getting outdated. Earlier implementations of Hadoop ecosystem used MapReduce for ETL / ELT work. Spark framework is 10x faster than Hadoop MapReduce frame work. On a lighter note some companies are not moving away from MapReduce because of their employees who transformed from Java developer to Hadoop developer. They still love to work in MapReduce and Pig environment :) HDFS Data Layers: The data in HDFS file system can organize in to the following four layers. Raw Data: It is the combination of Structures, Unstructured and Semi-structured data. It represents the same source data copy without any modifications. Work Data: Prepared and processed data through ETL / ELT process. May contain some erroneous data, incomplete transformations. But, it is all structured data. ...Click here for full details