A Research Paper On Distributed Storage And Processing Of Big Data Using The Hadoop Ecosystem

The exponential growth of data generated by web applications, sensors, and digital transactions has created an urgent need for scalable frameworks capable of storing and processing massive datasets efficiently. Apache Hadoop, an open-source framework based on the MapReduce programming model, has emerged as a widely adopted solution for distributed big data processing. This paper presents a case study on the application of Hadoop\\\'s Distributed File System (HDFS) and MapReduce programming model to analyze large-scale web server log data. A dataset of approximately 2 GB, comprising 5 million log entries, was processed on a 4-node Hadoop cluster to extract insights such as request frequency, error rate, and peak traffic hours. The study compares the performance of MapReduce-based processing against traditional single-node sequential processing, demonstrating significant improvements in processing time and scalability as data volume increases. The results confirm that Hadoop\\\'s distributed architecture provides an efficient, fault-tolerant, and cost-effective solution for big data analytics, particularly suited to batch processing of unstructured and semi-structured data.