Published September 7, 2026 | Version v2.0.0

Hadoop Infrastructure for Big Data using Docker Compose

Authors/Creators

  • 1. Universidad de Konstanz

Description

Hadoop Big Data Infrastructure v2.0.0

Version 2.0.0 introduces a major update to the Docker Compose infrastructure used for Big Data learning, experimentation and technical demonstrations.

Main improvements

  • Persistent storage for HDFS NameNode and DataNode.
  • Persistent Hive Metastore using PostgreSQL.
  • Persistent storage for Zeppelin notebooks.
  • Persistent storage for Kafka and ZooKeeper.
  • Persistent MySQL storage used by Hue.
  • Persistent storage for StreamSets.
  • Persistent repositories and state for Apache NiFi.
  • Apache NiFi updated and fixed to version 2.11.0.
  • NiFi configured in standalone mode with HTTPS access.
  • StreamSets startup issue fixed by configuring the required nofile limit.
  • Docker services configured with restart: unless-stopped.
  • Improved separation between container lifecycle and persistent data.
  • Updated docker-compose.yml, hadoop-hive.env, README.md and CITATION.cff.

Persistence

This release introduces explicit Docker volumes for services that maintain state.

Notes

If you use this repository, please cite it as follows.

Files

juliopez/Hadoop-v2.0.0.zip

Files (60.2 MB)

Name Size
md5:80f6e99cdfdd11d00efc4a89f14ed2da
60.2 MB Preview Download

Additional details

Related works

Is supplement to
Software: https://github.com/juliopez/Hadoop/tree/v2.0.0 (URL)

Software