pipeoop is a real-time and batch data processing pipeline for an industrial spray painting gun machine. Design Specs - WIP
- Build our services.
$ docker-compose build
- Run our services (you can run in in detached mode with
-d).
$ docker-compose up
Read the documentation here.
Note: It might take a while to set up our
HDFSservice and requires at least 8gb RAM.
- Optional: You can connect to our
HDFSservice by findingclouderacontainer's id:
docker exec -it $(docker container ls | grep cloudera | cut -d " " -f1) bash
Read the documentation here.
- Optional: You can go to http://localhost:8888/ to access the
HDFSserver with Hue.
username: cloudera
password: cloudera
5.1. To stop our Docker containers:
$ docker-compose down
5.2 To stop our Docker containers and delete their data (databases, cache, etc):
$ docker-compose down -v --rmi
Read the documentation here.
To run our project, you'll need Docker and Docker Compose.
Fork repository, make changes, send us a pull request. We will review your changes and apply them to the master branch shortly, provided they don't violate our quality standards. To avoid frustration, before sending us your pull request, please run all services provided in docker-compose.yml file, and make sure everything runs according to our design specs:
$ docker-compose up
Note: Each directory contains data configurations or programs used in our pipeline, if you need to create a new service, please attach all the required files in its directory.
- Alexandru Stoica - Initial work - Master