ECHO (Orchestration platform for Hybrid dataflows across Cloud and Edge) is a platform that operates across diverse distributed resources to enable execution of complex analytical dataflows. It can operate on diverse data models - streams, micro-batches and files, and interface with native runtime engines like TensorFlow and Sorm to execute them. It manages the applcation's lifecycle, and can schedule the dataflow on Edge, Fog and Cloud resources while enabling dynamic task migration between the resources.
Publication: ECHO: An Adaptive Orchestration Platform for Hybrid Dataflows across Cloud and Edge, Pushkara Ravindra, Aakash Khochare, Siva Prakash Reddy, Sarthak Sharma, Prateeksha Varshney, Yogesh Simmhan, 15th International Conference on Service-Oriented Computing (ICSOC) 2017, Malaga, Spain, https://doi.org/10.1007/978-3-319-69035-3_28 and https://arxiv.org/abs/1707.00889
There are multiple example dataflows that are present in the input-dags folder.
The Framework consists of mainly two modules
-
ECHO Master Service - This is the central command module that manages all the resources and the applications that make up the framework. It is the point of contact for any user wishing to deploy a new Application onto the platform, or to register a new edge device into the platform
-
ECHO Platform Service - This is the module that runs on every individual edge device. It maintains a link with the Master Service and executes the control signals received from the master.
The Resource Directory is present in the Registry module. The target folder will contain HyperCatServer.war which should be placed in the ROOT directory of a tomcat server. The Resource Directory can then be accessed as <ip_of_tomcat_server>:8080/cat
The platform uses MQTT messages to transmit control signals between the Master Service and the Platform Service. ECHO uses mosquitto MQTT service, which must be installed on the server running the Master Service. The service can be installed via apt using sudo apt-get install mosquitto
Submit Application to the master by sending a POST request to the <ip_of_cloud_master>:8099/DAG with the DAG representing the application as a request body.
Please refer the paper for detailed information about the platform - https://arxiv.org/abs/1707.00889