Application Aware Software Defined Flows of Workflow Ensembles

Computational science depends on complex, data intensive applications operating on datasets from a variety of scientific instruments. A major challenge is the integration of data into the scientist’s workflow. Recent advances in dynamic, networked cloud resources provide the building blocks to construct reconfiguration, end-to-end infrastructure that can increase scientific productivity, but applications are not taking advantage of them. In our previous work, we introduced Dy-N amo, that enabled CASA scientists to improve the efficiency of their operations and effortlessly leverage capabilities of the cloud resources available to them that previously remained underutilized. However, the provided workflow automation did not satisfy all the operational requirements of CASA. Custom scripts were still in production to manage workflow triggering, while multiple layer2 connections would have to be allocated to maintain network QoS requirements. In this work, we enhance the DyNamo system with ensemble workflow management capabilities, end-to-end infrastructure monitoring, as well as more advanced network manipulation mechanisms. To accommodate CASA’s operational needs we also extended the newly integrated Pegasus Ensemble Manager with file and time based triggering functionality, that improves managing workflow ensembles. Additionally, Virtual Software Defined Exchange (vSDX) capabilities have been extended, enabling link adaptation, flow prioritization and traffic control between endpoints. We evaluate the effects of the DyNamo’s vSDX policies by using two CASA workflow ensembles competing for network resources, and we show that traffic shaping of the ensembles can lead to a fairer use of the network links.