Design of a Cloud Approach for Dataset integration (Task 1):\u00a0<\/strong> Next-generation scientific discoveries Data analysis for anomaly detection during workflow execution (Task 2):<\/strong> We plan to expand Dynamic workflow execution on HPC platforms (Task 3):<\/strong> Today, GinFlow is not designed to be Distributed Infrastructure Support for Workflow and Data Management Design of a Cloud Approach for Dataset integration (Task 1):\u00a0 Next-generation scientific discoveries are at the boundaries of datasets, e.g., across multiple science disciplines, institutions and spatial and temporal scales. Today, data integration processes and methods are largely ad-hoc or manual. A generalized resource infrastructure that integrates … <\/p>\n
\nare at the boundaries of datasets, e.g., across multiple science disciplines, institutions and spatial
\nand temporal scales. Today, data integration processes and methods are largely ad-hoc or manual.
\nA generalized resource infrastructure that integrates knowledge of the data and the processing tasks
\nbeing performed by the user in the context of the data and resource lifecycle is needed.
\nClouds provide an important infrastructure platform that can be leveraged by including knowledge
\nfor distributed data integration and that will be the focus of this research area. In 2017, we will work
\non the cloud system design leveraging the work done at LBNL on the design of the E-HPC system. We
\nwill further work on real-time data processing platforms targeting elasticity in the context of clouds to
\ndynamically adjust the resource allocation to the needs.<\/p>\n
\nthe work done in 2016 to detect anomalies during the execution of scientific workflows. We will use several
\nscientific computing workflows as exemplars, study the places where integrity failures could occur, and
\nexamine places at any point during the workflow, including the scientific instruments, networks, and HPC
\nsystems, where provenance data could tell us more about the existence or cause of the failure. Where
\nprovenance data can be captured, we will develop systems to capture and analyze that data to build
\na proof of concept. Where provenance data cannot currently be captured, we will recommend ways in
\nwhich hardware or software designs could be altered in future implementations to capture such data. If
\ntime allows, we will prove the necessity and\/or sufficiency of such data to demonstrate completeness of
\nthe approach. Amir Teshome Wonjiga\u2019s internship will\u00a0 be related to this task.<\/p>\n
\ndeployed over an HPC environment. In particular, there is no proper scheduling strategies to optimise the
\nmapping of the tasks over compute nodes. One direction we will explore is to make GinFlow HPC-ready.
\nIn particular, we plan to devise strategies to dynamically assign CPU power to tasks as the execution
\nmoves forward in the graph of tasks. This would allow on one side to make GinFlow more HPC ready,
\nand in particular deployable over the NERSC computing platform, and on<\/p>\n","protected":false},"excerpt":{"rendered":"