Integrating HPC, AI, and workflows for scientific data analysis

2024

Article

Badia R.M., Berti-Equille Laure, Ferreira da Silva R., Leser U.

Dagstuhl Reports, 2024, 13 (8), 129-164 ISSN 2192-5283

Dagstuhl Seminar, 23352, Leibniz (AUT), 2023/08/27-2023/09/01

The Dagstuhl Seminar 23352, titled 'Integrating HPC, AI, and Workflows for Scientific Data Analysis,' held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.

Intelligence artificielle [122INTAR]

Fonds IRD [F B010092358]

fdi:010092358