Developing and reusing bioinformatics data analysis pipelines using scientific workflow systems

Marine Djaffardjy (1) , George Marchment (1) , Clémence Sebe (1) , Raphael Blanchet (2) , Khalid Bellajhame (3) , Alban Gaignard (2) , Frédéric Lemoine (4, 5) , Sarah Cohen-Boulakia (1)

1 LISN - Laboratoire Interdisciplinaire des Sciences du Numérique
2 ITX - unité de recherche de l'institut du thorax UMR1087 UMR6291
3 LAMSADE - Laboratoire d'analyse et modélisation de systèmes pour l'aide à la décision
4 Génomique évolutive des virus à ARN - Evolutionary genomics of RNA viruses
5 Hub Bioinformatique et Biostatistique - Bioinformatics and Biostatistics HUB

Data analysis pipelines are now established as an effective means for specifying and executing bioinformatics data analysis and experiments. While scripting languages, particularly Python, R and notebooks, are popular and sufficient for developing small-scale pipelines that are often intended for a single user, it is now widely recognized that they are by no means enough to support the development of large-scale, shareable, maintainable and reusable pipelines capable of handling large volumes of data and running on high performance computing clusters. This review outlines the key requirements for building large-scale data pipelines and provides a mapping of existing solutions that fulfill them. We then highlight the benefits of using scientific workflow systems to get modular, reproducible and reusable bioinformatics data analysis pipelines. We finally discuss current workflow reuse practices based on an empirical study we performed on a large collection of workflows.

Keywords: Scientific workflows; Bioinformatics; Reuse; Reproducibility

Domains: Computer Science [cs]

Marine Djaffardjy, George Marchment, Clémence Sebe, Raphael Blanchet, Khalid Bellajhame, et al.. Developing and reusing bioinformatics data analysis pipelines using scientific workflow systems. Computational and Structural Biotechnology Journal, 2023, 21, pp.2075-2085. ⟨10.1016/j.csbj.2023.03.003⟩. ⟨hal-04037221v2⟩