Leveraging the power of high performance computing for next generation sequencing data analysis: tricks and twists from a high throughput exome workflow

PLoS One. 2015 May 5;10(5):e0126321. doi: 10.1371/journal.pone.0126321. eCollection 2015.

Abstract

Next generation sequencing (NGS) has been a great success and is now a standard method of research in the life sciences. With this technology, dozens of whole genomes or hundreds of exomes can be sequenced in rather short time, producing huge amounts of data. Complex bioinformatics analyses are required to turn these data into scientific findings. In order to run these analyses fast, automated workflows implemented on high performance computers are state of the art. While providing sufficient compute power and storage to meet the NGS data challenge, high performance computing (HPC) systems require special care when utilized for high throughput processing. This is especially true if the HPC system is shared by different users. Here, stability, robustness and maintainability are as important for automated workflows as speed and throughput. To achieve all of these aims, dedicated solutions have to be developed. In this paper, we present the tricks and twists that we utilized in the implementation of our exome data processing workflow. It may serve as a guideline for other high throughput data analysis projects using a similar infrastructure. The code implementing our solutions is provided in the supporting information files.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Computational Biology / methods*
  • Computing Methodologies*
  • Electronic Data Processing / methods
  • High-Throughput Nucleotide Sequencing / methods*
  • Humans
  • Sequence Analysis, DNA / methods*
  • Workflow

Grants and funding

PN was supported by German Cancer Aid, collaborative project “Comprehensive molecular and histopathological characterization of small cell lung cancer”, www.krebshilfe.de; Federal Ministry for Education and Research, NGFNplus/EMINet 01GS08120, www.bmbf.de; EuroEPINOMICS consortium coordinated by the European Science Foundation from the German Research Foundation, Nu50/8-1, www.euroepinomics.org; German Research Foundation, KFO-286, www.dfg.de. SM was supported by Koeln Fortune Program / Faculty of Medicine, University of Cologne, http://www.medfak.uni-koeln.de/index.php?id=195&L=0&type=atom%27A%3D0. The HPC-infrastructure was funded by Federal Ministry for Education and Research, SuGI 01IG07002A, www.bmbf.de; German Research Foundation, CHEOPS, www.dfg.de. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.