Module Requirements#

Overview#

Below is a table detailing minimum resource requirements for each module in the duet and make_reference pipeline, relative to each profile. Your cloud tenancy or HPC cluster should be able to allocate a set of VMs/nodes with these resources.

The module specific settings and profiles can be found in the ./pipelines/duet/1.7.0/modules and ./pipelines/duet/1.7.0/conf directories under your current instance directory. We do not recommend editing these files directly as any local change will not be reflected when you upgrade the pipeline.

You can edit resource requirements by referencing specific modules in the nextflow_override.config file in the biomodal script folder to accommodate available hardware resources. Please see examples in section Set queue, CPU, RAM and DISK per module for more information on how to increase or decrease the resource requirements. However, the duet pipeline is less likely to run successfully if the minimum resources are too restrictive.

Important

The module requirement is per sample per node on a HPC cluster node or cloud VM. You need to multiply the number of samples by the resource requirement if you intend to only use a single node or VM.

If you are running the duet or make_reference pipeline on a larger local node you can set a maximum ceiling for CPU and RAM per module. Please see section Limit local resources in the documentation for information of how to use parameters to limit the resource requirements for the local and local_deep_seq profiles.

Module requirements#

Module / Requirement type

CPU (cores)

RAM (GB)

Disk (GB)

CPU (cores)

RAM (GB)

Disk (GB)

CPU (cores)

RAM (GB)

Disk (GB)

Profile name:

default

default

default

deep_seq

deep_seq

deep_seq

super_seq

super_seq

super_seq

ASM

16

32

-

-

-

-

32

128

1600

BAMLET

8

8

-

16

16

800

16

16

1600

BAM_TO_CRAM

6

6

-

-

-

-

-

-

-

BWA_MEM2 (Human genome)

16

64

-

32

64

800

64

128

3200

BWA_MEM2 (Mouse genome)

16

64

-

32

128

800

64

256

3200

DEEPTOOLS_PLOTFINGERPRINT

4

16

-

-

-

-

8

32

1600

DEEPVARIANT_CALLER

16

64

-

32

64

-

32

64

-

DQSPY

8

32

-

-

-

-

-

-

-

EPIQUANT_BAM2ZARR (CG)

8

8

-

16

16

-

32

32

-

EPIQUANT_BAM2ZARR (non-CG)

16

32

-

16

16

-

32

32

-

EPIQUANT_FRAGMENTOMICS

16

16

-

-

-

-

-

-

-

EPIQUANT_MBIAS or GENOME_MBIAS

4

4

-

16

16

-

32

32

-

FASTQC

2

4

-

-

-

-

-

-

-

FLEXBAR_PRE_RESOLUTION

2

2

-

-

-

-

-

-

-

FRAGMENT_LENGTH_DEDUP

8

8

-

16

16

2000

16

32

3200

GENETIC_ACCURACY

8

16

-

-

-

-

-

-

-

HAPLOTYPE_CALLER

16

64

-

32

64

-

32

128

1600

JOIN_FRAGMENTOMICS_ZARR_STORES

4

16 (8 for 5bp and mosaic_5bp modes)

-

-

-

-

-

-

-

JOIN_ZARR_STORES

16

32

-

-

-

-

-

-

-

JOIN_ZARR_STORES (GC)

8

16

-

-

-

-

-

-

-

LONG_CTRL_GENETIC_ACCURACY

1

1

-

-

-

-

-

-

-

MAKE_NON_POSITIONAL_QTABLE

1

1

-

-

-

-

-

-

-

MERGE_LANES_FRAGMENT_LENGTH_DEDUP

8

8

-

16

16

2000

16

32

3200

MERGE_LANES_ULTIMA_DEDUP

8

8

-

32

128

1600

32

128

3200

MERGE_LANES_UMI_DEDUP

8

8

-

16

16

2000

16

32

3200

MERGE_UMI_STATS

2

2

-

-

-

-

-

-

-

MODALITY_EXPORT

8

16 (CG) / 32 (other)

-

-

-

-

-

-

-

MODALITY_FRAGMENTOMICS_EXPORT

4

16

-

-

-

-

-

-

-

MULTIQC

2

8

-

-

-

-

-

-

-

MUTECT2

16

64

-

32

64

-

64

256

1600

PICARD_GCBIAS

2

8

-

-

-

-

-

-

-

PICARD_TARGETED_METRICS

2

8

-

8

32

-

8

32

-

PICARD_TARGETED_METRICS (twist methylome)

8

32

-

8

32

-

8

32

-

PIPELINE_REPORT

1

1

-

-

-

-

-

-

-

PRELUDE

32

32

-

64

64

-

96

96

1600

PRESEQ

4

8

-

-

-

800

-

-

1600

PUBLISH_CRAMS

-

-

-

-

-

800

-

-

1600

QUALIMAP_BAMQC

4

16

-

-

-

-

-

-

-

SAMTOOLS_MERGE_INTERVALS

16

16

200

-

-

-

16

16

3200

SAMTOOLS_MERGE_LANES (Human genome)

8

8

-

8

16

800

8

32

3200

SAMTOOLS_MERGE_LANES (Mouse genome)

8

8

-

16

64

800

32

96

3200

SEQTK_SAMPLE

8

8

-

32

64

-

32

64

1600

SPLIT_CRAM_BY_READS

8

64

500

-

-

2000

-

-

6000

SPLIT_CRAM_BY_SHARDS

2

4

500

-

-

2000

-

-

6000

SPLIT_FASTQS_PAIRED

8

16

1000

-

-

2000

-

-

3000

SPLIT_FASTQS_SINGLE

8

16

1000

-

-

2000

-

-

3000

SPLIT_INTERVALS

1

4

-

-

-

-

-

-

-

TSS_BIAS

2

8

-

-

-

-

-

-

1600

ULTIMA_DEDUP

8

8

-

32

128

1600

32

128

3200

UMI_DEDUP

8

8

-

8

32

800

16

32

3200

Note

A dash in a profile column means that the profile does not override the module default for that resource.

Reference pipeline module requirements#

Module / Requirement type

CPU (cores)

RAM (GB)

COMBINE_GENOME_AND_CTRLS

1

2

MAKE_BWAMEM2_INDEX

2

128

MAKE_BWAMEM2_INDEX (>=4.6Gbp)

2

320

MAKE_FASTA_INDEX_GENOME

1

1

MAKE_FASTA_INDEX_SHORT_CTRLS

1

1

MAKE_FASTA_INDEX_LONG_CTRLS

1

1

MAKE_BED_FILES_GENOME

1

1

MAKE_GENOME_PRIMARY_ASSEMBLY

1

1

MAKE_BED_FILES_SHORT_CTRLS

1

1

MAKE_BED_FILES_LONG_CTRLS

1

1

MAKE_SHORT_CTRLS_SIZES_NAMES

1

1

MAKE_LONG_CTRLS_SIZES_NAMES

1

1

MAKE_GENOME_SIZES_NAMES

1

1

MAKE_MOD_REF_FILES

1

1

MAKE_DICT_GENOME

2

4

MAKE_DICT_COMBINED

2

4