Difference between revisions of "GotCloud"
Line 30: | Line 30: | ||
Next, use the following commands to open an [[SLURM Interactive|SRUN Interactive Session]], load the module for {{PAGENAME}}, and execute the {{PAGENAME}} binary: | Next, use the following commands to open an [[SLURM Interactive|SRUN Interactive Session]], load the module for {{PAGENAME}}, and execute the {{PAGENAME}} binary: | ||
<pre style="white-space:pre-wrap; width:68%; border:1px solid lightgrey; background:#000000; color:white;">[user@login0 ~]$ srun --time=48:00:00 --nodes=1 -- | <pre style="white-space:pre-wrap; width:68%; border:1px solid lightgrey; background:#000000; color:white;">[user@login0 ~]$ srun --time=48:00:00 --nodes=1 --cpus-per-task=1 --pty /bin/bash | ||
[user@wh-520-4-1 ~]$ module load apps/gotcloud/1.17.5 | [user@wh-520-4-1 ~]$ module load apps/gotcloud/1.17.5 | ||
[user@wh-520-4-1 ~]$ gotcloud align -conf ./test.conf -out ./out | [user@wh-520-4-1 ~]$ gotcloud align -conf ./test.conf -out ./out |
Revision as of 14:22, 31 May 2022
Description
From the GotCloud Home Page: To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called Genomes on the Cloud (GotCloud).
GotCloud contains Mapping & Variant Calling Pipelines.
Key Features:
- Connects sequence analysis tools together in automated pipeline
- Alignment, quality control, variant calling
- Robust against unexpected system failure using GNU make
- easy restart after failure
- Massively parallel, can run hundreds of jobs
- Splits large jobs into many pieces
- Simplifies running on clusters
- Scalable to tens of thousands of samples
- Easy to use - Automates series of configurable steps
- user doesn’t have to understand/configure/know the many tools required to create high quality results
- Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
- Run on local machines/clusters
GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user’s computer, on an instance in a compute cloud, and/or can split the work up onto a cluster of machines or instances.
Version
- 1.17.5
Authorized Users
CIRCE
account holdersRRA
account holdersSC
account holders
Platforms
CIRCE
clusterRRA
clusterSC
cluster
Modules
GotCloud requires the following module file to run:
apps/gotcloud/1.17.5
- See Modules for more information.
Running GotCloud on CIRCE/SC
The GotCloud user guide is essential to understanding the application and making the most of it. The guide and this page should help you to get started with your simulations. Please refer to the Documentation section for a link to the guide.
- Note on CIRCE: Make sure to run your jobs from your $WORK directory!
- Note: Scripts are provided as examples only. Your SLURM executables, tools, and options may vary from the example below. For help on submitting jobs to the queue, see our SLURM User’s Guide.
Interactive Mode
Next, use the following commands to open an SRUN Interactive Session, load the module for GotCloud, and execute the GotCloud binary:
[user@login0 ~]$ srun --time=48:00:00 --nodes=1 --cpus-per-task=1 --pty /bin/bash [user@wh-520-4-1 ~]$ module load apps/gotcloud/1.17.5 [user@wh-520-4-1 ~]$ gotcloud align -conf ./test.conf -out ./out
Batch Job submission
To run batch jobs on the CIRCE/SC cluster, users will need to submit their jobs to the scheduling environment if their jobs take more than 20 minutes to run on a standard PC.
If, for example, you wish to align GotCloud’s test data, you would set up a submit script to use GotCloud like this
#!/bin/bash # #SBATCH --job-name=gotcloud-test #SBATCH --time=01:00:00 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --output=output.%j.gotcloud-test #### SLURM 1 processor gotcloud test to run for 1 hours. # Load the gotcloud module: module load apps/gotcloud/1.17.5 # Start GotCloud, telling GotCloud to schedule tasks using SLURM. Use -batchopts flag to pass additional parameters gotcloud align -conf ./test.conf -out ./out -batchtype=slurm
Next, you can change to your job’s directory, and run the sbatch command to submit the job:
[user@login0 ~]$ cd my/jobdir [user@login0 jobdir]$ sbatch ./gotcloud-test.sh
- You can view the status of your job with the “squeue -u <username>” command
Documentation
Home Page, User Guides, and Manuals
- GotCloud Home Page:
- GotCloud Documentation
- GotCloud Cluster Utilization Documentation
Benchmarks, Known Tests, Examples, Tutorials, and Other Resources
- GotCloud Reference and Resource Files
More Job Information
See the following for more detailed job submission information:
Reporting Bugs
Report bugs with GotCloud to the IT Help Desk: rc-help@usf.edu