This tutorial requires that
- you have a user account at CSC
- your account belongs to a project that has access to Puhti service. - your account belongs to the Gaussian users group
This tutorial is done on Puhti
💬 GREASY metascheduler is a tool for task/job farming and running “embarrassingly parallel” jobs.
đź’¬ As an example, we have several similar molecular structures and would like to know how they are energetically related to each other
mkdir -p /scratch/project_xxxx/yourcscusername/gaussian_greasy # replace xxxx to match the project name and yourcscusername to match – well – your user name...
cd /scratch/project_xxxx/gaussian_greasy
wget https://a3s.fi/C6H12_structures_10/C6H12_structures_10.tgz
tar -xzf C6H12_structures_10.tgz
cd C6H12_structures_10
đź’¬ Gaussian is a program for molecular electronic structure calculations.
module load openbabel
obabel *.mol -ocom -m
com
format that is used by gaussianđź’¬ In this example ee want to do do a b3lyp/cc-pVDZ
calculation on these structures.
b3lyp/cc-pVDZ
keyword at the beginning of all the com files:
for i in *.com; do sed -i '1s/^/#b3lyp\/cc-pVDZ \n/' $i; done
%NProcShared=4
to the input files:
for i in *.com; do sed -i '1s/^/%NProcShared=4\n/' $i; done
đź’¬ A tasklist can sometimes be lengthy so rather than typing the list by hand it is more feasible to use a Bash script that will create a suitable tasklist file for GREASY.
cd ..
generate_tasklist.sh
nano generate_tasklist.sh
#!/bin/bash
#
submission_dir=$PWD # Directory from where the job is submitted
com_dir=${submission_dir}/C6H12_structures_10 # Subdirectory containing the com files
Ntasks=$(ls -l ${com_dir}/*.com|wc -l) # Number of tasks equals the number of com-files
Ncores=4 # Number of threads per task
rm -f greasy_"${Ntasks}".tasklist # Remove possible old tasklist
for f in ${com_dir}/*.com; # Loop over all com files and create a separate
do # output directory named after the input file name
input_base=`basename ${f%%.*}`
mkdir -p output/${input_base}
# Write all the Gaussian command lines into a common tasklist file
echo "g16 < ${f} > output/${input_base}/${input_base}.log" >> greasy_"${Ntasks}".tasklist
done
nano
and save the file. bash ./generate_tasklist.sh
greasy_10.tasklist
that contains the Gaussian executing commands for the 10 com
files on separate lines.more
, less
or cat
. The file should look like:
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/10737.com > output/10737/10737.log
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/10775.com > output/10775/10775.log
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/10776.com > output/10776/10776.log
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/11109.com > output/11109/11109.log
...and stuff
đź’¬ Submitting a GREASY tasklist is in a way equal to submitting a sbatch jobfile.
module load greasy gaussian
sbatch-greasy
to start the the 10 Gaussian jobs:
sbatch-greasy --cores 4 --time 02:00 --nodes 1 --account project_xxxx greasy_10.tasklist # replace xxxx to match your project name
đź’¬ The command submits the GREASY job using the newly generated GREASY tasklist greasy_10.tasklist
and the following resource requests:
--cores 4
for each job (matches to the spesification in gaussian input files)--time 02:00
--nodes 1
--account project_xxxx
project_xxxx
with the actual project name the prompt asks you to choose one Task list "greasy_10.tasklist" includes 10 tasks.
The first two rows of the task list are:
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/10737.com > output/10737/10737.log
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/10775.com > output/10775/10775.log
-------------------------------------------------------------------------
Submitting GREASY job consisting of 10 tasks to 1 nodes.
The job will run 10 tasks at the time each using 4 cores.
The maximum runtime reseved to process all the tasks is 0 h 5 m.
Job submitted with ID slurmjobid
You can monitor the progress of the task with command:
squeue -j slurmjobid
Once the job has started you can monitor the progress of the job with command:
tail -f greasy-slurmjobid.log
squeue -j slurmjobid # replace slurmjobid with the job ID
squeue --me
squeue -u $USER
grep Summary greasy-*.log
INFO: Summary of 10 tasks: 9 OK, 1 FAILED, 0 CANCELLED, 0 INVALID.
greasy_10.tasklist-undefined.rst
cat greasy_10.tasklist-undefined.rst
# Greasy restart file generated at 2021-03-09 20:10:27
# Original task file: /scratch/project_xxxx/gaussian_greasy/greasy_10.tasklist
# Log file: /scratch/project_xxxx/gaussian_greasy/greasy-5162452.log
#
# Warning: Task 10 failed
g16 < /scratch/project_xxxx/gaussian_greasy/C6H12_structures_10/7787.com > output/7787/7787.log
# End of restart file
7787.com
has failed! tail output/7787/7787.log
Charge and Multiplicity card seems defective:
Wanted an integer as input.
?
Error termination via Lnk1e in /appl/soft/chem/gaussian/G16RevC.01_new/g16/l101.exe at Tue Mar 9 20:10:15 2021.
Job cpu time: 0 days 0 hours 0 minutes 0.8 seconds.
Elapsed time: 0 days 0 hours 0 minutes 0.2 seconds.
File lengths (MBytes): RWF= 6 Int= 0 D2E= 0 Chk= 1 Scr= 1
cat C6H12_structures_10/7*.com
sed -i "6s/^/7787/" C6H12_structures_10/7787.com
sbatch-greasy --cores 4 --time 02:00 --nodes 1 --account project_xxxx greasy_10.tasklist-undefined.rst # replace xxxx to match your project name
grep Summary greasy-*.log
b3lyp/cc-pVDZ
energies for the 10 structures:
grep -rnw 'output/' -e 'E(RB3LYP)'
output/12446/12446.log:265: SCF Done: E(RB3LYP) = -235.836869989 A.U. after 13 cycles
output/10737/10737.log:265: SCF Done: E(RB3LYP) = -235.826753630 A.U. after 13 cycles
output/10776/10776.log:246: SCF Done: E(RB3LYP) = -235.851091573 A.U. after 12 cycles
output/10775/10775.log:246: SCF Done: E(RB3LYP) = -235.835303716 A.U. after 12 cycles
output/11742/11742.log:246: SCF Done: E(RB3LYP) = -235.845122585 A.U. after 13 cycles
output/7024/7024.log:246: SCF Done: E(RB3LYP) = -235.875921299 A.U. after 11 cycles
output/553629/553629.log:262: SCF Done: E(RB3LYP) = -235.838082463 A.U. after 11 cycles
output/12201/12201.log:246: SCF Done: E(RB3LYP) = -235.823223660 A.U. after 13 cycles
output/7787/7787.log:246: SCF Done: E(RB3LYP) = -235.882771348 A.U. after 10 cycles
output/11109/11109.log:265: SCF Done: E(RB3LYP) = -235.823585171 A.U. after 12 cycles