Merge pull request #196 from akohlmey/charmm-cmap-updates

In this directory, you should find:

1) A perl script called "charmm2lammps.pl"

2) A perl script called "lammps2pdb.pl"

3) An "example" folder containing an example of how to use these tools.

3) Two folders containing examples of how to use these tools.

4) An "other" folder containing other potentially useful tools.

In addition, you will need to provide the following input for

<forcefield> <project> [-option=value ...]" where <forcefield> is the

name of the CHARMM FF you are using, (i.e. all22_prot), and <project>

is the common name of your *.crd and *.psf files.  Possible options

are listed next.  If the option requires a parameter, the syntax

should be -option=value (e.g. -border=5).

are listed below.  If the option requires a parameter, the syntax

must be like this: -option=value (e.g. -border=5).

-help        display available options

-charmm      add charmm types to LAMMPS data file

-nohints     do not add charmm types and style hints to LAMMPS data file

-water       add TIP3P water [default: 1 g/cc]

-ions        add (counter)ions using Na+ and Cl- [default: 0 mol/l]

-center      recenter atoms

-ax          rotation around x-axis

-ay          rotation around y-axis

-az          rotation around z-axis

-cd          correction for dihedral for carbohydrate systems

-cmap        add CMAP section to data file and fix cmap command lines in

             input script" (NOTE: requires use of *.pdb file)

trajectory from the LAMMPS output.

1) Get the pdb file you want to model. http://www.rcsb.org/pdb/ For

this example, we will use 1ac7.pdb

this examples, we will use either 1ac7.pdb or 1gb1.pdb

2) If there are multiple models in the pdb file, choose the one you

want and delete the others. Save the pared-down file as 1ac7_pared.pdb

3) Download the charmm FF files and choose the one you want from the

tarball.  We will use all27_na for this example.

http://www.pharmacy.umaryland.edu/faculty/amackere/force_fields.htm

toppar_c31b1.tar.gz

tarball.  We will use all27_na for 1AC7 and all36_prot for 1GB1. The

required files for the example are included in their folders. You can

download complete CHARMM force field packages from:

http://mackerell.umaryland.edu/charmm_ff.shtml

4) Create a *.pgn file for use with psfgen (you will need to have VMD

installed, http://www.ks.uiuc.edu/Research/vmd/ ). This is the hardest

writepsf charmm 1ac7.psf

exit

5) Type "vmd -e 1ac7.pgn" to build the 1ac7.psf file, and the new

For 1GB1 the corresponding 1gb1.psf file has been created with CHARMM-GUI,

http://www.charmm-gui.org

5) Type "vmd -dispdev none -e 1ac7.pgn" to build the 1ac7.psf file, and the new

   1ac7.pdb file.

6) Run charmm2lammps.pl by typing:

"perl charmm2lammps.pl all27_na 1ac7 -charmm -border=1 -pdb_ctrl -water -ions"

"perl charmm2lammps.pl all27_na 1ac7 -border=2.0 -pdb_ctrl -water -ions"

or

"perl charmm2lammps.pl all36_prot 1gb1 -border=2.0 -cmap=36 -water -ions"

7) Run lammps by typing: "lmp < 1ac7.in"

7) Run lammps by typing: "lmp_mpi -in 1ac7.in" or "lmp_mpi -in 1gb1.in"

8) Run lammps2pdb.pl by typing: "perl lammps2pdb.pl 1ac7"

** Additional notes:

The charmm2lammps.pl script takes the pdb and psf files for the 1ac7

molecule and converts them into LAMMPS format. The -water option

or 1gb1 molecules and converts them into LAMMPS format. The -water option

embeds the molecule in water on a crystal lattice. The -border option

includes a layer of water surrounding the minimum dimensions of the

molecule. The -pdb_ctrl option produces the 1ac7_ctrl.pdb file that

can be visualized in a standard visualization package such as VMD. The

-charmm option put comments into the LAMMPS data file (everything

can be visualized in a standard visualization package such as VMD. By

default, comments are added into the LAMMPS data file (everything

after the # sign is a comment) for user convenience in tracking atom

types etc. according to CHARMM nomenclature.

types etc. according to CHARMM nomenclature. If this is not desired,

the -nohints option can be used to turn off this function.

The example molecule provided above (i.e., 1ac7) is a DNA fragment.

If instead, a peptide longer than 2 amino acid residues or a protein

is to be modeled, the '-cmap' option should be used. This will add a

section at the end of the data file with the heading of 'CMAP' that

The provided 1ac7 example molecule is a DNA fragment. For peptides

longer than 2 amino acid residues or a protein is to be modeled,

e.g. the 1gb1 binding domain of a protein, the '-cmap' option should

be used. This will add CMAP section at the end of the data file that

will contain cmap crossterm corrections for the phi-psi dihedrals for

the amino acid residues. You will then need to also copy the

appropriate file for the cmap crossterms into your directory and be

sure that you are using the appropriate cmap crossterms that go with

the respective version of the charmm force field that is being used

(e.g, cmap22.data or cmap36.data). This is necessary to account for

the fact that the CHARMM group has provided updated cmap correction

(e.g, charmm22.cmap or charmm36.cmap). This is necessary to account

for the fact that the CHARMM group has provided updated cmap correction

terms for use with the c36 and more recent version of the charmm

protein force field.  Copies of cmap22.data and cmap36.data are

provided in the tools/ch2lmp directory.

protein force field.  Copies of charmm22.cmap and charmm36.cmap are

provided in the potentials directory.

The default timestep in the LAMMPS *.in file is set to 0.5 fs, which

The default timestep in the LAMMPS *.in file is set to 1.0 fs, which

can typically be increased to 2 fs after equilibration if the bonds

involving H are constrained via shake. Also, after equilibration, the

delay on neigh_modify can probably increased to 5 or so to improve

The -ions option allows the user to neutralize the simulation cell

with Na+ or Cl- counterions if the molecule has a net charge

Additional salt can be added by increasing the default concentration

(e.g. -ions=0.5).

(i.e., -ions=0.5).

** In the "other" file folder, you will find:

3) A 3rd party perl script called "crd2pdb.pl"

4) A 3rd party fortran code called "pdb_to_crd.f"

#    20161001   Added 'CMAP crossterms' section at the end of the data file

#    20161001   Added instructions in CMAP section to fix problem if 'ter'

#                 is not designated in the .pdb file to identify last amino acid

#    20161005   Added tweak to embed command line in generated LAMMPS input

#

#    General    Many thanks to Paul S. Crozier for checking script validity

#               against his projects.

    my $notes;

    $program            = "charmm2lammps";

    $version		= "1.9.0";

    $version            = "1.9.1";

    $year               = "2016";

    $add                = 1;

    $water_dens         = 0;

    $notes .= "      - project.in            suggested LAMMPS input script\n";

    $notes .= "      - project_ctrl.pdb      control file when requested\n";

    # record full command line for later use

    $cmdline = "$program.pl " . join(" ",@ARGV);

    foreach (@ARGV)

    {

      if (substr($_, 0, 1) eq "-")

  sub WriteLAMMPSInput

  {

    open(LAMMPS, ">$project.in");                       # input file

    printf(LAMMPS "# Created by $program v$version on %s\n", `date`);

    printf(LAMMPS "# Created by $program v$version on %s", `date`);

    printf(LAMMPS "# Command: %s\n\n", $cmdline);

    printf(LAMMPS "units           real\n");            # general

    printf(LAMMPS "neigh_modify    delay 2 every 1\n\n");

    printf(LAMMPS "atom_style      full\n");            # styles

      printf(LAMMPS "\n");

    }

    printf(LAMMPS "special_bonds   charmm\n");          # invoke charmm

    printf(LAMMPS "thermo          1\n");		# set thermo style

    printf(LAMMPS "thermo          10\n");              # set thermo style

    printf(LAMMPS "thermo_style    multi\n");

    printf(LAMMPS "timestep        0.5\n\n");		# 0.5 ps time step

    printf(LAMMPS "minimize 0.0 0.0 1000 10000\n\n");   # take off the edge

    printf(LAMMPS "timestep        1.0\n\n");           # 1.0 ps time step

    printf(LAMMPS "minimize 0.0 0.0 50 200\n\n");       # take of the edge

    printf(LAMMPS "reset_timestep  0\n");

    printf(LAMMPS "fix             1 all nve\n");

    printf(LAMMPS "fix             2 all shake 1e-6 500 0 m 1.0\n")

      if ($shake eq "");                                # shake all H-bonds

    printf(LAMMPS "fix             2 all shake 1e-6 500 0 m 1.0 a %s\n",$shake)

      if ($shake ne "");                                # add water if present

    printf(LAMMPS "velocity        all create 0.0 12345678 dist uniform\n\n");

    printf(LAMMPS "restart         10 $project.restart1 $project.restart2\n");

    printf(LAMMPS "dump            1 all atom 10 $project.dump\n");

    printf(LAMMPS "restart         500 $project.restart1 $project.restart2\n");

    printf(LAMMPS "dump            1 all atom 100 $project.dump\n");

    printf(LAMMPS "dump_modify     1 image yes scale yes\n\n");

    printf(LAMMPS "run             20\n");		# run for 20 time steps

    printf(LAMMPS "thermo          100\n");		# set thermo style

    printf(LAMMPS "run             1000\n");		# run for 1000 time steps

    close(LAMMPS);

  }

# Created by charmm2lammps v1.9.1 on Wed Oct  5 18:08:55 EDT 2016

# Command: charmm2lammps.pl -water -ions -border=2.0 all36_prot 1gb1

units           real

neigh_modify    delay 2 every 1

atom_style      full

bond_style      harmonic

angle_style     charmm

dihedral_style  charmm

improper_style  harmonic

pair_style      lj/charmm/coul/long 8 12

pair_modify     mix arithmetic

kspace_style    pppm 1e-6

read_data       1gb1.data

special_bonds   charmm

thermo          10

thermo_style    multi

timestep        1.0

minimize 0.0 0.0 50 200

reset_timestep  0

fix             1 all nve

fix             2 all shake 1e-6 500 0 m 1.0 a 112

velocity        all create 0.0 12345678 dist uniform

restart         500 1gb1.restart1 1gb1.restart2

dump            1 all atom 100 1gb1.dump

dump_modify     1 image yes scale yes

thermo          100

run             1000

LAMMPS (5 Oct 2016)

package omp 0

WARNING: OpenMP support not enabled during compilation; using 1 thread only. (../fix_omp.cpp:123)

# Created by charmm2lammps v1.9.1 on Wed Oct  5 18:08:55 EDT 2016

# Command: charmm2lammps.pl -water -ions -border=2.0 all36_prot 1gb1

units           real

neigh_modify    delay 2 every 1

atom_style      full

bond_style      harmonic

angle_style     charmm

dihedral_style  charmm

improper_style  harmonic

pair_style      lj/charmm/coul/long 8 12

pair_modify     mix arithmetic

kspace_style    pppm 1e-6

read_data       1gb1.data

  orthogonal box = (-20.8413 -14.5892 -13.805) to (23.4963 14.9692 10.827)

  2 by 2 by 2 MPI processor grid

  reading atoms ...

  2833 atoms

  scanning bonds ...

  4 = max bonds/atom

  scanning angles ...

  6 = max angles/atom

  scanning dihedrals ...

  24 = max dihedrals/atom

  scanning impropers ...

  2 = max impropers/atom

  reading bonds ...

  2177 bonds

  reading angles ...

  2209 angles

  reading dihedrals ...

  2707 dihedrals

  reading impropers ...

  137 impropers

  4 = max # of 1-2 neighbors

  9 = max # of 1-3 neighbors

  17 = max # of 1-4 neighbors

  21 = max # of special neighbors

special_bonds   charmm

thermo          10

thermo_style    multi

timestep        1.0

minimize 0.0 0.0 50 200

WARNING: Resetting reneighboring criteria during minimization (../min.cpp:168)

PPPM initialization ...

WARNING: Using 12-bit tables for long-range coulomb (../kspace.cpp:316)

  G vector (1/distance) = 0.278254

  grid = 45 32 30

  stencil order = 5

  estimated absolute RMS force accuracy = 0.000317429

  estimated relative force accuracy = 9.55928e-07

  using double precision FFTs

  3d grid and FFT values/proc = 15180 5760

Last active /omp style is kspace_style pppm/omp

Neighbor list info ...

  1 neighbor list requests

  update every 1 steps, delay 0 steps, check yes

  max neighbors/atom: 2000, page size: 100000

  master list distance cutoff = 14

  ghost atom cutoff = 14

  binsize = 7 -> bins = 7 5 4

Memory usage per processor = 19.7492 Mbytes

---------------- Step        0 ----- CPU =      0.0000 (sec) ----------------

TotEng   =     -3599.1065 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -3599.1065 E_bond   =       169.6834 E_angle  =       103.6483 

E_dihed  =       583.9669 E_impro  =         0.0335 E_vdwl   =      -776.0899 

E_coul   =     35419.6941 E_long   =    -39100.0428 Press    =      3623.5803 

---------------- Step       10 ----- CPU =      0.2563 (sec) ----------------

TotEng   =     -5168.1778 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -5168.1778 E_bond   =       120.7538 E_angle  =       171.5023 

E_dihed  =       583.0143 E_impro  =         6.5958 E_vdwl   =     -1034.8022 

E_coul   =     34737.4006 E_long   =    -39752.6424 Press    =    -10232.1944 

---------------- Step       20 ----- CPU =      0.4597 (sec) ----------------

TotEng   =     -5784.4945 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -5784.4945 E_bond   =       110.1359 E_angle  =       262.6659 

E_dihed  =       573.8164 E_impro  =         8.5071 E_vdwl   =     -1050.9646 

E_coul   =     34464.2659 E_long   =    -40152.9212 Press    =    -10843.0328 

---------------- Step       30 ----- CPU =      0.6813 (sec) ----------------

TotEng   =     -6342.0862 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -6342.0862 E_bond   =       146.0247 E_angle  =       272.7224 

E_dihed  =       557.6616 E_impro  =         7.1074 E_vdwl   =      -984.4028 

E_coul   =     34031.1722 E_long   =    -40372.3718 Press    =    -13674.9873 

---------------- Step       40 ----- CPU =      0.8599 (sec) ----------------

TotEng   =     -6821.3956 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -6821.3956 E_bond   =       147.4491 E_angle  =       271.5247 

E_dihed  =       548.2429 E_impro  =         7.1832 E_vdwl   =      -873.4714 

E_coul   =     33514.8072 E_long   =    -40437.1312 Press    =    -12024.8390 

---------------- Step       50 ----- CPU =      1.0662 (sec) ----------------

TotEng   =     -7278.3435 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -7278.3435 E_bond   =       181.9934 E_angle  =       288.9107 

E_dihed  =       541.0050 E_impro  =         7.1673 E_vdwl   =      -501.3825 

E_coul   =     32665.6923 E_long   =    -40461.7297 Press    =    -10391.9829 

Loop time of 1.06631 on 8 procs for 50 steps with 2833 atoms

99.3% CPU use with 8 MPI tasks x 1 OpenMP threads

Minimization stats:

  Stopping criterion = max iterations

  Energy initial, next-to-last, final = 

        -3599.10649881     -7238.80368107     -7278.34352699

  Force two-norm initial, final = 1109.81 247.325

  Force max component initial, final = 107.492 37.5055

  Final line search alpha, max atom move = 0.00133024 0.0498915

  Iterations, force evaluations = 50 87

MPI task timing breakdown:

Section |  min time  |  avg time  |  max time  |%varavg| %total

---------------------------------------------------------------

Pair    | 0.46313    | 0.53526    | 0.61611    |   6.9 | 50.20

Bond    | 0.0070326  | 0.01008    | 0.013407   |   1.8 |  0.95

Kspace  | 0.3364     | 0.41785    | 0.49165    |   8.0 | 39.19

Neigh   | 0.035014   | 0.035175   | 0.035337   |   0.1 |  3.30

Comm    | 0.058533   | 0.05973    | 0.060792   |   0.3 |  5.60

Output  | 0.00039005 | 0.00040504 | 0.00049496 |   0.2 |  0.04

Modify  | 0.0015707  | 0.0022006  | 0.0025775  |   0.6 |  0.21

Other   |            | 0.005615   |            |       |  0.53

Nlocal:    354.125 ave 376 max 344 min

Histogram: 3 1 0 1 2 0 0 0 0 1

Nghost:    7405 ave 7531 max 7307 min

Histogram: 1 2 1 0 0 1 1 1 0 1

Neighs:    178281 ave 194848 max 158123 min

Histogram: 2 0 1 0 0 1 0 1 2 1

Total # of neighbors = 1426250

Ave neighs/atom = 503.442

Ave special neighs/atom = 4.67702

Neighbor list builds = 4

Dangerous builds = 0

reset_timestep  0

fix             1 all nve

fix             2 all shake 1e-6 500 0 m 1.0 a 112

  168 = # of size 2 clusters

  64 = # of size 3 clusters

  41 = # of size 4 clusters

  658 = # of frozen angles

velocity        all create 0.0 12345678 dist uniform

restart         500 1gb1.restart1 1gb1.restart2

dump            1 all atom 100 1gb1.dump

dump_modify     1 image yes scale yes

thermo          100

run             1000

PPPM initialization ...

WARNING: Using 12-bit tables for long-range coulomb (../kspace.cpp:316)

  G vector (1/distance) = 0.278254

  grid = 45 32 30

  stencil order = 5

  estimated absolute RMS force accuracy = 0.000317429

  estimated relative force accuracy = 9.55928e-07

  using double precision FFTs

  3d grid and FFT values/proc = 15180 5760

Last active /omp style is kspace_style pppm/omp

Neighbor list info ...

  1 neighbor list requests

  update every 1 steps, delay 2 steps, check yes

  max neighbors/atom: 2000, page size: 100000

  master list distance cutoff = 14

  ghost atom cutoff = 14

  binsize = 7 -> bins = 7 5 4

Memory usage per processor = 20.5221 Mbytes

---------------- Step        0 ----- CPU =      0.0000 (sec) ----------------

TotEng   =     -7547.0017 KinEng   =         0.0000 Temp     =         0.0000 

PotEng   =     -7547.0017 E_bond   =        52.4535 E_angle  =       149.7924 

E_dihed  =       541.0050 E_impro  =         7.1673 E_vdwl   =      -501.3825 

E_coul   =     32665.6923 E_long   =    -40461.7297 Press    =     -9600.9476 

---------------- Step      100 ----- CPU =      0.8242 (sec) ----------------

TotEng   =     -7049.9503 KinEng   =       688.1123 Temp     =       113.4757 

PotEng   =     -7738.0625 E_bond   =        53.8035 E_angle  =       139.7846 

E_dihed  =       529.5460 E_impro  =         6.1806 E_vdwl   =        52.8666 

E_coul   =     31951.7370 E_long   =    -40471.9810 Press    =     -5991.4443 

---------------- Step      200 ----- CPU =      1.6765 (sec) ----------------

TotEng   =     -7050.0350 KinEng   =       956.0848 Temp     =       157.6667 

PotEng   =     -8006.1198 E_bond   =        47.1331 E_angle  =       140.9769 

E_dihed  =       517.5474 E_impro  =         5.7244 E_vdwl   =       635.2166 

E_coul   =     31127.8100 E_long   =    -40480.5282 Press    =      -920.9624 

---------------- Step      300 ----- CPU =      2.5246 (sec) ----------------

TotEng   =     -7050.0984 KinEng   =      1049.7346 Temp     =       173.1104 

PotEng   =     -8099.8330 E_bond   =        47.4551 E_angle  =       156.5090 

E_dihed  =       520.6006 E_impro  =         9.4008 E_vdwl   =       451.0904 

E_coul   =     31202.3817 E_long   =    -40487.2707 Press    =     -2352.9550 

---------------- Step      400 ----- CPU =      3.3982 (sec) ----------------

TotEng   =     -7050.1671 KinEng   =      1099.2465 Temp     =       181.2753 

PotEng   =     -8149.4136 E_bond   =        56.6375 E_angle  =       164.5897 

E_dihed  =       528.8356 E_impro  =         8.8390 E_vdwl   =       525.0114 

E_coul   =     31060.0380 E_long   =    -40493.3650 Press    =     -2146.9087 

---------------- Step      500 ----- CPU =      4.2800 (sec) ----------------

TotEng   =     -7050.2027 KinEng   =      1134.4924 Temp     =       187.0877 

PotEng   =     -8184.6951 E_bond   =        53.5903 E_angle  =       169.1090 

E_dihed  =       522.9488 E_impro  =         7.9830 E_vdwl   =       496.8404 

E_coul   =     31058.9967 E_long   =    -40494.1633 Press    =     -2537.1829 

---------------- Step      600 ----- CPU =      5.1548 (sec) ----------------

TotEng   =     -7050.1894 KinEng   =      1122.6756 Temp     =       185.1390 

PotEng   =     -8172.8649 E_bond   =        47.9283 E_angle  =       168.7967 

E_dihed  =       518.7200 E_impro  =         9.2011 E_vdwl   =       499.6778 

E_coul   =     31076.1916 E_long   =    -40493.3805 Press    =     -1939.2216 

---------------- Step      700 ----- CPU =      6.0377 (sec) ----------------

TotEng   =     -7050.2850 KinEng   =      1154.4452 Temp     =       190.3781 

PotEng   =     -8204.7301 E_bond   =        54.3594 E_angle  =       183.8772 

E_dihed  =       513.0797 E_impro  =         9.6842 E_vdwl   =       522.0052 

E_coul   =     31008.3632 E_long   =    -40496.0990 Press    =     -2059.4885 

---------------- Step      800 ----- CPU =      6.9161 (sec) ----------------

TotEng   =     -7050.2852 KinEng   =      1146.2741 Temp     =       189.0306 

PotEng   =     -8196.5593 E_bond   =        48.8126 E_angle  =       171.3411 

E_dihed  =       525.8682 E_impro  =        10.5588 E_vdwl   =       538.6905 

E_coul   =     31003.8447 E_long   =    -40495.6752 Press    =     -1458.5181 

---------------- Step      900 ----- CPU =      7.8101 (sec) ----------------

TotEng   =     -7050.3025 KinEng   =      1154.2134 Temp     =       190.3398 

PotEng   =     -8204.5159 E_bond   =        51.1010 E_angle  =       177.9642 

E_dihed  =       520.6908 E_impro  =         8.9539 E_vdwl   =       521.5994 

E_coul   =     31012.6307 E_long   =    -40497.4560 Press    =     -1683.9131 

---------------- Step     1000 ----- CPU =      8.7313 (sec) ----------------

TotEng   =     -7050.3166 KinEng   =      1179.9351 Temp     =       194.5816 

PotEng   =     -8230.2517 E_bond   =        57.8905 E_angle  =       190.1556 

E_dihed  =       518.1288 E_impro  =        10.0403 E_vdwl   =       557.6413 

E_coul   =     30933.6472 E_long   =    -40497.7554 Press    =     -1643.1247 

Loop time of 8.73139 on 8 procs for 1000 steps with 2833 atoms

Performance: 9.895 ns/day, 2.425 hours/ns, 114.529 timesteps/s

99.6% CPU use with 8 MPI tasks x 1 OpenMP threads

MPI task timing breakdown:

Section |  min time  |  avg time  |  max time  |%varavg| %total

---------------------------------------------------------------

Pair    | 3.2841     | 3.856      | 4.393      |  17.3 | 44.16

Bond    | 0.041216   | 0.074389   | 0.11041    |   7.2 |  0.85

Kspace  | 2.8668     | 3.4354     | 4.0036     |  18.9 | 39.35

Neigh   | 0.3024     | 0.30315    | 0.30402    |   0.1 |  3.47

Comm    | 0.58794    | 0.60511    | 0.61362    |   1.0 |  6.93

Output  | 0.010607   | 0.010696   | 0.010783   |   0.0 |  0.12

Modify  | 0.40802    | 0.41482    | 0.42598    |   0.8 |  4.75

Other   |            | 0.03184    |            |       |  0.36

Nlocal:    354.125 ave 390 max 323 min

Histogram: 2 0 2 0 1 0 0 0 1 2

Nghost:    7219.12 ave 7453 max 6941 min

Histogram: 1 1 1 1 0 0 0 0 2 2

Neighs:    179095 ave 217661 max 144647 min

Histogram: 1 0 1 1 1 2 1 0 0 1

Total # of neighbors = 1432758

Ave neighs/atom = 505.739

Ave special neighs/atom = 4.67702

Neighbor list builds = 39

Dangerous builds = 0

Total wall time: 0:00:09