Merge branch 'balance2' into weighted-balancing

[Syntax:]

balance thresh style args ... keyword value ... :pre

balance thresh style args ... keyword args ... :pre

thresh = imbalance threshhold that must be exceeded to perform a re-balance :ulb,l

one style/arg pair can be used (or multiple for {x},{y},{z}) :l

    Niter = # of times to iterate within each dimension of dimstr sequence

    stopthresh = stop balancing when this imbalance threshhold is reached

  {rcb} args = none :pre

zero or more optional keywords with their respective arguments may be appended :l

keyword = {out} or {weight} :l

  {out} arg = filename

    filename = write each processor's sub-domain to a file

  {weight} style args = use weighted atom counts to compute the per processor load

zero or more keyword/arg pairs may be appended :l

keyword = {weight} or {out} :l

  {weight} style args = use weighted particle counts for the balancing

    {style} = {group} or {neigh} or {time} or {var} or {store}

      {group} args = Ngroup groupID-1 weight-1 groupID-2 weight-2...

      {group} args = Ngroup group1 weight1 group2 weight2 ...

        Ngroup = number of groups with assigned weights

        groupID-1, groupID-2, ... = group names

        weight-1, weight-2, ...   = corresponding weight factors

        group1, group2, ... = group IDs

        weight1, weight2, ...   = corresponding weight factors

      {neigh} factor = compute weight based on number of neighbors

        factor = scaling factor (> 0)

      {time} factor = compute weight based on time spend computing

        factor = scaling factor (> 0)

      {var} name = take weight from atom-style variable

        name = name of the atom style variable

      {store} name = store weight in custom atom property

        name = name of the atom property (without d_ prefix) :pre

        name = name of the atom-style variable

      {store} name = store weight in custom atom property defined by "fix property/atom"_fix_property_atom.html command

        name = atom property name (without d_ prefix)

  {out} arg = filename

    filename = write each processor's sub-domain to a file :pre

:ule

[Examples:]

This command adjusts the size and shape of processor sub-domains

within the simulation box, to attempt to balance the number of

particles and thus indirectly the computational cost (load)

more evenly across processors.  The load balancing is "static"

in the sense that this command performs the balancing once, before

or between simulations.

The processor sub-domains will then remain static during the

subsequent run.  To perform "dynamic" balancing, see the "fix

particles and thus indirectly the computational cost (load) more

evenly across processors.  The load balancing is "static" in the sense

that this command performs the balancing once, before or between

simulations.  The processor sub-domains will then remain static during

the subsequent run.  To perform "dynamic" balancing, see the "fix

balance"_fix_balance.html command, which can adjust processor

sub-domain sizes and shapes on-the-fly during a "run"_run.html.

With the optional {weight} keyword different weight factors can be

assigned to particles according several styles and balancing will

be performed on the weighted particle counts. Multiple weight

styles may be given and they are processed in order by multiplying

the existing weight factor, which defaults to 1.0 with the newly

computed weight factor. The {store} weight style is an exception,

as does not compute a weight, but instead stores the current

accumulated weights in a custom per-atom property defined with

"fix property/atom"_fix_property_atom.html.

The {group} weight style assigns fixed weight factors according

to which group atoms belong to. The {group} style keyword is

followed by the number of groups with custom weights

(default weight is 1.0) and pairs of group ID and the corresponding

weight factor. The final weight for each atom is the product of

all individual weight factors from the groups it belongs to.

An atom with a total weight of 5 then be will be considered to

have 5x the computational cost than an atom with the default weight

of 1.0.

The {neigh} weight style assigns weights computed from the number

of neighbors divided by the avergage number of neighbors. The

scaling factor argument determines the relative impact of this

weight factor. This weight style will use the first suitable neighbor

list that is internally available and by inactive and print a

warning, if there is not suitable list available. This is typically

the case before the first "run"_run.html or "minimize"_minimize.html

command is issued.

The {time} weight style allows to incorporate "timer data"_timer.html 

into the load balancing cost function. The required weight factor

rgument (a number > 0) determines to which degree timing information

is included. The timer information is taken from the preceding run.

If no such information is available, e.g. at the beginning of an input,

of when the "timer"_timer.html level is set to either {loop} or {off},

this style is ignored.

The {var} weight style allows to set per-atom weights from an

atom-style "variable"_variable.html into the load balancing cost

function.

Load-balancing is typically most useful if the particles in the

simulation box have a spatially-varying density distribution or

where the computational cost varies signficantly between different

atoms. E.g. a model of a vapor/liquid interface, or a solid with

an irregular-shaped geometry containing void regions, or

"hybrid pair style simulations"_pair_hybrid.html which combine

pair styles with different computational cost.  In these cases, the

LAMMPS default of dividing the simulation box volume into a

regular-spaced grid of 3d bricks, with one equal-volume sub-domain

per procesor, may assign numbers of particles per processor in a

way that the computational effort varies significantly.  This can

lead to poor performance when the simulation is run in parallel.

simulation box have a spatially-varying density distribution or when

the computational cost varies signficantly between different atoms or

particles.  E.g. a model of a vapor/liquid interface, or a solid with

an irregular-shaped geometry containing void regions, or "hybrid pair

style simulations"_pair_hybrid.html which combine pair styles with

different computational cost.  In these cases, the LAMMPS default of

dividing the simulation box volume into a regular-spaced grid of 3d

bricks, with one equal-volume sub-domain per procesor, may assign

numbers of particles per processor in a way that the computational

effort varies significantly.  This can lead to poor performance when

the simulation is run in parallel.

The balancing can be performed with or without per-particle weighting.

Without any particle weighting, the balancing attempts to assign an

equal number of particles to each processor.  With weighting, the

balancing attempts to assign an equal weight to each processor, which

typically means a different number of atoms per processor.  Details on

the various weighting options are given below.

Note that the "processors"_processors.html command allows some control

over how the box volume is split across processors.  Specifically, for

The requested load-balancing operation is only performed if the

current "imbalance factor" in particles owned by each processor

exceeds the specified {thresh} parameter.  The imbalance factor is

defined as the maximum number of particles owned by any processor,

divided by the average number of particles per processor.  Thus an

imbalance factor of 1.0 is perfect balance.

defined as the maximum number of particles (or weight) owned by any

processor, divided by the average number of particles (or weight) per

processor.  Thus an imbalance factor of 1.0 is perfect balance.

As an example, for 10000 particles running on 10 processors, if the

most heavily loaded processor has 1200 particles, then the factor is

a logical 3d grid of processors.  They operate by changing the cutting

planes (or lines) between processors in 3d (or 2d), to adjust the

volume (area in 2d) assigned to each processor, as in the following 2d

diagram where processor sub-domains are shown and atoms are colored by

the processor that owns them.  The leftmost diagram is the default

partitioning of the simulation box across processors (one sub-box for

each of 16 processors); the middle diagram is after a "grid" method

has been applied.

diagram where processor sub-domains are shown and particles are

colored by the processor that owns them.  The leftmost diagram is the

default partitioning of the simulation box across processors (one

sub-box for each of 16 processors); the middle diagram is after a

"grid" method has been applied.

:image(JPG/balance_uniform_small.jpg,JPG/balance_uniform.jpg),image(JPG/balance_nonuniform_small.jpg,JPG/balance_nonuniform.jpg),image(JPG/balance_rcb_small.jpg,JPG/balance_rcb.jpg)

:c

The {rcb} style is a "tiling" method which does not produce a logical

3d grid of processors.  Rather it tiles the simulation domain with

rectangular sub-boxes of varying size and shape in an irregular

fashion so as to have equal numbers of particles (or an equal

load in case weighted load-balancing is requested) in each sub-box,

as in the rightmost diagram above.

fashion so as to have equal numbers of particles (or weight) in each

sub-box, as in the rightmost diagram above.

The "grid" methods can be used with either of the

"comm_style"_comm_style.html command options, {brick} or {tiled}.  The

All the processors are also partitioned into 2 groups, half assigned

to the box on the lower side of the cut, and half to the box on the

upper side.  (If the processor count is odd, one side gets an extra

processor.)  The cut is positioned so that the number of atoms in the

lower box is exactly the number that the processors assigned to that

box should own for load balance to be perfect.  This also makes load

balance for the upper box perfect.  The positioning is done

iteratively, by a bisectioning method.  Note that counting atoms on

either side of the cut requires communication between all processors

at each iteration.

processor.)  The cut is positioned so that the number of particles in

the lower box is exactly the number that the processors assigned to

that box should own for load balance to be perfect.  This also makes

load balance for the upper box perfect.  The positioning is done

iteratively, by a bisectioning method.  Note that counting particles

on either side of the cut requires communication between all

processors at each iteration.

That is the procedure for the first cut.  Subsequent cuts are made

recursively, in exactly the same manner.  The subset of processors

assigned to each box make a new cut in the longest dimension of that

box, splitting the box, the subset of processsors, and the atoms in

the box in two.  The recursion continues until every processor is

assigned a sub-box of the entire simulation domain, and owns the atoms

in that sub-box.

box, splitting the box, the subset of processsors, and the particles

in the box in two.  The recursion continues until every processor is

assigned a sub-box of the entire simulation domain, and owns the

particles in that sub-box.

:line

This sub-section describes how to perform weighted load balancing via

the {weight} keyword.

One or more weight factors can be assigned to individual or sets of

particles.  By default all particles have an initial weight of 1.0.

After weighting is applied, a particle with a total weight of 5 will

be balanced with 5x the computational cost of a particle with the

default weight of 1.0.

If one or more weight styles are specified, they are processed in the

order they are specified.  Each style computes a factor which

multiplies the existing factor to produce a cummulative weight on

individual particles.

The {group} weight style assigns weight factors to specified groups of

particles.  The {group} style keyword is followed by the number of

groups, then pairs of group IDs and the corresponding weight factor.

A particle may belong to zero or one or more than one specified group.

Its final factor is simply the product of all individual weight

factors for the groups it belongs to.

The {neigh} weight style assigns a weight to each particle equal to

its number of neighbors divided by the avergage number of neighbors

for all particles.  The {factor} setting is then appied as an overall

scale factor to all the {neigh} weights.  Thus {factor} effectively

sets a relative impact for this weight style.  This weight style will

use the first suitable neighbor list it finds internally.  It will

print a warning if there is no neighbor list or it is not current,

e.g. if the balance command is used before a "run"_run.html or

"minimize"_minimize.html command is used, which can mean that no

neighbor list has yet been built.

The {time} weight style uses "timer data"_timer.html to calculate a

weight for each particle.  The {factor} setting is then appied as an

overall scale factor to all the {time} weights.  Effectively it sets a

relative impact for this weight style.  Timer information is taken

from the preceding run.  NOTE: Entire run or last portion of run?

Which sub-timings within the run?  How is it normalized?  If no such

information is available, e.g. at the beginning of an input, of when

the "timer"_timer.html level is set to either {loop} or {off}, this

style is ignored.

The {var} weight style assigns per-particle weights by evaluating an

atom-style "variable"_variable.html specified by {name}.

The {store} weight style does not compute a weight factor.  Instead it

stores the current accumulated weights in a custom per-atom property

specified by {name}.  This must be a property defined as {d_name} via

the "fix property/atom"_fix_property_atom.html command.  Note that

these custom per-atom properties can be output in a "dump"_dump.html

file, so this is a way to examine per-particle weights.

:line