GitLab

The increment was only happening when sleeping was required, but
obviously needs to happen always.

The previous code was obtaining the jobno and opening the log before
any necessary sleeping, which two undesirable effects:

1. The "Starting job n" messages looked fairly random when jobs were
delayed.
2. The initial batch of jobs were allocated not as hosts are actually
available, but when hosts are available after a delay.  For a fast job
with some hosts having a large number of CPUs, this means the job could
take unnecessarily long: e.g. imagine a 'hostA=1 hostB=99' with an
instantaneous job (and connection) executing 100 jobs: all jobnos would
be instantly grabbed by the 99 threads, but the last job on hostB
wouldn't run for 99 DELAY periods, even though the optimal solution is
50 jobs each (and a total time of 50 DELAYs instead of 99 DELAYs).

This fixes the problem by checking and obtaining a jobno after
finishing the delay.

abort_jobs was only checked when entering the job loop, but if a job
entered the loop then had to wait, it didn't check abort_jobs again, so
the delayed jobs would start up even after a Ctrl-C was received.

If a host supports many simultaneous jobs (e.g. a 20-core system),
establishing all 20 connections simultaneously is a bad idea; this adds
a minimum delay (defaulting to 0.25) between connections to the same
host.

FIRST lets you control the first PARALLEL_JOBNO (instead of always
starting from 1).  This is useful when you do something like
'$((PARALLEL_JOBNO % 10))' to select one of 10 different cases in the
job, so that you can restart exactly where you left off, instead of
always having to restart at case '1'.

Added --help argument (also HELP, -h, -?) to display full help.  Bad/no
arguments now displays a short usage message without the help details.

Allowed HOSTS, PER_HOST, and FIRST to be set through environment
variables, prefixed with PARALLEL_, e.g.

    $ export PARALLEL_FIRST=27
    $ parallel-runner 10 'echo hi'

now does the same thing as:

    $ parallel-runner FIRST=27 10 'echo hi'

Command line arguments have highest precedence, then env vars, then
file vars.

- Add finish date and total time to output
- Add overall job speed
- Moved hosts stats after "Finished" and overall summary
- Ident hosts for better appearance
- Fixed bad grammar in "Finished: 1 successful jobs, 1 failures."
- Fixed too many blank lines in output

My previous attempts weren't working: stdout was still occasionally
lost.  This redoes it to not require pseudoterminal allocation at all
by instead using a pipe to send input to the remote process, and using
an extra subprocess that blocks for input (which never comes), and
kills the subprocess if it complete (i.e. because of an EOF).

It seems to work: I haven't lost any output yet.

This also adds some various logging improvements, showing
connections/disconnections in the log as well as exit status (plus a
reinterpretation of bash's 128+n as a N terminal signal).

The previous commit didn't fix things--some stdout could still be lost.
This is another attempt to fix the problem by making the log file
unbuffered.

Otherwise there is potential conflict since the parent and child both
have the file open--but this can cause data loss, since they'll not
have the same file position after the fork.

So reopen for each job, and close (in the parent) immediately after the
fork.

(Moved from one of my private repositories).