parallel - Man Page

• Gzip several files at once, using all cores: parallel gzip ::: path/to/file1 path/to/file2 ...
• Read arguments from stdin, run 4 jobs at once: ls *.txt | parallel -j4 gzip
• Convert JPEG images to PNG using replacement strings: parallel convert {} {.}.png ::: *.jpg
• Parallel xargs, cram as many args as possible onto one command: args | parallel -X command
• Break stdin into ~1M blocks, feed each block to stdin of new command: cat big_file.txt | parallel --pipe --block 1M command
• Run on multiple machines via SSH: parallel -S machine1,machine2 command ::: arg1 arg2
• Download 4 files simultaneously from a text file containing links showing progress: parallel -j4 --bar --eta wget -q {} :::: path/to/links.txt
• Print the jobs which parallel is running in stderr: parallel -t command ::: args

tldr.sh

parallel [options] [command]-- [argument ...]

parallel [options]-- [command ...]

parallel runs the specified command, passing it a single one of the specified arguments. This is repeated for each argument. Jobs may be run in parallel. The default is to run one job per CPU.

If no command is specified before the --, the commands after it are instead run in parallel.

-j maxjobs

Use to limit the number of jobs that are run at the same time.

-l maxload

Wait as needed to avoid starting new jobs when the system's load average is not below the specified limit.

-i

Normally the command is passed the argument at the end of its command line. With this option, any instances of "{}" in the command are replaced with the argument.

-n

Number of arguments to pass to a command at a time. Default is 1. Incompatible with -i

parallel sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

This runs three subshells that each print a message, delay, and print another message. If your system has multiple CPUs, parallel will run some of the jobs in parallel, which should be clear from the order the messages are output.

parallel -j 3 ufraw -o processed -- *.NEF

This runs three ufraw processes at the same time until all of the NEF files have been processed.

parallel -j 3 -- ls df "echo hi"

This runs three independent commands in parallel.

Its exit status is the combination of the exit statuses of each command ran, ORed together. (Thus, if any one command exits nonzero, parallel as a whole will exit nonzero.)

All output to stdout and stderr is serialised through a corresponding internal pipe, in order to prevent annoying concurrent output behaviour. Note that serialisation is not done on any other file descriptors and file position based access to a nonstandard file descriptor might have unexpected results.

Tollef Fog Heen

env_parallel(1), niceload(1), parallel_examples(7), parcat(1), parset(1), pdfgrep(1), sem(1).