Parallel Processing HOWTO: multi-core, MPI and clusters on Linux

Updated November 2026

A program runs faster in parallel only if its work can be split into pieces that run at the same time. On Linux today you have four practical levels: run many independent jobs at once (xargs, GNU parallel), use threads inside one program (OpenMP), pass messages between processes on one or many machines (MPI), and hand the work to a batch cluster (Slurm). This page shows each one with a working example.

The classic version of this HOWTO, from the days of SMP kernels, Beowulf clusters and MMX, is kept by the Linux Documentation Project at tldp.org. What it called "SMP Linux" is now simply every Linux system: the kernel schedules threads across all cores without any special build.

Choose the right level

Which tool fits which problem
Your problemUseMachines
Thousands of files to convert, compress or checkxargs -P or GNU parallelOne (or a few over SSH)
One program with a big loop over arraysOpenMP, C++ threadsOne
A simulation too big for one machine's memoryMPI (Open MPI, MPICH)Many
Many users sharing a clusterSlurm (with any of the above inside a job)Many
Dense number crunching, machine learningGPU: CUDA (NVIDIA) or ROCm/HIP (AMD)One or many

Start at the top. Running independent jobs side by side needs no code changes and often gives the biggest win.

Know your cores first

nproc            # processing units available to you
lscpu            # sockets, cores per socket, threads per core, caches, NUMA nodes

A "thread per core" value of 2 means simultaneous multithreading (Hyper-Threading): 8 cores show up as 16 CPUs. Heavy floating-point work often gains little from the second thread of a core.

Expect less than a perfect speed-up. If 90% of a program's run time can run in parallel and 10% cannot, eight cores give at most 1 / (0.10 + 0.90 / 8) = 1 / 0.2125 ≈ 4.7 times the speed, not 8. This limit is called Amdahl's law. Measure the serial part before buying more cores.

Many independent jobs

xargs -P

GNU xargs is installed everywhere. -P sets how many commands run at once (the default is 1), and -n 1 passes one file to each command. Use -print0 and -0 so that file names with spaces work:

find . -name '*.log' -print0 | xargs -0 -P "$(nproc)" -n 1 gzip -9

The xargs manual warns that output from parallel commands comes out in no fixed order and can be mixed up. Write each job's output to its own file, or use GNU parallel.

GNU parallel

GNU parallel (release 20261022 as of November 2026) keeps output from different jobs separate, can keep input order, logs every job and can spread work over other machines by SSH. Install it with sudo apt install parallel, sudo dnf install parallel or sudo pacman -S parallel. By default it runs one job per CPU thread.

# one job per argument after :::
parallel gzip -9 ::: *.log

# {} is the input, {.} removes the extension, {/} is the base name
parallel ffmpeg -i {} {.}.flac ::: music/*.wav

# read file names from find, NUL-separated, keep output in input order
find . -name '*.txt' -print0 | parallel -0 -k wc -l {}

# every combination of two input lists: a-1 a-2 b-1 b-2
parallel echo {1}-{2} ::: a b ::: 1 2

For long runs, add a progress bar, a job log and a stop rule:

parallel --bar --joblog jobs.tsv --halt now,fail=1 ./process.sh {} ::: data/*.csv

jobs.tsv gets one tab-separated line per job with its start time, run time and exit value. --halt now,fail=1 kills the remaining jobs as soon as one fails. Test any new command line with --dry-run first: it prints the commands without running them.

To use other machines, give them with -S (for example -S 8/node1,8/node2 for eight jobs on each). The remote hosts need SSH key login and GNU parallel installed.

Citation notice. GNU parallel prints a request to cite it in academic work. Run parallel --citation once to read it and stop the message.

Threads in one program: OpenMP

OpenMP adds parallel loops to C, C++ and Fortran with compiler directives. GCC and Clang support it with -fopenmp. This program estimates π by adding up many thin slices of a curve:

/* pi_omp.c - estimate pi by integrating 4/(1+x^2) from 0 to 1 */
#include <omp.h>
#include <stdio.h>

int main(void)
{
    const long n = 200000000;          /* number of slices */
    const double h = 1.0 / n;
    double sum = 0.0;

    double t0 = omp_get_wtime();
    #pragma omp parallel for reduction(+:sum)
    for (long i = 0; i < n; i++) {
        double x = (i + 0.5) * h;
        sum += 4.0 / (1.0 + x * x);
    }
    double t1 = omp_get_wtime();

    printf("pi = %.12f using %d threads in %.2f s\n",
           sum * h, omp_get_max_threads(), t1 - t0);
    return 0;
}
gcc -O2 -fopenmp pi_omp.c -o pi_omp
OMP_NUM_THREADS=1 ./pi_omp
OMP_NUM_THREADS=4 ./pi_omp

On a test machine the run time dropped from 0.25 s with one thread to 0.07 s with four, with the same result. Without OMP_NUM_THREADS the runtime uses all CPUs.

The reduction(+:sum) clause matters. Without it, all threads would add to the same sum at once, a race condition that gives a different wrong answer on each run. Variables declared inside the loop (x) are private to each thread automatically.

Other ways to use threads: POSIX threads (pthread_create) in C, std::thread and std::jthread in C++, and parallel algorithms such as std::sort(std::execution::par, …). See the C++ HOWTO for build flags and ThreadSanitizer, which finds races.

Message passing: MPI

MPI (Message Passing Interface) runs several copies of a program, called ranks, that exchange data by sending messages. The same program works on one machine or across a cluster. Open MPI's current release series is 5.0 (v5.0.11 as of November 2026); MPICH is the other main implementation.

# Debian, Ubuntu
sudo apt install openmpi-bin libopenmpi-dev

# Fedora: the wrappers are not on PATH until you load the module
sudo dnf install openmpi-devel
module avail mpi
module load mpi/openmpi-x86_64

The same π sum, split across ranks. Each rank adds every size-th slice, then MPI_Reduce adds the partial sums on rank 0:

/* pi_mpi.c - the same sum split across MPI processes */
#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    int rank, size;
    const long n = 200000000;
    const double h = 1.0 / n;
    double local = 0.0, total = 0.0;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);    /* my number: 0 .. size-1 */
    MPI_Comm_size(MPI_COMM_WORLD, &size);    /* how many processes */

    for (long i = rank; i < n; i += size) {  /* every size-th slice */
        double x = (i + 0.5) * h;
        local += 4.0 / (1.0 + x * x);
    }

    MPI_Reduce(&local, &total, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
    if (rank == 0)
        printf("pi = %.12f from %d processes\n", total * h, size);

    MPI_Finalize();
    return 0;
}
mpicc -O2 pi_mpi.c -o pi_mpi
mpirun -n 4 ./pi_mpi

Expected output: pi = 3.141592653590 from 4 processes. Use mpic++ for C++. Without -n, mpirun starts one process per slot; with no hostfile or scheduler, a machine has one slot per processor core.

To run across machines, list them in a hostfile. slots is the most processes to start on each host:

# hosts.txt
node1.example.com slots=16
node2.example.com slots=16
mpirun --hostfile hosts.txt -n 32 ./pi_mpi

Every node needs the same Open MPI version, the program at the same path (a shared NFS directory is easiest), and SSH key login from the first node to the others. For cluster storage, see the NFS HOWTO.

Clusters: Slurm

What used to be called a Beowulf cluster, ordinary Linux machines on a fast network, is today usually run by a batch scheduler. Slurm (documentation for version 26.05 as of November 2026) is the most common. You write a job script, submit it, and Slurm starts it when the cores you asked for are free.

#!/bin/bash
#SBATCH --job-name=pi
#SBATCH --nodes=2
#SBATCH --ntasks=32
#SBATCH --time=00:10:00
#SBATCH --output=pi-%j.out

srun ./pi_mpi
sbatch pi.sbatch      # prints: Submitted batch job 4711
squeue -u "$USER"     # your waiting and running jobs
scancel 4711          # cancel a job
sinfo                 # partitions and node states

%j in the output name becomes the job ID. srun starts the MPI ranks inside the allocation; mpirun ./pi_mpi with no -n also works there, because Open MPI reads the process count and hosts from Slurm.

For an OpenMP program, ask for one task with several CPUs and pass the count on:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --time=00:05:00

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun --cpus-per-task="$SLURM_CPUS_PER_TASK" ./pi_omp

For many independent runs, a job array replaces GNU parallel: #SBATCH --array=1-100 starts 100 copies, and each one reads its number from $SLURM_ARRAY_TASK_ID.

GPUs and vector units

The old HOWTO's "SIMD within a register" with MMX is now done by the compiler: at -O2 and -O3, GCC and Clang turn simple loops into SSE, AVX or NEON vector instructions. -march=native uses every instruction your CPU has, but the binary may then fail on older CPUs.

For far more parallel arithmetic, use a GPU. NVIDIA's CUDA Toolkit (13.4 as of November 2026) lists Debian 12 and 13, Ubuntu 24.04 and 26.04 LTS, Fedora 44 and RHEL 9 and 10 among its supported systems; AMD's ROCm platform provides the HIP runtime for its GPUs. Both need a matching kernel driver, so install them from the vendor's instructions for your exact distribution release.

Mistakes to avoid

Sources