Using mpi4py on SeaWulf

Using mpi4py on SeaWulf

Overview

mpi4py provides Python bindings for the Message Passing Interface (MPI) standard, allowing Python programs to exploit multiple processors on multiple compute nodes. On SeaWulf, there are two recommended ways to access mpi4py: using the pre-installed system module or creating a custom Anaconda environment.

Method 1: Using the System Module

The simplest way to use mpi4py is to load the pre-configured system module. This is ideal if you do not require a complex custom environment or specific version constraints.

module load mpi4py/latest

This module includes a compatible Python version and the necessary underlying MPI libraries.

Method 2: Using a Custom Conda Environment

If your workflow requires specific Python packages (e.g., specific versions of NumPy, SciPy, or pandas) alongside MPI, we recommend creating a custom environment using Anaconda.

1. Load Anaconda

module load anaconda/3

2. Create the Environment

Create a new environment and install mpi4py. This ensures that a compatible MPI implementation is installed directly within your environment.

conda create --name my_parallel_env mpi4py numpy pandas

3. Activate the Environment

source activate my_parallel_env

Running mpi4py Jobs

Once your environment is set up (either via the module or Conda), you can execute your Python script using mpirun. The mpirun command is provided by the mpi4py installation within your environment.

Example submission script command:

mpirun -n $SLURM_NTASKS python my_script.py

Important Configuration Notes

Do not load other MPI modules.
When using mpi4py/latest or an Anaconda environment with mpi4py installed, a suitable MPI implementation is already included. You should not load other OpenMPI or Intel MPI modules (e.g., module load openmpi/...) alongside mpi4py. Doing so may cause version conflicts and job failures.

If you are using a Conda environment, ensure you install mpi4py via the conda install command rather than pip whenever possible. This ensures that the binary MPI dependencies are correctly configured for the environment.

Example Python Script

Below is a simple "Hello World" example to test your mpi4py setup:

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print(f"Hello from process {rank} out of {size}!")