UMBC logo
UMBC High Performance Computing Facility
How to use PETSc on tara

Introduction

The PETSc (Portable, Extensible Toolkit for Scientific Computation) library "is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations." It provides a useful framework of parallel data structures, solvers, preconditioners, etc, so that you can focus on solving your PDE problem. In this tutorial, we'll show how to build and run one of the examples that comes with PETSc, which solves a "variable Poisson" problem. We will not discuss the details of problem here, we will just focus on getting the example to run on tara. Before you begin, make sure to read the tutorial for compiling C programs.

Where to find documentation

PETSC is thoroughly documented by its authors at the PETSc home page. On tara, PETSc is installed in the following locations (corresponding to the currently available MPI/compiler combinations): Each location has an "include" directory with PETSc header files which you can browse.

Example

The example we'll use is ex13.c . An easy way to get the example to your tara account is with wget. Notice that we are retrieving the plain C file, and not the HTML-ized version. 
[araim1@tara-fe1 petsc-poisson]$ wget http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/
src/ksp/ksp/examples/tutorials/ex13.c
(The command should be one continuous line). The next thing we'll need is a Makefile 
EXECUTABLE := petsc_ex13
OBJS := ex13.o

CFLAGS := -O3 -c99 -Minform=warn -fastsse

INCLUDES := -I$(PETSC_DIR)/include
LIBLOCS := -L$(PETSC_DIR)/lib
LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm

CC := mpicc $(INCLUDES)

%.o: %.c %.h
    $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) -c $< -o $@

$(EXECUTABLE): $(OBJS)
    $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) $(OBJS) -o $@ $(LIBLOCS) $(LDFLAGS)

clean:
    -rm -f *.o $(EXECUTABLE)
Download: ../code/petsc-poisson/Makefile
The most important lines, which will allow us to use the PETSc library are: 
INCLUDES := -I$(PETSC_DIR)/include
LIBLOCS := -L$(PETSC_DIR)/lib
LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm
Notice that the environment variable PETSC_DIR is used, but not defined in this file. We will define it in our session before proceeding (if you forget to do this, compiling and execution will fail!). We will assume you're using the default PGI+MVAPICH2 switcher setting, so execute the following commands. 
[araim1@tara-fe1 petsc-poisson]$ export PETSC_ARCH=cxx-opt
[araim1@tara-fe1 petsc-poisson]$ export PETSC_DIR=/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt
[araim1@tara-fe1 petsc-poisson]$ export LD_LIBRARY_PATH=:/lib
[araim1@tara-fe1 petsc-poisson]$
Note that you need to set the environment variables PETSC_ARCH, PETSC_DIR, and LD_LIBRARY_PATH in each tara session, any time you want to use PETSc. To automate this, add the three above lines into

/home/<your_username>/.bash_profile

Now we should be able to compile the example 
[araim1@tara-fe1 petsc-poisson]$ make
mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
-O3 -c99 -Minform=warn -fastsse   -c -o ex13.o ex13.c
mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
-O3 -c99 -Minform=warn -fastsse  
-I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
ex13.o -o petsc_ex13 -L/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib 
-lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm
[araim1@tara-fe1 petsc-poisson]$
Since this is a small example, we can try to run it directly on the command line 
[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 
m 6 n 7 error norm 0.000925256
m 6 n 7 error norm 0.000925256
[araim1@tara-fe1 petsc-poisson]$
The example ran, but all we got at the end was the error norm. Let's see what other kinds of things we can do with the example by specifying the "-help' flag. 
[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 -help
Solves a variable Poisson problem with KSP.

--------------------------------------------------------------------------
Petsc Release Version 3.0.0, Patch 9, Fri Oct 23 13:37:28 CDT 2009
       The PETSc Team
    petsc-maint@mcs.anl.gov
 http://www.mcs.anl.gov/petsc/
See docs/copyright.html for copyright information
See docs/changes/index.html for recent updates.
See docs/troubleshooting.html for problems.
See docs/manualpages/index.html for help. 
Libraries linked from /usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib
--------------------------------------------------------------------------
Options for all PETSc programs:
... (tons of options) ...
[araim1@tara-fe1 petsc-poisson]$
To illustrate one option, let's try "-ksp_view" to view the parameters of the linear solver 
[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 -ksp_view
KSP Object:
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
  left preconditioning
PC Object:
  type: lu
    LU: out-of-place factorization
      matrix ordering: nd
    LU: tolerance for zero pivot 1e-12
    LU: factor fill ratio needed 2.38043
         Factored matrix follows
        Matrix Object:
          type=seqaij, rows=42, cols=42
          package used to perform factorization: petsc
          total: nonzeros=438, allocated nonzeros=438
            not using I-node routines
  linear system matrix = precond matrix:
  Matrix Object:
    type=seqaij, rows=42, cols=42
    total: nonzeros=184, allocated nonzeros=210
      not using I-node routines
m 6 n 7 error norm 0.000925256
...
[araim1@tara-fe1 petsc-poisson]$
Running the example through the batch system requires no special effort. As long as your environment variables are set correctly, your executable should be able to find the PETSc runtime libraries. As a quick example, the following batch script should be sufficient. 
#!/bin/bash
#SBATCH --job-name=petsc_ex13
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --partition=develop
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

srun ./petsc_ex13
Download: ../code/petsc-poisson/mvapich2.slurm