This document provides specific instructions for installing PIO using a variety of compilers on a few commonly used super computers. Click on the link below to go directly to the machine of interest.

• Yellowstone (NCAR's 1.5-petaflop IBM Supercomputer)
• Edison (A NERSC Cray XC30 Supercomputer)
• Mira (IBM Blue Gene Supercomputer at ALCF)
• Blue Waters (NCSA's 1.3-petaflop Cray Supercomputer)
• Hobart (The NCAR CGD local cluster)
• Linux with MPICH (Standard Linux box with MPICH)

Yellowstone

1. Directory setup

   Download a copy of the PIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

2. Modules

   Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building PIO on Yellowstone.

   • Intel

     %> module reset
     %> module unload netcdf
     %> module swap intel intel/15.0.3
     %> module load git/2.3.0
     %> module load cmake/3.0.2
     %> module load netcdf-mpi/4.3.3.1
     %> module load pnetcdf/1.6.1
     

   • GNU

     %> module reset
     %> module unload netcdf
     %> module swap intel gnu/4.8.2
     %> module load git/2.3.0
     %> module load cmake/3.0.2
     %> module load netcdf-mpi/4.3.3.1
     %> module load pnetcdf/1.6.1
     

   • PGI

     %> module reset
     %> module unload netcdf
     %> module swap intel pgi/13.3
     %> module load git/2.3.0
     %> module load cmake/3.0.2
     %> module load netcdf-mpi/4.3.3.1
     %> module load pnetcdf/1.6.1
     

3. Environment Variables

   The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
   CC=mpicc
   FC=mpif90
   

4. Build

   Building PIO requires running the CMake configure and then make. In the PIO_build directory type
   %> cmake ../PIO_source/
   %> make
   

Edison

1. Directory setup

   Download a copy of the PIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

2. Modules

   Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building PIO on Edison.

   • Intel

     %> module purge
     %> module load PrgEnv-intel
     %> module load craype-ivybridge
     %> module load cray-shmem
     %> module load cray-mpich
     %> module load torque
     %> module load git/2.4.6
     %> module load cmake/3.0.0
     %> module load cray-hdf5-parallel/1.8.14
     %> module load cray-netcdf-hdf5parallel/4.3.3.1
     %> module load cray-parallel-netcdf/1.6.1
     

   • GNU

     %> module purge
     %> module load PrgEnv-gnu
     %> module load craype-ivybridge
     %> module load cray-shmem
     %> module load cray-mpich
     %> module load torque
     %> module load git/2.4.6
     %> module load cmake/3.0.0
     %> module load cray-hdf5-parallel/1.8.14
     %> module load cray-netcdf-hdf5parallel/4.3.3.1
     %> module load cray-parallel-netcdf/1.6.1
     

   • Cray

     %> module purge
     %> module load PrgEnv-cray
     %> module load craype-ivybridge
     %> module load cray-shmem
     %> module load cray-mpich
     %> module swap cce cce/8.4.0.223
     %> module load torque
     %> module load git/2.4.6
     %> module load cmake/3.0.0
     %> module load cray-hdf5-parallel/1.8.14
     %> module load cray-netcdf-hdf5parallel/4.3.3.1
     %> module load cray-parallel-netcdf/1.6.1
     

3. Environment Variables

   The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
   CC=cc
   FC=ftn
   

4. Build

   Building PIO requires running the CMake configure and then make. In the PIO_build directory type
   %> cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE \
   -DPREFER_STATIC=TRUE \
   -DNetCDF_PATH=${NETCDF_DIR} \
   -DPnetCDF_PATH=${PARALLEL_NETCDF_DIR} \
   -DHDF5_PATH=${HDF5_DIR} \
   -DMPI_C_INCLUDE_PATH=${MPICH_DIR}/include \
   -DMPI_Fortran_INCLUDE_PATH=${MPICH_DIR}/include \
   -DMPI_C_LIBRARIES=${MPICH_DIR}/lib/libmpich.a \
   -DMPI_Fortran_LIBRARIES=${MPICH_DIR}/lib/libmpichf90.a \
   -DCMAKE_SYSTEM_NAME=Catamount \
   ../PIO_source/
   %> make

Mira/Cetus

1. Directory setup

   Download a copy of the PIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

2. Softenv packages and environment variables

   It is not necessary to edit your .soft file on Mira inorder to build PIO. Execute the following commands to temporarily load packages into your softenv. These packages use the IBM/XL compiler.
   %> soft add +mpiwrapper-xl (or switch from the default in your softenv)
   %> soft add @ibm-compilers-2015-02
   %> soft add +cmake
   %> soft add +git
   

   And then set the following environment variables to add in the rest of the library paths.
   %> setenv LIBZ /soft/libraries/alcf/current/xl/ZLIB
   %> setenv HDF5 /soft/libraries/hdf5/1.8.14/cnk-xl/V1R2M2-20150213
   %> setenv NETCDF /soft/libraries/netcdf/4.3.3-f4.4.1/cnk-xl/V1R2M2-20150213
   %> setenv PNETCDF /soft/libraries/pnetcdf/1.6.1/cnk-xl/V1R2M2-20150213
   %> setenv CC /soft/compilers/wrappers/xl/mpixlc_r
   %> setenv FC /soft/compilers/wrappers/xl/mpixlf90_r
   

3. Build

   Building PIO requires running the CMake configure and then make. In the PIO_build directory type
   %> cmake -DPREFER_STATIC=TRUE ../PIO_source/
   %> make

Blue Waters

1. Directory setup

   Download a copy of the PIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

2. Modules

   Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building PIO on Hobart.

   • Intel

     %> module swap PrgEnv-cray PrgEnv-intel
     %> module load torque
     %> module load git
     %> module load cmake
     %> module load cray-hdf5-parallel/1.8.14
     %> module load cray-netcdf-hdf5parallel/4.3.3.1
     %> module load cray-parallel-netcdf/1.6.1
     

   • PGI

     %> module swap PrgEnv-cray PrgEnv-pgi
     %> module load torque
     %> module load git
     %> module load cmake
     %> module load cray-hdf5-parallel/1.8.14
     %> module load cray-netcdf-hdf5parallel/4.3.3.1
     %> module load cray-parallel-netcdf/1.6.1
     

3. Environment Variables

   The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
   CC=cc
   FC=ftn
   

4. Build

   Building PIO requires running the CMake configure and then make. In the PIO_build directory type
   %> cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE \
   -DPREFER_STATIC=TRUE \
   -DNetCDF_PATH=${NETCDF_DIR} \
   -DPnetCDF_PATH=${PARALLEL_NETCDF_DIR} \
   -DHDF5_PATH=${HDF5_DIR} \
   -DMPI_C_INCLUDE_PATH=${MPICH_DIR}/include \
   -DMPI_Fortran_INCLUDE_PATH=${MPICH_DIR}/include \
   -DMPI_C_LIBRARIES=${MPICH_DIR}/lib/libmpich.a \
   -DMPI_Fortran_LIBRARIES=${MPICH_DIR}/lib/libmpichf90.a \
   -DCMAKE_SYSTEM_NAME=Catamount \
   ../PIO_source/
   %> make

Hobart

1. Directory setup

   Download a copy of the PIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

2. Modules

   Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building PIO on Hobart.

   • Intel

     %> module purge
     %> module load compiler/intel/15.0.2.164
     %> module load tool/parallel-netcdf/1.6.1/intel
     

   • Nag

     %> module purge
     %> module load compiler/nag/6.0
     %> module load tool/parallel-netcdf/1.6.1/nag/openmpi
     

   • PGI

     %> module purge
     %> module load compiler/pgi/15.1
     %> module load tool/parallel-netcdf/1.6.1/pgi/mvapich2
     

3. Environment Variables

   The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
   CC=mpicc
   FC=mpif90
   

4. Build

   Building PIO requires running the CMake configure and then make. In the PIO_build directory type
   %> cmake -DNETCDF_DIR=$NETCDF_PATH -DPNETCDF_DIR=$PNETCDF_PATH ../PIO_source/
   %> make

   ParallelIO does not require Parallel netcdf to run, so if you decide to use the GNU compiler on Hobart (not described here) without the parallel-netcdf library, use the cmake configure flags:
   %> cmake -DNETCDF_DIR=$NETCDF_PATH -DWITH_PNETCDF=FALSE ../PIO_source/
   %> make

Linux with MPICH

1. Installing MPICH

   Download from the MPICH2 downloads page. (These instructions were tested using version 3.2). Untar with:

   tar zxf mpich-3.2.tar.gz

   Build with:

   cd mpich-3.2 && ./configure --prefix=/usr/local && make all check
   && sudo make install 

   Now you should be able to access mpicc, mpifort, and mpirun from the command line. (If not, make sure /usr/local/bin is in path.)

   Note that if you wish to use valgrind on your programs, you should configure MPICH like this:

   CPPFLAGS=-I/usr/include/valgrind ./configure --prefix=/usr/local --enable-g=mem,meminit

   For this to work you must have packages valgrind and valgrind-devel installed.

2. Installing Zlib

   Download from the zlib downloads page. (These instructions were tested using version 1.2.11). Untar with:

   tar zxf zlib-1.2.11.tar.gz

   Build with:

   cd zlib-1.2.11
   CC=mpicc ./configure --prefix=/usr/local/zlib-1.2.11_mpich-3.2
   make all check
   sudo make install 

3. Installing SZlib

   Download szlib from the HDF5 szlib page. (These instructions were tested using version 2.1). Untar with:

   tar zxf szip-2.1.tar.gz

   Build with:

   cd slib-2.1
   CC=mpicc ./configure --prefix=/usr/local/szip-2.1_mpich-3.2
   make all check
   sudo make install 

4. Installing pNetCDF

   Download parallel-netcdf from the parallel-netcdf download page. (These instructions were tested using version 1.8.1). Untar with:

   tar zxf parallel-netcdf-1.8.1.tar.gz

   Build with:

   cd parallel-netcdf-1.8.1
   FC=mpifort CC=mpicc CFLAGS=-fPIC ./configure --prefix=/usr/local/pnetcdf-1.8.1_mpich-3.2
   make all check
   sudo make install 

5. Installing HDF5

   Download HDF5 from the HDF5 download page.

   These instructions were tested using version 1.10.1. Untar with:

   tar zxf hdf5-1.10.1.tar.gz

   . Note that in my case I need to add /usr/local/bin to the PATH for su, because that is where mpicc is found.

   Build with:

   cd hdf5-1.10.1
   CC=mpicc ./configure --with-zlib=/usr/local/zlib-1.2.11_mpich-3.2 --with-szlib=/usr/local/szip-2.1_mpich-3.2 --prefix=/usr/local/hdf5-1.10.1_mpich-3.2 --enable-parallel
   make all check
   sudo PATH=$PATH:/usr/local/bin make install

6. Installing NetCDF-4 C Library

   Download the netcdf C library from the NetCDF download page. Untar with:

   tar zxf netcdf-c-4.5.0-rc1.tar.gz

   Build with:

   cd netcdf-c-4.5.0-rc1
   CPPFLAGS='-I/usr/local/zlib-1.2.11_mpich-3.2 -I/usr/local/szip-2.1_mpich-3.2/include -I/usr/local/hdf5-1.10.1_mpich-3.2/include' LDFLAGS='-L/usr/local/zlib-1.2.11_mpich-3.2/lib -L/usr/local/szip-2.1_mpich-3.2/lib -L/usr/local/hdf5-1.10.1_mpich-3.2/lib' CC=mpicc ./configure --enable-parallel-tests --prefix=/usr/local/netcdf-4.4.1_mpich-3.2
   make all check
   sudo make install 

   Note that you may not build netCDF with it's built-in parallel-netCDF support, if you are also planning to use the parallel-netCDF library with PIO. For PIO, parallel-netCDF must be installed independently of netCDF.

7. Installing NetCDF-4 Fortran Library

   Download the netcdf Fortran library from the NetCDF download page. Untar with:

   tar zxf netcdf-fortran-4.4.4.tar.gz

   Build with:

   cd netcdf-fortran-4.4.4
   CC=mpicc LD_LIBRARY_PATH=/usr/local/netcdf-4.4.1_mpich-3.2/lib FC=mpifort CPPFLAGS=-I/usr/local/netcdf-4.4.1_mpich-3.2/include LDFLAGS=-L/usr/local/netcdf-4.4.1_mpich-3.2/lib ./configure --enable-parallel-tests --prefix=/usr/local/netcdf-fortran-4.4.4_c_4.4.1_mpich-3.2
   make all
   sudo make install 

   Note that make check did not work because of a failure in a test. This has been reported to the netCDF team.

8. Installing ParallelIO Library

   Clone the ParallelIO library.

   Create a build directory and run cmake.

   cd ParallelIO
   mkdir build
   cd build
   CC=mpicc FC=mpifort cmake -DNetCDF_C_PATH=/usr/local/netcdf-4.4.1_mpich-3.2 -DNetCDF_Fortran_PATH=/usr/local/netcdf-fortran-4.4.4_c_4.4.1_mpich-3.2 -DPnetCDF_PATH=/usr/local/pnetcdf-1.8.1_mpich-3.2 -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On ..
   make
   make check
   sudo make install
   

   When debugging build issues, it's helpful to be able to do a clean build from the command line, including tests:

   cd build
   rm -rf * && CFLAGS='-Wall -g' FFLAGS=-g CC=mpicc FC=mpifort cmake -DNetCDF_C_PATH=/usr/local/netcdf-4.4.1_mpich-3.2 -DNetCDF_Fortran_PATH=/usr/local/netcdf-fortran-4.4.4_c_4.4.1_mpich-3.2 -DPnetCDF_PATH=/usr/local/pnetcdf-1.8.1_mpich-3.2 -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On -DPIO_ENABLE_LOGGING=On .. && make VERBOSE=1 all tests check
   

   To build with address sanitizer:

   rm -rf * && CFLAGS='-Wall -g  -fsanitize=address -fno-omit-frame-pointer' FFLAGS='-g  -fsanitize=address -fno-omit-frame-pointer' CC=mpicc FC=mpifort cmake -DNetCDF_C_PATH=/usr/local/netcdf-4.4.1_mpich-3.2 -DNetCDF_Fortran_PATH=/usr/local/netcdf-fortran-4.4.4_c_4.4.1_mpich-3.2 -DPnetCDF_PATH=/usr/local/pnetcdf-1.8.1_mpich-3.2 -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On -DPIO_ENABLE_LOGGING=On .. && make VERBOSE=1 all tests check
   

   Note the optional CFLAGS=-g which allows the use of a debugger with this code. Also note the optional VERBOSE=1 provided to make, which causes the build commands to be made visible.

   Note also the -DPIO_ENABLE_LOGGING=On which is helpful for debugging but should probably not be used in production builds.

9. Building PIO with autotools.

   To build the PIO library with autotools, clone the repo and use a command like this:

   autoreconf -i && LD_LIBRARY_PATH=/usr/local/netcdf-4.4.1_mpich-3.2/lib CC=mpicc CFLAGS='-g' CPPFLAGS='-I/usr/local/netcdf-4.4.1_mpich-3.2/include/ -I/usr/local/pnetcdf-1.8.1_mpich-3.2/include' LDFLAGS='-L/usr/local/netcdf-4.4.1_mpich-3.2/lib -L/usr/local/pnetcdf-1.8.1_mpich-3.2/lib' ./configure && make check
   

   To build with debug logging and the address sanitizer for memory checking (debugging builds only!):

   autoreconf -i && LD_LIBRARY_PATH=/usr/local/netcdf-4.4.1_mpich-3.2/lib CC=mpicc CFLAGS='-g -fsanitize=address -fno-omit-frame-pointer' CPPFLAGS='-I/usr/local/netcdf-4.4.1_mpich-3.2/include/ -I/usr/local/pnetcdf-1.8.1_mpich-3.2/include' LDFLAGS='-L/usr/local/netcdf-4.4.1_mpich-3.2/lib -L/usr/local/pnetcdf-1.8.1_mpich-3.2/lib' ./configure --enable-logging && make check
    

10. Building and Running Performance Tests

    Download a decomp file from our google code page. You can use any of those files, save them to build/test/performance. (These instructions were tested with the first one in the list.)

    Create a namelist file, named "pioperf.nl". Save this file in the subdirectory (note that it is in the BUILD directory):

    build/tests/performance/
    

    The contents of the namelist file should look like:

    &pioperf
    decompfile = "/u/sciteam/thayerca/scratch/pio_work/piodecomp30tasks01dims06.dat"
    pio_typenames = 'pnetcdf'
    niotasks = 30
    rearrangers = 1
    nvars = 2
    /
    

    You should change the path to your decomp file to wherever you saved it. You can add items to the list to run more tests, so, for instance, to test all of the types of io, your pio_typenames would look like: pio_typenames = 'pnetcdf','netcdf','netcdf4p','netcdf4c'

    HDF5 is netcdf4p, and Parallel-Netcdf is pnetcdf.

    Example to test with different numbers of tasks, you could do: niotasks = 30,15,5

    Example to test with both of the rearranger algorithms: rearrangers = 1,2

    Example to test with different numbers of variables: nvars = 8,5,3,2

    Once you have your decomp file and your namelist set up, run like this:

    mpiexec -n 30 ./pioperf
    

    If you run the most basic performance tests (as in the first pioperf.nl example), this script should only take a few minutes (less than 5) to complete. The output and results will be written to your pioperf.o$PBS_JOBID file. Adding more iotypes and rearrangers and variables to these tests will increase the time significantly.

Last updated: 05-16-2016