UC San Diego
Integrated Digital Infrastructure
SearchMenu

Compiling Jobs on TSCC

Serial and MPI Programs

This page describes how to compile codes for the Triton Shared Computing Cluster nodes, including the Petascale Data Analysis Facility (PDAF) nodes.

Porting Existing Programs to TSCC

If you have an existing serial or MPI-based parallel application program already running on a distributed-memory platform:

Copy your application source files to your $HOME directory or to a Data Oasis area for your account:

/oasis/tscc/scratch/<username> where <username> is your TSCC login name

Note that Data Oasis storage is not backed up and files stored on this system may be lost or destroyed without recourse to restore them. Long-term file storage should be maintained in your $HOME directory or project storage.

Compiling Serial and MPI Codes


Compiling MPI Codes

MPI source code should be recompiled for TSCC using the following default compiler commands:

  • mpicc [options] file.c (C and C++) myrinet/mx switch and the Portland Compiler
  • mpif77 [options] file.f (Fortran 77 source code) myrinet/mx switch and the Portland Compiler
  • mpif90 [options] file.f (free format code/dynamic memory allocation/object oriented Fortran source code) openmpi and the Intel compiler

Other MPI stack/compiler combinations may be obtained by choosing the appropriate modules. The choices include:

InfiniBand
  • PGI compilers + openmpi_ib
  • PGI compilers + mvapich2_ib
  • Intel compilers + openmpi_ib
  • Intel compilers + mvapich2_ib
  • GNU compilers + openmpi_ib
  • GNU compilers + mvapich2_ib
Compiling Serial Codes

Serial source code should be recompiled for the TSCC system with the following compiler commands:

Portland Group Compilers
  • pgcc [options] file.c (C and C++)
  • pgf77 [options] file.f (fixed form Fortran source code)
  • pgf90 [options] file.f90 (free format Fortran source code)
Intel Compilers
  • icc [options] file.c (C and C++)
  • ifort [options] file.f (fixed form Fortran source code)
  • ifort [options] file.f90 (free format Fortran source code)
Gnu Compilers
  • gcc [options] file.c (C and C++)
  • gfortran [options] file.f90 (free format Fortran source code)
Compatibility Options for Fortran

  • -Mcpp run the Fortran preprocessor on source files prior to compilation
  • -Dname[=value] specify name as a definition to use with conditional compilation directives or the Fortran preprocessor (-Mcpp)
  • -silent/-w/-Minform=severe suppress messages about use of non-standard Fortran
  • -byteswapio swap bytes from big-endian to little-endian or vice-versa for unformatted files
  • -i8 set size of INTEGER and LOGICAL variables to 8 bytes
  • -i4 set size of INTEGER and LOGICAL variables to 4 bytes
  • -i2 set size of INTEGER variables to 2 bytes
  • -r8 treat REAL and CMPLX types as REAL*8 and DCMPLX
  • -Msave save all local variables between calls (static allocation)
Detecting Programming Errors

  • -g produce symbolic debug information in object file (implies -O0); required for debugging with DDT
  • -C/-Mboundsarray bounds checking
  • -kTrap=(option, option...)specifies behavior on floating point exceptions (used only for main program):
    • dnorm trap on denormalized (very, very small) operands
    • divz trap on divide by zero
    • fp trap on floating point exceptions
    • inexact trap on inexact result
    • inv trap on invalid operands
    • none (default) disables all traps
    • ovf trap on floating point overflow
    • unf trap on floating point underflow
  • -traceback add debug information
Optimization

Optimization levels of the Portland Group compilers are:

  • -O0 No optimization
  • -O1 (default) Task scheduling within extended basic blocks is performed. Some register allocation; no global optimizations.
  • -O2 all level 1 optimizations and global scalar optimizations (optimization over all blocks)
  • -O3/O4 all level 1 and 2 optimizations as well as more agressive optimizations
  • -Mflushz flush very, very small values to zero.

Alternatively, the Portland Compiler User Guide recommends the use of the -fast flag. This option is host dependent and typically has the following effects:

  • set the optimization level at -O2
  • unroll loops (-Munroll=c:1)
  • do not generate code to set up a stack frame pointer for every function (-Mnoframe)
  • enable loop redundancy elimination
Numerical Libraries

The Portland Group compilers come with the Optimized ACML library (LAPACK/BLAS/FFT).

To link:

pg90/pgf77 myprog.f -llapack -lblas

Intel has developed Math Kernel Library (MKL) which contains many linear algebra, FFT and other useful numerical routines.

  • Basic linear algebra subprograms (BLAS) with additional sparse routines
  • Fast Fourier Transforms (FFT) in 1 and 2 dimensions, complex and real
  • The linear algebra package, LAPACK
  • A C interface to BLAS
  • Vector Math Library (VML)
  • Vector Statistical Library (VSL)
  • Multi-dimensional Discrete Fourier Transforms (DFTs)

Documentation is available in HTML and PDF formats in${MKL_ROOT}/../Documentation.

To link the MKL libraries


To link the MKL libraries, please refer to the Intel MKL Link Line Advisor Web page. This tool accepts inputs for several variables based on your environment and automatically generates a link line for you.

When using the output generated by this site, substitute the Triton path of the Intel MKL for the value $MKLPATH in the generated script. That value is${MKL_ROOT}/lib/em64t.

All third-party applications can be found in /opt, or view the complete description of Triton software packages on the TSCC Software page.