The CHEOPS cluster

The CHEOPS cluster is a high-performance computing cluster with 9712 Intel Nehalem compute cores spread over 841 nodes, which we employ in a broad variety of number crunching projects.

CHEOPS resources

HPC Antrag to get a user account on CHEOPS [pdf]
The CHEOPS primer / brief instructions [pdf]

ALPS on CHEOPS

A system-wide ALPS build of version 2.1.1 is available at

/opt/rrzk/software/alps/2.1.1
Compiling ALPS on the CHEOPS cluster
- Download and unzip the ALPS sources, which creates a new ALPS source directory
  curl -O https://alps.comp-phys.org/static/software/releases/alps-2.1.1-r6176-src-with-boost.tar.gz
  tar -xzvf alps-2.1.1-r6176-src-with-boost.tar.gz
- Create and move to a build directory
  mkdir ~/alps-2.1.1-bld-with-boost
  cd ~/alps-2.1.1-bld-with-boost
- Load a couple of necessary modules via:
  
  module load cmake/2.8.4 intel/12.0 mkl/10.3 openmpi/1.4.3 hdf5/1.8.8
- Call cmake to configure ALPS (the following is a single line):
  
  cmake -DBoost_ROOT_DIR:PATH=~/alps-2.1.1-r6176-src-with-boost/boost
  -DCMAKE_INSTALL_PREFIX=~/alps/2.1.1
  -DHDF5_LIBRARIES:FILEPATH=/opt/rrzk/lib/hdf5/1.8.8/lib/libhdf5.so
  -DHDF5_INCLUDE_DIR:PATH=/opt/rrzk/lib/hdf5/1.8.8/include
  -DZIP_LIBRARIES:FILEPATH=/opt/rrzk/lib/hdf5/1.8.8/lib/libz.so
  -DZIP_INCLUDE_DIR:PATH=/opt/rrzk/lib/hdf5/1.8.8/include
  -DSZIP_LIBRARIES:FILEPATH=/opt/rrzk/lib/hdf5/1.8.8/lib/libsz.so
  -DSZIP_INCLUDE_DIR:PATH=/opt/rrzk/lib/hdf5/1.8.8/include
  -DBLAS_LIBRARY=/opt/rrzk/compiler/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_rt.so
  -DLAPACK_LIBRARY=/opt/rrzk/compiler/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_rt.so
  -DFFTW_LIBRARIES:FILEPATH=/opt/rrzk/compiler/intel/composerxe-2011.5.220/mkl/lib/intel64/libfftw3xc_intel.a
  -DFFTW_INCLUDE_DIR:PATH=/opt/rrzk/compiler/intel/composerxe-2011.5.220/mkl/include/fftw
  -DALPS_BUILD_PYTHON=OFF
  ~/alps-2.1.1-r6176-src-with-boost/alps
- Compile the ALPS sources:
  
  make -j 4
  make test
  make install
- Congratulations! Your own ALPS build should now be available. The binaries are located in
  ~/alps/2.1.1/bin/

Job submission

An example submission script for an MPI job
#!/bin/csh
#PBS -q default
#PBS -l nodes=2:ppn=8
#PBS -l mem=24gb
#PBS -l walltime=01:00:00
#PBS -A AG-Trebst

setenv WORKDIR $HOME
setenv ALPSDIR $HOME/alps/2.1.1
setenv PBSODIR $HOME/Jobs

setenv NCORES `cat $PBS_NODEFILE | wc -l`
setenv NNODES `uniq $PBS_NODEFILE | wc -l`

cd $PBSODIR
module load intel/12.0
module load mkl/10.3
module load openmpi/1.4.3
module load hdf5/1.8.8

# output start time
echo start time
date
echo

# start MPI job
mpirun -np $NCORES $ALPSDIR/bin/spinmc --mpi $WORKDIR/scan.in.xml

# output finish time
echo finish time
date
echo

Job management and queue status

Max Gerlach has put together a number of useful scripts to manage your jobs and check the status of the queue: https://github.com/maxhgerlach/cheops-scripts